-
- Downloads
Prevent timestamps in node properties from being shifted
according to the timezone WriteNodeProperties is being run in. Due to our ORC exports using the `timestamp` instead of the `timestamp with timezone`, reader and writer need to agree out of bound on the timezone used in files they exchange. However, we don't do this: * `swh-dataset` uses pyorc, which uses the C++ ORC library, which assumes users (us) always write in GMT * `swh-graph` uses the Java ORC library, which assumes the system timezone (or `$TZ` if set) So when reading with a non-UTC timezone, the Java ORC library interprets timestamps in the dataset as being in the local timezone, and converts them to UNIX timestamps (number of seconds since epoch); then we use these converted timestamps and write them to `.property.author_timestamp.bin` and `.property.committer_timestamp.bin`. This commit regenerates the example graph to have the correct timestamps. It also applies the 39ed0d17 change that removes useless padding at the end of all property files. Resolves #4788
parent
ec014b29
No related branches found
No related tags found
Showing
- java/src/main/java/org/softwareheritage/graph/compress/ORCGraphDataset.java 8 additions, 0 deletions.../org/softwareheritage/graph/compress/ORCGraphDataset.java
- swh/graph/example_dataset/compressed/example.property.author_id.bin 0 additions, 0 deletions...example_dataset/compressed/example.property.author_id.bin
- swh/graph/example_dataset/compressed/example.property.author_timestamp.bin 0 additions, 0 deletions..._dataset/compressed/example.property.author_timestamp.bin
- swh/graph/example_dataset/compressed/example.property.author_timestamp_offset.bin 0 additions, 0 deletions...t/compressed/example.property.author_timestamp_offset.bin
- swh/graph/example_dataset/compressed/example.property.committer_id.bin 0 additions, 0 deletions...mple_dataset/compressed/example.property.committer_id.bin
- swh/graph/example_dataset/compressed/example.property.committer_timestamp.bin 0 additions, 0 deletions...taset/compressed/example.property.committer_timestamp.bin
- swh/graph/example_dataset/compressed/example.property.committer_timestamp_offset.bin 0 additions, 0 deletions...ompressed/example.property.committer_timestamp_offset.bin
- swh/graph/example_dataset/compressed/example.property.content.is_skipped.bin 0 additions, 0 deletions...ataset/compressed/example.property.content.is_skipped.bin
- swh/graph/example_dataset/compressed/example.property.content.length.bin 0 additions, 0 deletions...le_dataset/compressed/example.property.content.length.bin
- swh/graph/example_dataset/compressed/example.property.message.offset.bin 0 additions, 0 deletions...le_dataset/compressed/example.property.message.offset.bin
- swh/graph/example_dataset/compressed/example.property.tag_name.offset.bin 0 additions, 0 deletions...e_dataset/compressed/example.property.tag_name.offset.bin
- swh/graph/tests/luigi/test_compressgraph.py 21 additions, 1 deletionswh/graph/tests/luigi/test_compressgraph.py
- swh/graph/webgraph.py 1 addition, 0 deletionsswh/graph/webgraph.py
Loading
Please register or sign in to comment