Tags

v1.7.1

Doc:

* Advertise 2024-08-23-popular-500-python teaser dataset
* Advertise 2024-08-23 graph export
* The 2019-01-28-popular-3k-python dataset did not have a compressed graph
* Fix typo

Fixes:

* generate_subdataset: Fix crash on excessively large release.message values

Internal:

* journalprocessor: Fix typo spotted after codespell upgrade
* journalprocessor: Fix black formatting
* Apply swh-py-template v0.2.3

v1.7.0

* orc: Add Bloom filters to some columns
* Small fix in test_journal_processor to make mypy happy
* Update to swh.model 6.13 introducing ModelObjectType

v1.6.0

* Fetch all offsets before exporting any object
* journalprocessor: Add type annotations
* Simplify exporter initialization
* Exclude masked objects from export
* Advertise 2024-05-16 dataset

v1.5.1

- Publish edges export of the 2020-12-15 dataset
- add S3 link for 2020-12-15 compressed graph

v1.5.0

* python: Fix black formatting after bump to 23.1.0 in pre-commit
* Add latest blackify to git-blame-ignore-revs
* Add a “I will respect the ToS” button to the dataset index
* Fix typo in command name
* Advertise 2023-09-06-popular-1k teaser dataset

v1.4.0

- Migrate to copier, PEP420 and pyproject.toml based packaging.

v1.3.6

- fix deps for py37

version 1.3.5

v1.3.4

- Fix dependency issues between boto3 and types-requests

v1.3.3

* luigi: Work around absence of metadata files in old exports
* luigi: Add support for downloading exports in parallel
* luigi: Delete all files in the root dir instead of the root dir itself
* docs: Add links to Terms of Use and 'How to use SWH data'
* Fix Sphinx role (:cls: -> :class:)

v1.3.2

* Override docstrings to avoid broken refs to Luigi's doc

v1.3.1

* luigi: Fix sphinx warnings related to undefined labels

v1.3.0

* Advertise the 'history and history' compressed graph

v1.2.0

* luigi: Actually check whether RunExportAll is complete.
* Advertize 2022-12-07 dataset
* luigi: Make AthenaDatabaseTarget check tables exist

v1.1.0

* Rename 'object_type' to 'object_types' in export metadata
* luigi: Clarify the meaning of 'object_types'
* Invert symlink and content for the README file
* Bump mypy to 1.0.1 and isort to 5.11.5
* Fix tox and pytest config

v1.0.3

* luigi.UploadExportToS3: Skip upload of already-uploaded files
* luigi: Dynamically list directories instead of using object_types
* luigi: Read meta/export.json instead of relying on stamp files
* docs/index.rst: Add missing new line at end of file
* docs/index.rst: Fix sphinx tag name
* docs: Include module indices only when building standalone package doc

v1.0.2

* exporters/orc: Fix crash on visit status with no type
* luigi.CreateAthena: Fix validation of DB name
* luigi.RunExportAll: Default to exporting all formats

v1.0.1

* luigi: Rename classes to be globally unambiguous
* luigi: Send progress reports to the scheduler

v1.0.0

* Update for swh-objstorage >= 2.0.0
* docs/athena: Fix value of --location-prefix
* Fix link to the 2021-03-23 compressed dataset
* cli: Sort object types to be processed in the right order
* cli: Increase open file descriptor limit to support 256 open LevelDBs
* athena: Fix create_table to work with restricted permissions
* Add luigi tasks

version 0.3.2