-
- Downloads
Crates.io: Add last_update for each version of a crate
In order to reduce http api call amount made by the loader, download a crates.io database dump, and parse its csv files to get a last_update value for each versions of a Crate. Those values are sent to the loader through extra_loader_arguments 'crates_metadata'. 'artifacts' and 'crates_metadata' now uses "version" as key. Related T4104, D8171
Showing
- swh/lister/crates/__init__.py 32 additions, 40 deletionsswh/lister/crates/__init__.py
- swh/lister/crates/lister.py 152 additions, 161 deletionsswh/lister/crates/lister.py
- swh/lister/crates/tests/__init__.py 0 additions, 26 deletionsswh/lister/crates/tests/__init__.py
- swh/lister/crates/tests/data/fake-crates-repository.tar.gz 0 additions, 0 deletionsswh/lister/crates/tests/data/fake-crates-repository.tar.gz
- swh/lister/crates/tests/data/fake_crates_repository_init.sh 47 additions, 44 deletionsswh/lister/crates/tests/data/fake_crates_repository_init.sh
- swh/lister/crates/tests/data/https_static.crates.io/db-dump.tar.gz 0 additions, 0 deletions...r/crates/tests/data/https_static.crates.io/db-dump.tar.gz
- swh/lister/crates/tests/data/https_static.crates.io/db-dump.tar.gz_visit1 0 additions, 0 deletions...s/tests/data/https_static.crates.io/db-dump.tar.gz_visit1
- swh/lister/crates/tests/test_lister.py 80 additions, 79 deletionsswh/lister/crates/tests/test_lister.py
Loading
Please register or sign in to comment