Commits · bf030c0f0021cb87bc098ce77d12d82cebe5b427 · Platform / Development / swh-lister

Nov 15, 2019
- gnu.lister: Use url as primary key · bf030c0f
  Antoine R. Dumont authored 5 years ago
  
  Otherwise, we are failing unicity constraint. Related to T2070
  Verified
  
  bf030c0f
- gnu.lister.tests: Add missing assertion · daa9a270
  Antoine R. Dumont authored 5 years ago
  
  Verified
  
  daa9a270
- gnu.lister: Add missing retries_left parameter · 191043ff
  Antoine R. Dumont authored 5 years ago
  
  Verified
  
  191043ff
Nov 14, 2019
- debian.models: Migrate tests from storage to debian lister model · d2512012
  Antoine R. Dumont authored 5 years ago
  
  Related bb5d405
  Verified
  
  d2512012
Nov 13, 2019
- Fix bogus NotImplementedError on Area.index_uris · b2e5ce32
  Nicolas Dandrimont authored 5 years ago
  
  View commits for tag v0.0.40 v0.0.40
  
  b2e5ce32
Nov 12, 2019
- indexing lister: Avoid generating empty or duplicate ranges when partitioning · 773cd337
  Nicolas Dandrimont authored 5 years ago
  
  View commits for tag v0.0.39 v0.0.39
  
  773cd337
- indexing lister: Force bounds of integer ranges to be integers · 2c5528ef
  Nicolas Dandrimont authored 5 years ago
  
  2c5528ef
- Support zero-based indexes in indexing_lister · 1e7a905a
  Nicolas Dandrimont authored 5 years ago
  
  1e7a905a
- Add tests for IndexingLister.db_partition_indices function · e23960ed
  Nicolas Dandrimont authored 5 years ago
  
  e23960ed
- Register lister tasks even if they do not derive from SWHTask. · 8550c560
  vlorentz authored 5 years ago
  
  This happens if swh.scheduler.celery_backend.config isn't imported before the CLI runs.
  8550c560
Nov 08, 2019
- lister.debian: Actually use the db_engine passed to the hook function · ea7a08d0
  Antoine R. Dumont authored 5 years ago
  
  Verified
  
  ea7a08d0
- lister.cli: Clarify configuration loading step · eebbc859
  Antoine R. Dumont authored 5 years ago
  
  Verified
  
  eebbc859
Nov 06, 2019
- swh.lister: Remove completely references to swh.storage.schemata · e8a67a76
  Antoine R. Dumont authored 5 years ago
  
  Related to 56d7cff6
  View commits for tag v0.0.38 v0.0.38 Verified
  
  e8a67a76
- tests: Bump dependency on latest swh-core · 81a31f3c
  Antoine R. Dumont authored 5 years ago
  
  This also modifies the test dataset to filename with url decoded filename. As this is what the latest pytest plugin requires.
  View commits for tag v0.0.37 v0.0.37 Verified
  
  81a31f3c
Nov 05, 2019
- lister.debian: Move `run` method parameters to constructor · e0dbca75
  Antoine R. Dumont authored 5 years ago
  
  View commits for tag v0.0.36 v0.0.36 Verified
  
  e0dbca75
- lister.debian: Default to run a listing on debian distribution · b745c5a7
  Antoine R. Dumont authored 5 years ago
  
  That fixes the `swh lister run --lister debian` cli entrypoint.
  Verified
  
  b745c5a7
- lister.debian: Fix task creation · a60e0bbc
  Antoine R. Dumont authored 5 years ago
  
  By adding a `retries_left`
  Verified
  
  a60e0bbc
- debian.lister: Send origin url as load-debian task parameter · f8727924
  Antoine R. Dumont authored 5 years ago
  
  Instead of the old origin dict. That's what the debian loaders (old and new) expect.
  Verified
  
  f8727924
Nov 04, 2019
- gnu.lister: Update docstrings and fix type annotations · 89b409d3
  Antoine R. Dumont authored 5 years ago
  
  Verified
  
  89b409d3
- gnu.lister: Unify timestamp formats to isoformat date in model · c6372eea
  Antoine R. Dumont authored 5 years ago
  
  Related T2023
  Verified
  
  c6372eea
- gnu.lister: Format timestamp to isoformat string for the tar loader · 93e7afda
  Antoine R. Dumont authored 5 years ago
  
  Related T2023
  Verified
  
  93e7afda
- gnu.tree: Move helper utilities function to the bottom of the file · 9fd64898
  Antoine R. Dumont authored 5 years ago
  
  Verified
  
  9fd64898
- lister.gnu: Move version parsing logic to the lister · 821f3f1c
  Antoine R. Dumont authored 5 years ago
  
  Related D2145 proposition
  Verified
  
  821f3f1c
- debian/lister: Use url parameter name instead of origin · 7c247c8a
  Antoine R. Dumont authored 5 years ago
  
  within the scheduled task. Related D2135
  Verified
  
  7c247c8a
- debian/model: Install lister model within the lister repository · 56d7cff6
  Antoine R. Dumont authored 5 years ago
  
  This is no longer shared between the new debian loader and the lister. The swh.storage.schemata module is still part of the swh.storage module though. As this is still a dependency for the current swh.loader.debian production loader. This will be cleaned up later. Related D2135
  Verified
  
  56d7cff6
Oct 28, 2019
- CRAN lister: fix compute_package_url interpolation · 44bc4462
  Stefano Zacchiroli authored 5 years ago
  
  44bc4462
- mypy: add typing annotations for novel lister abstractions · 6159faa2
  Stefano Zacchiroli authored 5 years ago
  
  6159faa2
- CRAN lister: make shelling out decoding compatible with Python 3.5 · 7dfd811e
  Stefano Zacchiroli authored 5 years ago
  
  7dfd811e
- typing: minimal changes to make a no-op mypy run pass · 974f80f9
  Stefano Zacchiroli authored 5 years ago
  
  974f80f9
Oct 18, 2019

Stop binding tasks to a specific instance of the celery app · 78105940

The celery.shared_task decorator allows late-binding of tasks to any celery app,
which is well suited for our "task plugin" architecture.

78105940

Oct 15, 2019
- debian.lister: Add integration test which checks scheduled tasks · a64ae964
  Antoine R. Dumont authored 5 years ago
  
  Related T2032
  Verified
  
  a64ae964
- pypi.tests: Remove trailing _ in test method name · 960868ba
  Antoine R. Dumont authored 5 years ago
  
  Verified
  
  960868ba
- npm.lister: Align docstrings · b73c657e
  Antoine R. Dumont authored 5 years ago
  
  Verified
  
  b73c657e
- npm.tests: Add an integration test on listing with pagination · b4867ccd
  Antoine R. Dumont authored 5 years ago
  
  Related T2032
  Verified
  
  b4867ccd
Oct 14, 2019
- tests: Update pytest_plugin according to latest version change · a8cde12d
  Antoine R. Dumont authored 5 years ago
  
  Verified
  
  a8cde12d
- tox.ini: Install ipdb dependency for development environment · 9a1314c6
  Antoine R. Dumont authored 5 years ago
  
  Verified
  
  9a1314c6
- tests/conftest: Log the db url used by tests · fcd85216
  Antoine R. Dumont authored 5 years ago
  
  Verified
  
  fcd85216
Oct 12, 2019
- lister.core.db_utils: Remove dead code · 6b1c3d1f
  Antoine R. Dumont authored 5 years ago
  
  Verified
  
  6b1c3d1f
- bitbucket.lister: Add integration test which checks scheduled tasks · f92ac836
  Antoine R. Dumont authored 5 years ago
  
  Related T2032
  Verified
  
  f92ac836
- github.lister: Add integration test which checks scheduled tasks · 0b8b1419
  Antoine R. Dumont authored 5 years ago
  
  Related T2032
  Verified
  
  0b8b1419