nixguix: Randomize order of listed origins
The end goal is to ingest sparsely the origins, that would avoid hitting the various servers around the same time for colocated origins in the upstream manifest (especially file or tarball).
Related to T3781 Depends on !427 (closed)
Migrated from D8605 (view on Phabricator)
Merge request reports
Activity
Build is green
Patch application report for D8605 (id=31084)
Could not rebase; Attempt merge onto fa1205c4...
Updating fa1205c..41eef0a Fast-forward requirements-swh.txt | 2 +- setup.py | 1 + swh/lister/__init__.py | 22 ++ swh/lister/gnu/tree.py | 21 +- swh/lister/nixguix/__init__.py | 38 +++ swh/lister/nixguix/lister.py | 374 +++++++++++++++++++++ swh/lister/nixguix/tasks.py | 14 + swh/lister/nixguix/tests/__init__.py | 0 .../nixguix/tests/data/guix-swh_sources.json | 19 ++ .../nixguix/tests/data/nixpkgs-swh_sources.json | 52 +++ swh/lister/nixguix/tests/test_lister.py | 244 ++++++++++++++ swh/lister/nixguix/tests/test_tasks.py | 27 ++ swh/lister/tests/test_cli.py | 4 + 13 files changed, 800 insertions(+), 18 deletions(-) create mode 100644 swh/lister/nixguix/__init__.py create mode 100644 swh/lister/nixguix/lister.py create mode 100644 swh/lister/nixguix/tasks.py create mode 100644 swh/lister/nixguix/tests/__init__.py create mode 100644 swh/lister/nixguix/tests/data/guix-swh_sources.json create mode 100644 swh/lister/nixguix/tests/data/nixpkgs-swh_sources.json create mode 100644 swh/lister/nixguix/tests/test_lister.py create mode 100644 swh/lister/nixguix/tests/test_tasks.py
Changes applied before test
commit 41eef0a80a64d896c8c556d39b7730c8d53b5669 Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Tue Oct 4 10:57:32 2022 +0200 nixguix: Randomize listed origins to ingest sparsely origins Especially the first time around, that would avoid hitting the various servers around the same time for grouped origins (especially file or tarball). Related to [T3781](https://forge.softwareheritage.org/T3781 'view original for T3781 on Phabricator') commit 94b6dbea0a7f602be0711a3bb1f9bb9e16fc48ce Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Sat Oct 1 16:41:48 2022 +0200 nixguix: Document lister Related to [T3781](https://forge.softwareheritage.org/T3781 'view original for T3781 on Phabricator') commit 6d2e7aa17808e39ba9f493b65d662d0ddef5796c Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Sat Oct 1 16:12:46 2022 +0200 nixguix: Register task Related to [T3781](https://forge.softwareheritage.org/T3781 'view original for T3781 on Phabricator') commit fbfdf88ea4fe79c4846ecd48f2a1322f5d3995fc Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Tue Aug 30 11:17:33 2022 +0200 nixguix: Add lister Related to [T3781](https://forge.softwareheritage.org/T3781 'view original for T3781 on Phabricator')
See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/734/ for more details.
Build is green
Patch application report for D8605 (id=31086)
Could not rebase; Attempt merge onto fa1205c4...
Updating fa1205c..cdeb7f5 Fast-forward requirements-swh.txt | 2 +- setup.py | 1 + swh/lister/__init__.py | 22 ++ swh/lister/gnu/tree.py | 21 +- swh/lister/nixguix/__init__.py | 38 +++ swh/lister/nixguix/lister.py | 374 +++++++++++++++++++++ swh/lister/nixguix/tasks.py | 14 + swh/lister/nixguix/tests/__init__.py | 0 .../nixguix/tests/data/guix-swh_sources.json | 19 ++ .../nixguix/tests/data/nixpkgs-swh_sources.json | 52 +++ swh/lister/nixguix/tests/test_lister.py | 244 ++++++++++++++ swh/lister/nixguix/tests/test_tasks.py | 27 ++ swh/lister/tests/test_cli.py | 4 + 13 files changed, 800 insertions(+), 18 deletions(-) create mode 100644 swh/lister/nixguix/__init__.py create mode 100644 swh/lister/nixguix/lister.py create mode 100644 swh/lister/nixguix/tasks.py create mode 100644 swh/lister/nixguix/tests/__init__.py create mode 100644 swh/lister/nixguix/tests/data/guix-swh_sources.json create mode 100644 swh/lister/nixguix/tests/data/nixpkgs-swh_sources.json create mode 100644 swh/lister/nixguix/tests/test_lister.py create mode 100644 swh/lister/nixguix/tests/test_tasks.py
Changes applied before test
commit cdeb7f59ed72ebda661cab55a14556ece8905d64 Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Tue Oct 4 10:57:32 2022 +0200 nixguix: Randomize order of listed origins The end goal is to ingest sparsely the origins, that would avoid hitting the various servers around the same time for colocated origins in the upstream manifest (especially file or tarball). Related to [T3781](https://forge.softwareheritage.org/T3781 'view original for T3781 on Phabricator') commit 94b6dbea0a7f602be0711a3bb1f9bb9e16fc48ce Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Sat Oct 1 16:41:48 2022 +0200 nixguix: Document lister Related to [T3781](https://forge.softwareheritage.org/T3781 'view original for T3781 on Phabricator') commit 6d2e7aa17808e39ba9f493b65d662d0ddef5796c Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Sat Oct 1 16:12:46 2022 +0200 nixguix: Register task Related to [T3781](https://forge.softwareheritage.org/T3781 'view original for T3781 on Phabricator') commit fbfdf88ea4fe79c4846ecd48f2a1322f5d3995fc Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Tue Aug 30 11:17:33 2022 +0200 nixguix: Add lister Related to [T3781](https://forge.softwareheritage.org/T3781 'view original for T3781 on Phabricator')
See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/735/ for more details.
mentioned in merge request !429 (closed)
Some references in the commit message have been migrated:
- T3781 is now swh/meta#3781 (closed)
Build is green
Patch application report for D8605 (id=31087)
Could not rebase; Attempt merge onto fa1205c4...
Updating fa1205c..1b4fe51 Fast-forward requirements-swh.txt | 2 +- setup.py | 1 + swh/lister/__init__.py | 22 ++ swh/lister/gnu/tree.py | 21 +- swh/lister/nixguix/__init__.py | 38 +++ swh/lister/nixguix/lister.py | 373 +++++++++++++++++++++ swh/lister/nixguix/tasks.py | 14 + swh/lister/nixguix/tests/__init__.py | 0 .../nixguix/tests/data/guix-swh_sources.json | 19 ++ .../nixguix/tests/data/nixpkgs-swh_sources.json | 52 +++ swh/lister/nixguix/tests/test_lister.py | 244 ++++++++++++++ swh/lister/nixguix/tests/test_tasks.py | 27 ++ swh/lister/tests/test_cli.py | 4 + 13 files changed, 799 insertions(+), 18 deletions(-) create mode 100644 swh/lister/nixguix/__init__.py create mode 100644 swh/lister/nixguix/lister.py create mode 100644 swh/lister/nixguix/tasks.py create mode 100644 swh/lister/nixguix/tests/__init__.py create mode 100644 swh/lister/nixguix/tests/data/guix-swh_sources.json create mode 100644 swh/lister/nixguix/tests/data/nixpkgs-swh_sources.json create mode 100644 swh/lister/nixguix/tests/test_lister.py create mode 100644 swh/lister/nixguix/tests/test_tasks.py
Changes applied before test
commit 1b4fe51f62c706a9ef77b8eea74e111bb8be3542 Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Tue Oct 4 10:57:32 2022 +0200 nixguix: Randomize order of listed origins The end goal is to ingest sparsely the origins, that would avoid hitting the various servers around the same time for colocated origins in the upstream manifest (especially file or tarball). Related to [T3781](https://forge.softwareheritage.org/T3781 'view original for T3781 on Phabricator') commit 94b6dbea0a7f602be0711a3bb1f9bb9e16fc48ce Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Sat Oct 1 16:41:48 2022 +0200 nixguix: Document lister Related to [T3781](https://forge.softwareheritage.org/T3781 'view original for T3781 on Phabricator') commit 6d2e7aa17808e39ba9f493b65d662d0ddef5796c Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Sat Oct 1 16:12:46 2022 +0200 nixguix: Register task Related to [T3781](https://forge.softwareheritage.org/T3781 'view original for T3781 on Phabricator') commit fbfdf88ea4fe79c4846ecd48f2a1322f5d3995fc Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Tue Aug 30 11:17:33 2022 +0200 nixguix: Add lister Related to [T3781](https://forge.softwareheritage.org/T3781 'view original for T3781 on Phabricator')
See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/736/ for more details.
mentioned in issue swh/meta#3781 (closed)