Ingest crates.io (Rust)
- swh/devel/swh-lister#1424, swh/devel/swh-lister!270 (closed): Implement lister
- swh/devel/swh-loader-core!424 (closed): Implement loader
- #4104: Run in docker
- swh/devel/swh-docs!298 (closed): Document lister
- Document loader
- Deploy on staging
- Call for public review
- Deploy on production
Migrated from T4104 (view on Phabricator)
- Task#4998Extend archive coverage [Roadmap - Collect]
- Task#4999Extend archive coverage [Roadmap - Collect]
- Task#5000Extend archive coverage [Roadmap - Collect]
- swh/devel/swh-docs #4167
Activity
marked this issue as related to swh/devel/swh-lister#1424
marked this issue as related to swh/devel/swh-loader-core#1423
added Archive coverage Crates lister Crates loader priority:Normal labels
marked this issue as related to swh/devel/swh-docs#4167
mentioned in commit swh/devel/swh-lister@985b71e8
mentioned in commit swh/devel/swh-loader-core@a097a946
mentioned in commit swh/devel/swh-loader-core@50bde53d
Hello, The crates lister (stateless) and loader have landed. I just solved some discovered issues while running lister and loader on the Docker env ( swh/devel/swh-loader-core!426 (closed) ).
The lister and the loader runs fine through cli and scheduler. As of today the lister get 86979 crates origins in less than 5 minutes. The loader runs fine too through the scheduler.
What the next steps here? Do you want to move to production or does it need to be incremental first?
What the next steps here?
As per the plan in the description, trigger a run in our staging infra (production-like but not production yet). After that, we'll do a public call for review on the swh-devel mailing list. And if all is fine, that can run in production.
Do you want to move to production or does it need to be incremental first?
I guess you can start working on making the listing incremental if you see a way to do it already.
Is the documentation for that new lister/loader documented already? (I do not recall).
Cheers,
marked the checklist item swh/devel/swh-docs!298 (closed): Document lister as completed
changed milestone to %Extend archive coverage [Roadmap - Collect]
removed Archive coverage label
added #4998 as child task
added #4999 as child task
added #5000 as child task
added ll-status::in-progress label
added octobus_sloan label
mentioned in commit anlambert/swh-loader-core@0c382896
mentioned in merge request swh/devel/swh-loader-core!530 (merged)
mentioned in commit anlambert/swh-loader-core@89659947
added ll:ready-for-staging label and removed ll-status::in-progress label