Skip to content

Stateful Julia lister

Franck Bret requested to merge franckbret/swh-lister:julia-incremental into master

Add a state to the lister to store the "last_commit" of the Julia Registry Toml file each time the lister is executed.

Use Dulwich to retrieve the differences between last_commit and HEAD as new origins.

It's a follow up of !489 (comment 147366)

The incremental lister has been tried in Docker environment and looks to go fine.

[2023-10-12 14:09:44,592: INFO/ForkPoolWorker-1] Task swh.lister.julia.tasks.JuliaListerTask[f5a929d3-f2cc-46a4-bbf1-28c3ad387d7d] succeeded in 169.99034333304735s: {'pages': 1, 'origins': 9837}

[2023-10-12 14:12:58,127: INFO/ForkPoolWorker-1] Task swh.lister.julia.tasks.JuliaListerTask[addab00c-a48b-438f-8898-31ecf7bb46ee] succeeded in 135.97636930301087s: {'pages': 1, 'origins': 0}

[2023-10-13 09:03:23,359: INFO/ForkPoolWorker-1] Task swh.lister.julia.tasks.JuliaListerTask[57727653-2faf-4234-aba7-1ac6182aa311] succeeded in 196.83375675894786s: {'pages': 1, 'origins': 4}

Refs. swh/meta#5065

Edited by Antoine R. Dumont

Merge request reports