Skip to content

Reuse swh.loader.core's base loader class

Antoine R. Dumont requested to merge generated-differential-D260-source into generated-differential-D260-target

One-step closer on the loaders' api unification.

This uses most of the loader-core's mechanism except for the queue part. I kept the git loader's state logic (which stays consistently hidden in the fetch_data and store_data methods).

What do you think?

Test Plan

Manual tests with both git.loader and git.updater are ok:

$ python3 -m swh.loader.git.loader http://ardumont/dot-files /home/tony/repo/public/dot-files/.git
2017-10-30 19:17:44,029 23390 Creating git origin for http://ardumont/dot-files
2017-10-30 19:17:44,030 23390 Done creating git origin for http://ardumont/dot-files
2017-10-30 19:17:44,031 23390 Creating origin_visit for origin 1 at time 2017-10-30 18:17:44.028937+00:00
2017-10-30 19:17:44,033 23390 Done Creating origin_visit for origin 1 at time 2017-10-30 18:17:44.028937+00:00
2017-10-30 19:17:46,870 23390 Sending 3351 contents
2017-10-30 19:17:47,230 23390 Done sending 3351 contents
2017-10-30 19:17:47,945 23390 Sending 3421 directories
2017-10-30 19:17:49,779 23390 Done sending 3421 directories
2017-10-30 19:17:50,043 23390 Sending 2066 revisions
2017-10-30 19:17:50,329 23390 Done sending 2066 revisions
2017-10-30 19:17:50,335 23390 Sending 18 occurrences
2017-10-30 19:17:50,340 23390 Done sending 18 occurrences
2017-10-30 19:17:50,342 23390 Updating origin_visit for origin 1 with status full
2017-10-30 19:17:50,342 23390 Done updating origin_visit for origin 1 with status full
{'status': 'eventful'}
$ python3 -m swh.loader.git.updater https://github.com/hylang/hy
2017-10-30 19:17:55,738 23494 Creating git origin for https://github.com/hylang/hy
2017-10-30 19:17:55,741 23494 Done creating git origin for https://github.com/hylang/hy
2017-10-30 19:17:55,743 23494 Creating origin_visit for origin 2 at time 2017-10-30 18:17:55.737036+00:00
2017-10-30 19:17:55,746 23494 Done Creating origin_visit for origin 2 at time 2017-10-30 18:17:55.737036+00:00
Counting objects: 15639, done.
Compressing objects: 100% (91/91), done.
Total 15639 (delta 72), reused 84 (delta 45), pack-reused 15503
2017-10-30 19:18:41,058 23494 Listed 773 refs for repo https://github.com/hylang/hy
2017-10-30 19:18:45,981 23494 Sending 4652 contents
2017-10-30 19:18:46,525 23494 Done sending 4652 contents
2017-10-30 19:18:48,013 23494 Sending 8123 directories
2017-10-30 19:18:50,074 23494 Done sending 8123 directories
2017-10-30 19:18:51,135 23494 Sending 2842 revisions
2017-10-30 19:18:51,540 23494 Done sending 2842 revisions
2017-10-30 19:18:52,496 23494 Sending 21 releases
2017-10-30 19:18:52,504 23494 Done sending 21 releases
2017-10-30 19:18:52,507 23494 Sending 773 occurrences
2017-10-30 19:18:52,540 23494 Done sending 773 occurrences
2017-10-30 19:18:52,541 23494 Updating origin_visit for origin 2 with status full
2017-10-30 19:18:52,542 23494 Done updating origin_visit for origin 2 with status full
{'status': 'eventful'}

Migrated from D260 (view on Phabricator)

Merge request reports