swh.loader.package: Implement a method to prepare package visit

Implement methods to convert the metadata available for each version of the focused package to a standard format. The metadata of all the versions of a focused package can either be passed by the lister or could be obtained by an API call. fetch_metadata() method can be overridden to get the metadata of the focused package by the appropriate means and convert it into a standard format.

Later, an origin is created by using the metadata provided by the fetch_metadata() method.

Related T1389#31790

Closes T1389

---

Migrated from D1694 (view on Phabricator)

Build is green See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/119/ for more details.

This diff is in its nascent stage, the methods that have # Done as a comment are completed. Rest of them are still in the process. There are also some rogue comments present, those are just for my help during development and will remove them once the code is complete.

Build is green See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/122/ for more details.

Complete prepare method and download.py Note: Many docstrings are out of sync, they don't tell what the method is really doing

Build is green See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/123/ for more details.

D1729 and !369 (closed) are the proposed implementation of GNU and CRAN Loader using the base loader

• Fix docstring and make construct_revision class

Build is green See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/127/ for more details.

The base loader is almost complete with just a few more methods to implement.

Can you please help me with test cases. I am not sure how to proceed with writing test cases.

• swh.loader.base: Improve docstrings

Build is green See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/128/ for more details.

• Make generate_snapshot() method

Build is green See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/129/ for more details.

I run GNU loader(after making some, slight modifications), it ran successfully (status was full).

But when I open the loaded file, it gives a 404 error.

I am not able to understand what could be the problem.

And moreover, there are some problem in configuring the loader(ie it is not able to detect gnu.yml file)

https://forge.softwareheritage.org/source/swh-core/browse/master/swh/core/config.py$342-347 Somehow base_filename is None here. It should have been loader/gnu or loader/cran(depending on the loader used) For testing I hardcoded here as base_filename = 'loader/gnu' and it ran fine. But I am not able to find why base_filename is None in first place

• swh.loader.base: The complete snapshot building process The base loader is complete but there are small problems I am facing in a few steps
• How to get the branch name
• How to find the HEAD for the branch, and there could be some package manager where it is not possible to find head because of lack of metadata, how to deal with those cases

There are some places in the code where I used a bit different approach than what was used in other loaders(like pypi and npm) just to make the process more generalized. (ie so that it does not get hindered by the lack of metadata. I have marked those places as # SEE ME

Build is green See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/130/ for more details.

Thanks for working on it.

From a first look:

• Keep the stuff in swh/loader/core, not swh/loader/base.

• Remove the AbstractAttribute class

• Add docstring to method (explicit rather than implicit)

• Add missing tests

-> the loader-core's tests define a DummyLoader with a basic implementation. Then those check that it behaves accordingly. You should probably do something along the same lines here.

Again, i need to have some time to do a second reading pass.

Cheers,

Merge request was returned for changes

Keep the stuff in swh/loader/core, not swh/loader/base. I choose to keep in in a separate folder just for better handling. I mean there are many files in the core itself, and adding those will move it more cluttered. Can I do this in the end, before landing as it, at it is difficult for me to work on base loader when everything is in a common folder(ie core)?

Remove the AbstractAttribute class Is there any way to guarantee that those variables are declared when making some loader using base loader. Because if, by mistake, those are not mentioned then the code would break.

I choose to keep in in a separate folder just for better handling. So far, the module's name is directly linked to it.

I mean there are many files in the core itself, and adding those will move it more cluttered. Can I do this in the end, before landing as it, at it is difficult for me to work on base loader when everything is in a common folder(ie core)?

Sure.

That actually means that you most probably use the wrong tool for the job. Your editor should help you with that. You should not be bothered by cluttered files...

Is there any way to guarantee that those variables are declared when making some loader using base loader. Because if, by mistake, those are not mentioned then the code would break.

Yes, by adding tests ;)