Close T1206

Summary:
After the conversation in D424, we've decided to move towards a
'releases/<version>' namespace for snapshots generated by the PyPI loader.

Default version pointers now use a branch alias instead of an explicit pointer.

Test Plan: Updated integration tests

Reviewers: zack

Differential Revision: https://forge.softwareheritage.org/D429

Summary:
This would record a partial visit if the loading of an artifact failed
for a reason or another.

In the current implementation, failure to load an artifact would completely halt
the loader, but we may change that in the future.

Depends on D427

Test Plan: Integration tests updated

Reviewers: ardumont

Differential Revision: https://forge.softwareheritage.org/D428

Summary:
The load is eventful if we have loaded new artifacts

Depends on D426

Test Plan: Integration tests have been updated to check for the new status

Reviewers: ardumont

Differential Revision: https://forge.softwareheritage.org/D427

Summary:
The release provided by the PyPI API is the default version that is
presented by the web interface, as well as the one that is used for pip install.
It makes sense to present it as the default branch in the archive.

Depends on D425

Test Plan: The snapshot tests have been updated for the new default branch

Reviewers: ardumont

Differential Revision: https://forge.softwareheritage.org/D426

Summary:
Fairly simple change that allows us to provide a default branch

Depends on D424

Test Plan: unit test added

Reviewers: ardumont

Differential Revision: https://forge.softwareheritage.org/D425

Summary:
This modification is two-fold:
 - For releases with a single sdist, generate a branch named refs/tags/<version>
 - For releases with multiple sdists, generate branches named refs/tags/<ver>/<filename>

Depends on D423

Test Plan: tests have been updated

Reviewers: ardumont

Differential Revision: https://forge.softwareheritage.org/D424

Summary:
This makes sure that snapshots really have the shape we're expecting, rather
than just making sure revisions are pointed at properly

Test Plan: tests have been updated

Reviewers: ardumont

Differential Revision: https://forge.softwareheritage.org/D423

Summary:
This makes sure we account for potential removals of versions from one visit to
the next.

Depends on D421

Test Plan: Updated the "revision unpublished" test case to match the new snapshot

Reviewers: ardumont

Differential Revision: https://forge.softwareheritage.org/D422

Summary: Depends on D420

Test Plan: new unit test added

Reviewers: ardumont

Differential Revision: https://forge.softwareheritage.org/D421

Summary: This better matches the actual functionality of the function

Test Plan: make test has been adapted

Reviewers: ardumont

Differential Revision: https://forge.softwareheritage.org/D420

Antoine R. Dumont authored 6 years ago and Nicolas Dandrimont committed 6 years ago

Summary:
In this scenario, a visit has already taken place for the origin. So,
an existing snapshot exists.

But, this time, the PyPI project has changed and a new release has
been uploaded.

The visit must then result in a new snapshot.

The snapshot shares the same history as prior visit.

Related T421

To be rebased and merged after olasd's diffs (done once).

Test Plan: make test

Reviewers: olasd

Differential Revision: https://forge.softwareheritage.org/D416

Summary:
This allows us to easily apply our common filter ("only process sdists") to all
versions.

Test Plan: trivial change

Reviewers: ardumont

Differential Revision: https://forge.softwareheritage.org/D418

Summary: Depends on D413

Test Plan: formatting fix

Reviewers: ardumont

Differential Revision: https://forge.softwareheritage.org/D414

Summary:
That's needed as we check its contents once per version referenced in the list
of upstream releases.

Test Plan: Running the loader twice on the same module now only loads each artifact once

Differential Revision: https://forge.softwareheritage.org/D413

Summary:
Very helpful for debugging

Depends on D411

Test Plan: python3 -m swh.loader.pypi devpi-common does something useful

Reviewers: ardumont

Differential Revision: https://forge.softwareheritage.org/D412

And add:
- a development section to explain how to make the loader run locally
- terminology section used
- known edge cases

Related T421

Related D410

Related D410

This also checks the tarball size prior to reading the first bytes.

Related T421

Related T421

A release artifact is the file representing a given release

Related T421