It exists cases where a tarball to dowload is marked as gzipped in the
Content-Encoding HTTP response header while in fact it is not.

So handle ContentDecodingError exception that can be raised by the
dowload method: try to download tarball raw bytes again without
attempting to uncompress the input stream.

It enables to append the latest snapshot content of an origin each
time the loader is invoked.

The purpose if to keep track of all the origin artifacts loaded so
far in each new visit of the origin.

Closes T3347

Enable to check package documentation can be built without producing
sphinx warnings.

The sphinx environment is designed to be used in continuous integration
in order to prevent breaking documentation build when committing changes.

The sphinx-dev environment is designed to be used inside a full swh
development environment.

Related to T3258

It enables to successfully invoke make in the docs folder.

It has existed, at least at some point in the past, even though I'm
currently unable to reproduce a dsc with that field in it. (Hence the
lack of test fixture...)

They already write it with raw_extrinsic_metadata_add/extid_add,
and read it with extid_get_*.

This code was only kept for compatibility while we were migrating
the extids. This is now done, so this code is useless.

This contains the basic, from creating files to getting a working
(but untested and inefficient) loader.

Like the PyPI and NPM loader.

It allows scripts to use the method without creating a loader instance.

This allows future runs of a loader to know a package was already
loaded, without querying each of the revisions individually and
parsing their metadata.

Eventually, this will allow us to get rid of the 'metadata' column
on the 'revision' table entirely.

To check which packages are already downloaded.

For now, this lookup is done in addition to checking the artifacts
from the last snapshot's revisions' metadata, because we did not start
writing ExtIDs yet.
But the ExtID lookup will eventually replace the artifact-based lookup.

This will finally allow us to drop the 'metadata' field of Revision
objects.

We used a string instead of a tuple. It doesn't matter much because they
are only compared with each other, but let's not intentionally use
the wrong types when we don't need to.

A future commit will introduce resolve_revision_from_extid, so this commit
preemptively renames it to avoid any confusion.

In a future commit, we will need to go through all the PackageInfo
objects before running the loop, so we can get their ExtID and
fetch them from the storage.

So, we need to fetch them all before running the load loop,
using this listcomp.

These three loaders get intrinsic metadata from the archive, and use it
to build the revision object (mostly authoring and date), which means
they would not load the same revision as an other loader given the
same archive.

This reverts commit 6ae19e51.

I didn't mean to commit it now.

In a future commit, we will need to go through all the PackageInfo
objects before running the loop, so we can get their ExtID and
fetch them from the storage.

So, we need to fetch them all before running the load loop,
using this listcomp.

I found the old definition to be quite confusing when refactoring this code.

This is still a purely internal change for now, but it will be needed
to read/write ExtIDs from/to the storage.

All package loaders but deposit had logic to compute some object
from the new packageinfo, some other objects from the known artifacts,
and compare them.

This commit moves the comparison logic to the base class, and unifies
the two computation interfaces, respectively as an extid() method
on TPackageInfo and a method on the loader.

This unified object for comparison is a byte string,
which is internal to each loader for now, but a future commit
will read and write it from/to the ExtID storage instead of
computing it from the 'original_artifacts' present in
revision metadata.

We want to store these identifiers in the ExtID storage, which expects
a (preferably short) bytearray; but the 'artifact_identity' was a
list of (possibly long) strings and ints.

While this commit does not write them to the ExtID storage yet,
it makes these two loaders use them internally.

Assuming no sha256 collision, this does not change their behavior
when seen from the outside, with two exceptions:

* the list of keys to use is now configured with a template string
* configuring an unknown key now raises a KeyError instead of silently
  using a None value.

But we never use this configuration setting, so in practice there is no
change at all.

We will need it independently in a future commit

We will need it independently in a future commit

We will need it independently in a future commit

Instead of the sha256 + name + ... of all the files of a package.

This will be needed to transition to ExtID, as we can't reasonably write
this large set in the ExtID storage; and the sha256 of the .dsc is good
enough, as the .dsc contains hashes and names of other files.

It's stricter and more readable.

Due to an incomplete command passed to the HelpFormatter.format method,
the text alignment in the usage_prefix variable was not the same as the
command output generated by click.

Those tests started to fail since the recent growth of available loaders
as click wraps help lines to 80 columns.

The main change from using tar in the deposit has been released so we can flip from
archive.zip to archive.tar.

Related to T3094
Related to T3070

Needed to support swh-model 2.0.0.