It fixes some cases where the tag of interest was not fetched.

Related to #4751

Ensure to remove trailing slash in git URL when computing its basename
as an empty string is returned otherwise.

When a shallow fetch failed, typically when the ref is a commit short
hash, retry a full fetch in order for ref checkout to succeed.

Related to #4751.

It has been observed that the process used by SWH to checkout a remote
git reference can lead to different recursive nar hash values compared
to those computed by guix. This seems related to CR/LF normalization.

So prefer to align the process to checkout a remote git reference with
the one used by guix. It seems also faster than the previous approach.

Also refine the not found repository detection process as previously
some non related git errors could be missed.

Related to #4751.

The git directory loader is used to archive guix source packages where
source code is located in a git repository at a specific reference.

To ensure SWH archives the exact same set of source code files for a
guix package, the recursive NAR hash of the source code directory is
computed and compared against the one computed by guix.

Previously the loader was always fetching git submodules if some were
set for the git repository but guix only fetch those for a couple of
packages and not for all git based ones. This could result in directory
hash mismatch when the loader fetches the submodules while it should
have not.

In order to woraround this, first compute the NAR hash without fetching
submodules and if this results in a directory hash mismatch then retry
the operation with the submodules fetched.

Related to #4751.

Without this, some git clones are failing to be ingested because they referenced
submodule which is not initialized. This results in hash mismatch since the git tree
checkouted does not match the upstream nix/guix manifest.

Refs. #4751

Inspired from the pip cloning step [1]. This makes the cloning steps only fetch the
commit information and the tree at the current heads. Then a subsequent
switch (checkout) retrieves the tree at the reference we want. In effect, this retrieves
way faster the necessary tree needed to ingest the repository.

[1] it uses a blobless cloning though.

Refs. swh/meta#3781

Previous commit modified the dumb.check_protocol function to raise an
HTTPError exception when the request to check dumb protocol support
failed. As NotFound exception inherits from ValueError, the code for
checking dumb protocol support was executed even when a repository was
not found. So an HTTPError exception was raised with a 404 status code
and the NotFound exception was no longer propagated to the base loader
class, resulting in a failed visit status instead of a not_found one.

Some network issues can also happen when checking a git repository can
be cloned using the dump protocol so add HTTP retry feature to the
check_protocol function.

Most cli uses the - as separator and not the _

`git_disk` is kept as is because it's old.

This also fixes the git checkout related loader and task inconsistently named.

Refs. swh/infra/sysadm-environment#4906

It enables to use the loader through the following command.

$ swh loader run git_checkout <url> ref=<ref> checkums=<checksums>

The 0.21.4.1 is actually broken.

Refs. swh/meta#4979

Otherwise, we'd lose the context in the snapshot.

Refs. swh/meta#4979

This unifies with other swh import.

This provides the method `fetch_artifact`. It clones a repository at a specific branch,
tag or commit and ingests the DAG objects from the resulting directory tree. It also
checks the Nar checksums if provided.

Refs. swh/meta#4979

Co-Authored: Antoine Lambert <antoine.lambert@inria.fr>

`_parse_message` has a return type annotation since
https://github.com/jelmer/dulwich/commit/a8df40933d5fbde613b6019c6e7eb4606756ea06

Those cases have been observed when trying to clone no longer existing
BitBucket repositories.

Fix #4750.

GitHub API provides for each repository the pack file size in kibibytes
corresponding to a full clone.

As metadata for a GitHub repository are fetched at the beginning of the
loading process (currently only for origins discovered by the github lister),
parse their raw JSON bytes and store pack file size as a loader attribute.
Then, before fetching the pack file for a github origin without any base
snapshot in the archive, check the pack file size is not greater than the
threshold defined by the loader. If it is the case, abort the loading in
order to save some network bandwidth.

Related to #3652

Some network issues can happen when loading a git repository using the
dump protocol so add HTTP retry feature to the GitObjectsFetcher class.

First argument of the exception is not always a string so force its text
representation before searching in it.

Without the space before 'unavailable', it could accidentally match repo names

Git servers other than GitHub can send a slightly different error
message when a repository is no longer available so handle these
cases to ensure NotFound is raised and visit status to be not_found.

Add a statsd counter to report the total number of git objects per type
resolved from the archive but also the total number of objects that could
not be resolved.

Ensure counters are incremented exactly once per git object as the
content of the pack file is iterated four times, one per git object
type.

Ensure to check git objects resolved from the archive by the _resolve_ext_ref
method are not corrupted to enforce a loading will fail when encountering such
edge cases.

It exist cases where a git server can send a pack file containing external
references to objects stored in the local repository. In that case, data
for such objects are not included in the pack file and must be resolved
from the local git object store.

Such a pack file is typically generated when a git client asks for a thin
pack to the server but can also be generated when a client is not requesting
a thin pack.

It has been observed that GitHub can send such pack files to the loader for
origins already visited. The loader does not ask for thin pack files so it
might be an implementation issue of git-upload-pack from their side.

This is problematic as dulwich raises a KeyError exception when it finds
a pack file has external references not resolved in it and thus the git
loading fails.

In order to workaround that issue, implement the resolve_ext_ref callback
that can be passed as parameter to the PackInflater class. When dulwich
encounters an external reference in the pack file, it is calling that
callback to resolve object data. As an external object has already been
stored into the archive, we can reconstruct its git manifest from the
archive content and thus resolve the external reference in the pack file.

Fixes #4745

Related to swh/meta#4959

Related to swh/meta#4960

dulwich >= 0.21 raises an AssertionError when the pack is empty
rather than setting the PackData version and length to None.

So do not try to access the pack file when its size is 0.

Fixes #4747

This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Also include them the proper sphinx way.

Related to T4496

With the proper implementation of packfile negotiation, remotes can
return packfiles that do not contain all of our wanted objects. Consider
the following two histories:

* c1                                      * c1     ← [refs/tags/original]
↑                                         ↑ 
* c2 ← [refs/heads/main]                  |   * c3 ← [refs/heads/main]
                                          * c2     ← [refs/heads/broken]

The first visit of the origin would load commits c1 and c2, and write a
snapshot referencing c2.

During the second visit, the loader would tell the origin that c2 is
known, and that c1 and c3 are wanted (as new heads). The origin, knowing
that c1 is a parent of c2, would be allowed by the git protocol to send
a packfile containing only c3. Under these circumstances, the loader
cannot tell what object type the snapshot branch
[refs/tags/original] should point to.

The repository in tests has a similar structure ([refs/heads/master] is
in the history of [refs/tags/branch2-before-delete]), so refactor the
incremental load test to exercise this specific behavior. This test can
be moved to the common tests as well.

Even though this is only HEAD, we should make sure that it's filtered anyway.

In terms of mypy, this function is just doing some types-washing anyway.

As dulwich's client.fetch_pack expects an instance of history graph
walker with set of known heads, move the local heads caching from
`determine_wants` to the RepoRepresentation initialization logic.

Our previous code would always initialize the graph walker with an empty
set of heads (as the `graph_walker()` method is called before
`determine_wants()` has run, so `self.heads` was always empty), so we
would never actually fetch an incremental pack file.