Related to swh/meta#4959

This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Also include them the proper sphinx way.

Related to T4496

- pre-commit from 4.1.0 to 4.3.0,
- codespell from 2.2.1 to 2.2.2,
- black from 22.3.0 to 22.10.0 and
- flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies.

Also change flake8's repo config to github (the gitlab mirror
being outdated).

Use helper fixture loading_task_creation_for_listed_origin_test from
swh-loader-core and remove redundant tests.

Billiard version 4 is known to cause trouble in the git-loader's test
suite because of specific requirements set by the celery package:

  pkg_resources.ContextualVersionConflict: (billiard 4.0.2
   (/home/stsp/.virtualenvs/swh/lib/python3.9/site-packages),
  Requirement.parse('billiard<4.0,>=3.6.4.0'), {'celery'})

Until that is fixed, pin billiard to known-working versions.

Recent changes in swh-scheduler add new parameters to the celery tasks
produced from swh.scheduler.model.ListedOrigin instances.

So ensure to handle any new parameters by not hardcoding the expected
ones in task signatures.

Remove unsafe use of unnamed task parameters.

Add new tests checking task parameters produced from ListedOrigin
instances do no raise error when attempting to create a hg loader.

Related to T4187

That hook can be frustrating as it can discard a long commit message
if it finds a typo in it so better removing it.

Related to T3922

black is considered stable since release 22.1.0 and the version
we are currently using is quite outdated and not compatible with
click 8.1.0, so it is time to bump it to its latest stable release.

Please note that E501 pycodestyle warning related to line length
is replaced by B950 one from flake8-bugbear as recommended by black.
https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#line-length

Related to T3922

pytest-postgresql 3.1.3 and pytest-redis 2.4.0 added support for
pytest >= 7 so we can now drop the pytest pinning.

Due to test modules being copied in subdirectories of the
build directory by setuptools, it makes pytest fail by raising
ImportPathMismatchError exceptions when invoked from root
directory of the module.

So ignore the build folder to discover tests.

To install the new hook:

  $ pre-commit install -t commit-msg

This used to always output `None`.

Related to T3916

This "clone with timout" util just moved to core, let's use it.
The tests are not relevant anymore, delete them.

This also drops spurious copyright headers to those files if present.

Related to T3812

See inline comments: in essence, currently the first revision
in a run (incremental or not) will get the null revision as a
parent and no cached directory, which creates the wrong diff for that
revision (i.e. lots of files become added etc).

We fix this problem by doing a full listing of the first revision
in a run.

Filtering against the storage then asking for new revisions only makes
sense if there are revisions not in storage and not ancestors of heads.
If there are none, the current behavior lists all revisions, which is a
whole lot of wasted work.

This is no longer required.

Related to T3612

Related to T3612

Keep only the latest mercurial loader. In another commit, that loader will get renamed
appropriately.

Related to T3612

For bookmarks, the ids format listed are not aligned with the rest of the code. It's
human readable id as bytes string instead of bytes string. As it's not what's expected
by the caller, this failed the build.

This commit adds the extra mapping to fix the issue

Related to T3584

anomad-d is the `user`-`repository` name.

It's a repository which presents an anomaly in some undefined way yet. That anomaly
makes the ingestion fail.

It's something that's happening currently once in a while with the bitbucket ingestion.
This commit is just a preparatory work to analyze the problem either if i have time to
or if someone else wants to.

To analyze comment the xfail mark and let the test fail, then debug. That should fairly
help.

Related to T3584

This lift the endpoint adaptations which allow the filtering directly in the
server (storage > v0.37).

Related to T3567

If an uneventful visit happens, then an empty snapshot should happen. When filtering
revisions for speed, still the heads should appear in the snapshot.

Related to T3571

This uses the extid mapping from the storage to actually achieve this.

Related to T3563

For now this hardcodes the version to 1 for either reading or writing instructions.

This allows:
- store the new hashes with a version (actually no version means version 0).
- to keep the old loader mercurial ExtID references in the archives (no need to clean
  them up as that poses other problems regarding the journal)
- in effect unblock the current ingestion/updates of existing origins which already have
  more than one ExtIDs due to different incompatible versions.

The storage implementation does not allow filtering on the extid_version so it's up to
the loader to do the filtering. Hence the current implementation.

Prior to this, depending on the load on jenkins, the test could be flaky and fail for
the wrong reason [1]

[1] https://jenkins.softwareheritage.org/job/DLDHG/job/tests-on-diff/263/console

Since rDMODe09446a6f44b9070ea70a9760bc82dee0bbcb687, it must be an
iterable of bytes.

As discussed in T3352, the branching mechanism of Mercurial is more
featureful than that of Git's. The Snapshot model was not designed
with multiple heads, closed heads, bookmarks, etc. in mind, but with
only branches being "pointers" to (mostly) revisions.
As a workaround for a possible re-design of the Snapshot model (though
nothing of the sort is planned for now), we define a mapping that better
represents Mercurial's branching system.

This allows for handling multiple heads per branch and closed branches,
whose revisions (if not already covered by another branch) would
previously have been lost to the ether. Additionally, bookmarks are now
saved to get a better representation of the projects that do use them.