Side effect of adding typing to swh.core.utils.grouper.

Bump development tools: mypy, codespell, isort, ...

Move all tools configuration in pyproject.toml.

Remove no longer needed mypy overrides.

BaseLoader.load now returns a dict with an extra error field when
a loading fails.

SWH data model allows an origin to have multiple visit types, in particular
a mercurial origin can have visit types 'hg' and 'hg-checkout'.

We must ensure to retrieve the latest snapshot for a hg visit type in the
mercurial loader implementation as it can break incremental loading of a
mercurial origin having both visit types mentioned above.

Related to swh/meta#5092.

Side effect of swh.loader.core v5.18.0 release.

This should mitigate issues observed in production/staging environment.

Those are mercurial related and should not end up in the directory
archived by that loader.

Related to swh/infra/sysadm-environment#5256.

Related to swh-loader-core@c9b51f8b.

Related to swh/meta#5075.

When requesting to clone a repository at a given revision, the working
directory was not updated to that revision after the clone completes.

A utility function was renamed in swh-loader-core.

Convert README from markdown to ReST to make it embeddable in
docs/index.rst

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

VCS type now needs to be provided to swh.loader.core.nar.Nar constructor.

See swh-loader-core@b6de012f.

We don't import it since 28772169

This also fixes the hg checkout related loader and task inconsistently named (which
prevented them from being scheduled)

Refs. swh/infra/sysadm-environment#4906

Otherwise, we'd lose the context in the snapshot.

Refs. swh/meta#4979

This also unifies with other loaders. Prior to this commit the visit was in status
failed, now it's a not found like any other loaders.

Refs. swh/meta#4979

This provides the class HgDirectoryLoader with only method `fetch_artifact`. It clones a
repository at a specific changeset (or tag) and ingests the DAG objects from the
resulting directory tree. It also checks the checksums (Nar or standard) if provided.

Refs. swh/meta#4979

When a mercurial origin is loaded, SWH stores the mapping between
the original hg revision ids and SWH revision ids as extids.
This mapping is then reused in subsequent loads to determine which
new revisions should be archived.

However it exist cases where previously seen hg revisions were
removed from the mercurial repository between two loads by SWH.

Previously the code filtering such no longer existing revisions
was not catching the right exception and thus an error was raised
when attempting to reload such mercurial origins.

Related to #4583

Related to swh/meta#4960

Related to swh/meta#4959

This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Also include them the proper sphinx way.

Related to T4496

- pre-commit from 4.1.0 to 4.3.0,
- codespell from 2.2.1 to 2.2.2,
- black from 22.3.0 to 22.10.0 and
- flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies.

Also change flake8's repo config to github (the gitlab mirror
being outdated).