As stated in the official Python documentation, users should generally
prefer to use concurrent.futures which has a simpler interface that was
designed around threads from the start.

Previously when cooking a git bare repository, contents bytes were
fetched sequentially which could take a good amount of time for an
origin with a large revisions history.

In order to speedup the cooking process, retrieve the contents bytes
in parallel with the help of the concurrent.futures module from the
Python standard library which fits particularly well for making loops
of I/O-bound tasks concurrent and for issuing tasks asynchronously.

It enables to push new files to download asynchronously while fetching
sub-directories and thus slightly improve the overall cooking performance.

It should also reduce the memory consumption of the cooking process.

As in the git bare cooker, allow to optionally use objstorage directly
to fetch content bytes when cooking a directory.

Previously when cooking a directory, contents bytes were fetched
sequentially which could take a good amount of time for large
directories.

In order to speedup the cooking process, retrieve the contents bytes
in parallel with the help of the concurrent.futures module from the
Python standard library which fits particularly well for making loops
of I/O-bound tasks concurrent and for issuing tasks asynchronously.

This is required to allow mirrors specify these. A better solution might
be needed in the future, but this should do the job for now.

Try to be nice with sysadmins... Also the code in `backends.py` actually
depends on these names...

So that we can get rid of vault task types being created from swh-schedulers'
sql init scripts.

Due to an invalid test and reference path, the master reference was
missing in the git repository cooked from a SWH release.

The following git-fsck error was reported:

	notice: HEAD points to an unborn branch (master)

It might happen that the scheduler RPC service is failing when attempting
to submit a vault cooking task.

So prefer to insert cooking task info in vault database once the task
successfully created by the scheduler. Previously, the non created
cooking task was inserted in database with a null task id and a new
status which prevented to create a new one once the scheduler service
rpc issue fixed.

Related to swh/infra/sysadm-environment#5055.

This fixes the sdist build warnings about importable packages (swh.vault.sql,
swh.vault.sql.upgrades). [1]

This aligns with the swh.storage's setup.py.

[1] for package in swh.vault.sql swh.vault.sql.upgrades
```
    ############################
    # Package would be ignored #
    ############################
    Python recognizes '$package' as an importable package,
    but it is not listed in the `packages` configuration of setuptools.

    '$package' has been automatically added to the distribution only
    because it may contain data files, but this behavior is likely to change
    in future versions of setuptools (and therefore is considered deprecated).

    Please make sure that '$package' is included as a package by using
    the `packages` configuration field or the proper discovery methods
    (for example by using `find_namespace_packages(...)`/`find_namespace:`
    instead of `find_packages(...)`/`find:`).

    You can read more about "package discovery" and "data files" on setuptools
    documentation page.
```

Refs. swh/infra/sysadm-environment#5044

The symlink representation is slightly ambiguous, so not necessarily
very useful.
Additionally, it made directory-flat bundles differ from a non-recursive
'git clone' checkout, which confuses Guix and potentially other tools.

We may revisit this in the future, eg. by adding a dotfile in the
directory or metadata at the root.

This new method returns a direct download URL for a cooked bundle
if the vault cache backend supports the feature.

The backend implementation simply wraps a call to the download_url
method from the objstorage used as vault cache.

Related to #885.

Since commit swh/devel/swh-core@89d48572, the "swh db init" command attempts
to instantiate the postgresql backend of a swh module using the get_datastore
function of the module with a "cls" and "db" parameters.

While it works fine for most of swh modules, it does not for the vault as more
parameters are expected by the "get_vault" function and thus the database init
is failing. The issue was spotted in the docker environment after updating the
swh/stack image.

So extract postgresql backend initialization for the vault in a new class named
VaultBackendDataStore and set it as swh.vault.get_datastore attribute value.

It now needs types-click which is indeed a dependency of
swh.core[testing].

Latest sentry-sdk release now raises a BadDsn exception when
DSN format does not match the expected one.

That function from swh.core.config was removed in swh/devel/swh-core@2663c0a4.

GitLab will display the content of the README file when browsing the
repository. But in case the file is a symlink, it will display the
path pointed by the symlink. There is a 6 year old issue about this:
https://gitlab.com/gitlab-org/gitlab/-/issues/15093

We can workaround the issue by having the content at the root of the
repository and a symlink to this file in the `docs/` directory.

Tested in swh/devel/swh-py-template!27

Related to swh/meta#4960

Related to swh/meta#4959

This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Also include them the proper sphinx way.

Related to T4496

Previous URL was starting with /api/1/vault/git_bare/ leading to a 404
while the correct one must start with /api/1/vault/git-bare/.

- pre-commit from 4.1.0 to 4.3.0,
- codespell from 2.2.1 to 2.2.2,
- black from 22.3.0 to 22.10.0 and
- flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies.

Also change flake8's repo config to github (the gitlab mirror
being outdated).

to more reasonable versions...

instead of creating it in the VaultBackend constructor: when configured
(with host and port), SMTP.connect() is immediately called, which makes
it mandatory to have the smtp server up and running to be able to create
the VaultBackend object (which makes it hard to run properly in an elastic
environment like docker or k8s).

This also removes the fallback to hardcoded 'localhost:25' smtp server;
if the smtp server is not configured or not reachable, the call to
_smtp_send() will fail logging the failure (both in logging and sentry).

Hash dicts are now prefered by swh-objstorage, in order to support
individual hash collisions.

because this later is not yet available on debian, and we do not use new
API features anyway in test_cookers.py

Should fix deb packaging.