It can exist cases when a bundle has been cooked and marked as done in
the vault database but is no longer available in the vault cache.

Previously, when requesting a new cooking of such bundle, the vault
backend was not creating a new cooking task and thus the bundle could
not be downloaded anymore.

So ensure to check bundle presence in cache for such edge case and force
its recooking if it is not the case.

Related to swh/devel/swh-web#4804.

We no longer use legacy versions of pytest so we can safely
remove that code.

The oldest part of the scheduler API was updated to use model classes
(based on attr package) instead of dictionaries in order to improve
typing.

We are about to make the git-bare cooker public, so we decided to remove
this override. It means git-fsck will reject the objects, but we could not
find an acceptable alternative.

This is going to happen in swh-storage v3.0.0

Related to swh/meta#5075.

Previously only a couple of them were cherry-picked.

Also add objstorage typing and use swh.objstorage.interface.objid_from_dict
to remove some explicit type casting.

As stated in the official Python documentation, users should generally
prefer to use concurrent.futures which has a simpler interface that was
designed around threads from the start.

Previously when cooking a git bare repository, contents bytes were
fetched sequentially which could take a good amount of time for an
origin with a large revisions history.

In order to speedup the cooking process, retrieve the contents bytes
in parallel with the help of the concurrent.futures module from the
Python standard library which fits particularly well for making loops
of I/O-bound tasks concurrent and for issuing tasks asynchronously.

It enables to push new files to download asynchronously while fetching
sub-directories and thus slightly improve the overall cooking performance.

It should also reduce the memory consumption of the cooking process.

As in the git bare cooker, allow to optionally use objstorage directly
to fetch content bytes when cooking a directory.

Previously when cooking a directory, contents bytes were fetched
sequentially which could take a good amount of time for large
directories.

In order to speedup the cooking process, retrieve the contents bytes
in parallel with the help of the concurrent.futures module from the
Python standard library which fits particularly well for making loops
of I/O-bound tasks concurrent and for issuing tasks asynchronously.

This is required to allow mirrors specify these. A better solution might
be needed in the future, but this should do the job for now.

Try to be nice with sysadmins... Also the code in `backends.py` actually
depends on these names...

So that we can get rid of vault task types being created from swh-schedulers'
sql init scripts.

Due to an invalid test and reference path, the master reference was
missing in the git repository cooked from a SWH release.

The following git-fsck error was reported:

	notice: HEAD points to an unborn branch (master)

It might happen that the scheduler RPC service is failing when attempting
to submit a vault cooking task.

So prefer to insert cooking task info in vault database once the task
successfully created by the scheduler. Previously, the non created
cooking task was inserted in database with a null task id and a new
status which prevented to create a new one once the scheduler service
rpc issue fixed.

Related to swh/infra/sysadm-environment#5055.

This fixes the sdist build warnings about importable packages (swh.vault.sql,
swh.vault.sql.upgrades). [1]

This aligns with the swh.storage's setup.py.

[1] for package in swh.vault.sql swh.vault.sql.upgrades
```
    ############################
    # Package would be ignored #
    ############################
    Python recognizes '$package' as an importable package,
    but it is not listed in the `packages` configuration of setuptools.

    '$package' has been automatically added to the distribution only
    because it may contain data files, but this behavior is likely to change
    in future versions of setuptools (and therefore is considered deprecated).

    Please make sure that '$package' is included as a package by using
    the `packages` configuration field or the proper discovery methods
    (for example by using `find_namespace_packages(...)`/`find_namespace:`
    instead of `find_packages(...)`/`find:`).

    You can read more about "package discovery" and "data files" on setuptools
    documentation page.
```

Refs. swh/infra/sysadm-environment#5044

The symlink representation is slightly ambiguous, so not necessarily
very useful.
Additionally, it made directory-flat bundles differ from a non-recursive
'git clone' checkout, which confuses Guix and potentially other tools.

We may revisit this in the future, eg. by adding a dotfile in the
directory or metadata at the root.

This new method returns a direct download URL for a cooked bundle
if the vault cache backend supports the feature.

The backend implementation simply wraps a call to the download_url
method from the objstorage used as vault cache.

Related to #885.

Since commit swh/devel/swh-core@89d48572, the "swh db init" command attempts
to instantiate the postgresql backend of a swh module using the get_datastore
function of the module with a "cls" and "db" parameters.

While it works fine for most of swh modules, it does not for the vault as more
parameters are expected by the "get_vault" function and thus the database init
is failing. The issue was spotted in the docker environment after updating the
swh/stack image.

So extract postgresql backend initialization for the vault in a new class named
VaultBackendDataStore and set it as swh.vault.get_datastore attribute value.

It now needs types-click which is indeed a dependency of
swh.core[testing].

Latest sentry-sdk release now raises a BadDsn exception when
DSN format does not match the expected one.

That function from swh.core.config was removed in swh/devel/swh-core@2663c0a4.