Some archive files do not have any extension but we can guess their
format by getting their mime types.

So add a last resort processing step to get archive format from mime
type prior trying to extract it.

Related to T3468

Rust crate files are standard tar.gz archives so add support
to uncompress them through shutil.register_unpack_format.

For the record, those kind of source files archive are processed
by the nixguix loader.

By default shutil.unpack_archive will try to match the archive format
from its extension but it does not do it in a case insensitive way.
Thus trying to extract a file named "archive.ZIP" will fail.

So try to detect archive format in a case insensitive way by processing
the data returned by shutil.get_unpack_formats prior unpacking.

Nothing should be printed to standard output when calling
swh.core.tarball.uncompress.

Related to T3468

It's needed to properly read from storage.directory_get_entries,
which returns None instead of empty iterators when a directory
is not found (so clients can tell the difference with the empty
directory without hardcoding its hash).

Do not leak internal swh.core.tarball implementation in tests.

Also fix swh.core.tarball.uncompress documentation and implementation,
the function should not return a value.

Python zipfile module does not support the unpacking of ZIP archive
with legacy compression type 6 (implode).

Fallback using the unzip command in that case as it supports the
unpacking of such ZIP file.

Closes T3369

Use lowercase only for requirement names and sort them in
lexicographical order.

Move types-* in requirements-test.txt files as they are not
required for execution.

Remove no longer used requirements-test-db.txt file and associated
testing-db extra in setup.py.

This ensures mypy will not produce errors while being able to run tests
regardless pytest-postgresql version (<3 or >=3).

Two changes affect us:

* pytest_postgresql.factories.get_config was moved to pytest_postgresql.config.get_config
* all db_name keywords and attributes were renamed to dbname

For consistency, I renamed db_name to dbname everywhere, so this will
affect other SWH packages.

When building the global swh documentation, all commands defined
in other swh packages will be insterted in the swh.core cli
documentation but we only want to document the root command here.

This also fixes the remaining sphinx warning when building the global
swh documentation.

3.0 breaks backwards compatibility, we should handle the upgrade in a
separate patch, for now this ensures the rest of the pipelines can keep
running.

The previous implementation would not normalize permissions for
files detected as executable. This would make the loader crash when a
file has bogus permissions such as 0o100100 (executable but not
readable).

This adds a test that all possible file permissions properly normalize
to either 0o100644 or 0o100755.

Enable to check package documentation can be built without producing
sphinx warnings.

The sphinx environment is designed to be used in continuous integration
in order to prevent breaking documentation build when committing changes.

The sphinx-dev environment is designed to be used inside a full swh
development environment.

Related to T3258

Related to T2265

This is needed for the swh-counters module, one backend uses Redis to
manage the counters by themselves, and another one manages the history
files using prometheus.
Each one can possibly evolve separately or can use a different backend
implementation one day or another.

Related to T3165

I don't understand what this library does or why we needed it.
This commit replaces it with the good old functools.wraps(),
which fixes this crash that started occuring with
decorator 5.0.5:

```
ERROR    testapp:app.py:1892 Exception on /foo [POST]
Traceback (most recent call last):
  File "/home/dev/.local/lib/python3.7/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/dev/.local/lib/python3.7/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/dev/.local/lib/python3.7/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/dev/.local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/dev/.local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/dev/.local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/dev/.local/lib/python3.7/site-packages/decorator.py", line 231, in fun
    args, kw = fix(args, kw, sig)
  File "/home/dev/.local/lib/python3.7/site-packages/decorator.py", line 203, in fix
    ba = sig.bind(*args, **kwargs)
  File "/usr/lib/python3.7/inspect.py", line 3015, in bind
    return args[0]._bind(args[1:], kwargs)
  File "/usr/lib/python3.7/inspect.py", line 2930, in _bind
    raise TypeError(msg) from None
TypeError: missing a required argument: 'self'
```

https://github.com/micheles/decorator/issues/109

There were some typos in the factory's documentation,
and the instances were not documented at all.

Ensure tests can be executed using hypothesis >= 6 by suppressing
the function_scoped_fixture health check on test that uses a function
scope fixture in combination with @given that does not need to be reset
between individual hypothesis examples.

swh.storage will need it to specify explicitly migration files to run.

This fixes build [1]

[1] https://jenkins.softwareheritage.org/view/swh%20master/job/DCORE/job/tests/1183/console

It exists cases where a custom exception type cannot be rebuilt from its
serialized form.

As a fallback, build a base Exception with the message of the serialized
exception.

decoding custom format is still supported for backward compat.

Note that this revision modifies the exception type raised when attempting to
encode a naive datetime object for both formats (json amd msgpack): it now
raises a TypeError instead of a ValueError.

to make it clear which codecs are specific to the json format.

instead of json.dumps(..., cls=SWHJSONEncoder) and so.

This simplify a bit the code and make sure the json_dumps/loads
functions are actually tested.

it should not be used anymore.

we should be free of them by now. The journal has been recreated from
scratch since then.

This allows users to pass multiple --log-level values to the swh cli, to allow
finer grained configuration of the logging of some modules, without having to
resort to a full-fledged, very verbose log config file.

Close T2819.

We now match the command line output per section, in a fuzzier way than before.