This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

Before this commit, UploadExportToS3 and DownloadExportFromS3 assumed the
set of object types was the same as the set of directories, which is wrong:

* for the `edges` format, there is no origin_visit or origin_visit_status
  directory
* for both `edges` and `orc` formats, this was missing relational tables.

A possible fix would have been to use the `swh.dataset.relational.TABLES`
constant and keep ignoring non-existing dirs in the `edges`, but I decided to
simply list directories instead, as it will prevent future issues if we
decide to add directories that do not match any table in Athena for
whatever reason.

Stamp files are only useful while building, and not copied to and from S3,
so the check failed after a round-trip through S3.

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Related to T4496

To match actual names (dataset names are unprefixed while Athena DBs are)

Otherwise, UploadToS3, DownloadToS3, and RunAll would conflict with
tasks about to be defined in swh.graph; and Luigi requires task names
to be globally unique.

May be useful later, for traceability.

It allows other packages (eg. swh-graph) to depend on the presence of the local
dataset, with a configurable way to obtain it if missing

They are only useful while exporting the dataset -- after the export is
finished, meta.json is good enough and stamp files only save a couple
of minutes when only some objects types are needed (ie. never in practice)

This will allow running swh-graph tasks easily on machines that didn't
export the graph themselves.

Other tasks will import them in order to depend on tasks defined here

They are more tuned toward running automatically, as they call each
other as needed, and can be imported by workflows defined in other
modules (eg. the future swh.graph.luigi module).

So it can be reused by a Luigi task

For some reason, using a non-existing database works when working with
credentials with unnecessarily high privileges (though it is not clear
to me which permissions allow this).

Order was randomized since 4154d43a

- pre-commit from 4.1.0 to 4.3.0,
- codespell from 2.2.1 to 2.2.2,
- black from 22.3.0 to 22.10.0 and
- flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies.

Also change flake8's repo config to github (the gitlab mirror
being outdated).

* objstorage.add() now requires an id as second argument
* objstorage.get() now allows a dict of hashes instead of just
  sha1, and a future version will require this dict

We are removing support for the objstorage computing the object id itself.

Significantly improves performance by reducing the number of levels in
each DB, and thus reducing the amount of compaction.

Reviewers: #reviewers, olasd

Reviewed By: #reviewers, olasd

Subscribers: olasd

Differential Revision: https://forge.softwareheritage.org/D7711

We no longer support exporting the dataset as PostgreSQL dumps. It's
pretty much useless for big data analysis; we rather encourage
researchers to use big data engine (Hadoop, Hive, Presto...) which all
support the ORC format.

Alternatives to import the dataset on PostgreSQL include
https://github.com/HighgoSoftware/orc_fdw/ , or using the swh mirroring
pipeline to spawn a local storage instance with a PostgreSQL backend.