Stamp files are only useful while building, and not copied to and from S3,
so the check failed after a round-trip through S3.

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Related to T4496

To match actual names (dataset names are unprefixed while Athena DBs are)

Otherwise, UploadToS3, DownloadToS3, and RunAll would conflict with
tasks about to be defined in swh.graph; and Luigi requires task names
to be globally unique.

May be useful later, for traceability.

It allows other packages (eg. swh-graph) to depend on the presence of the local
dataset, with a configurable way to obtain it if missing

They are only useful while exporting the dataset -- after the export is
finished, meta.json is good enough and stamp files only save a couple
of minutes when only some objects types are needed (ie. never in practice)

This will allow running swh-graph tasks easily on machines that didn't
export the graph themselves.

Other tasks will import them in order to depend on tasks defined here

They are more tuned toward running automatically, as they call each
other as needed, and can be imported by workflows defined in other
modules (eg. the future swh.graph.luigi module).

So it can be reused by a Luigi task

For some reason, using a non-existing database works when working with
credentials with unnecessarily high privileges (though it is not clear
to me which permissions allow this).

Order was randomized since 4154d43a

- pre-commit from 4.1.0 to 4.3.0,
- codespell from 2.2.1 to 2.2.2,
- black from 22.3.0 to 22.10.0 and
- flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies.

Also change flake8's repo config to github (the gitlab mirror
being outdated).

* objstorage.add() now requires an id as second argument
* objstorage.get() now allows a dict of hashes instead of just
  sha1, and a future version will require this dict

We are removing support for the objstorage computing the object id itself.

Significantly improves performance by reducing the number of levels in
each DB, and thus reducing the amount of compaction.

Reviewers: #reviewers, olasd

Reviewed By: #reviewers, olasd

Subscribers: olasd

Differential Revision: https://forge.softwareheritage.org/D7711

We no longer support exporting the dataset as PostgreSQL dumps. It's
pretty much useless for big data analysis; we rather encourage
researchers to use big data engine (Hadoop, Hive, Presto...) which all
support the ORC format.

Alternatives to import the dataset on PostgreSQL include
https://github.com/HighgoSoftware/orc_fdw/ , or using the swh mirroring
pipeline to spawn a local storage instance with a PostgreSQL backend.

That hook can be frustrating as it can discard a long commit message
if it finds a typo in it so better removing it.

this option allows to restart consuming kafka a bit earlier than last
committed offsets.
This can be useful to test and debug.

to easily set the list of exported object types. If not set, export all
supported object types.

Note that ``--exclude`` is also applied.