- add the `get_datastore` function in `swh.indexer`
- add the `get_current_version` method in `IndexerBackend`,
- remove dbversion management from sql init script
- update tests accordingly

To install the new hook:

  $ pre-commit install -t commit-msg

Related to T3916

A future release of swh-model will change its constructor's signature (replace 'offset'
and 'negative_utc_offset' with 'offset_bytes).

Related to T3815

This also:
- drops spurious copyright headers to those files if present.
- drops a no longer relevant type ignore
- add another type ignore (to fix later)

Related to T3812

This fixes error instances [1]

[1] https://sentry.softwareheritage.org/share/issue/a5869f724a244fdab2e1cb0303a12fc1/

This fixes occurrences of [1]

[1] https://sentry.softwareheritage.org/share/issue/4b92519efe2a4ddd9455f0d600a614b9/

It renamed db_name to dbname, which is a breaking change.

Enable to check package documentation can be built without producing
sphinx warnings.

The sphinx environment is designed to be used in continuous integration
in order to prevent breaking documentation build when committing changes.

The sphinx-dev environment is designed to be used inside a full swh
development environment.

Related to T3258

The implementation is based on output from cff-converter python library.
Deviation has been made for license and organisation.legalName(replaced with organisation.name)

codemeta dir was updated last time in Jan 2019. In this commit
crosswalk.csv, CITATION and codemeta.jsonld have been updated.

Add testcase to detect CODEMETA.json

Simpler code and less error-prone.

self._index_contents was called multiple times in a loop with the same arguments,
except for the set of hashes to exclude.

It means that, if there were N pages of hashes to exclude, each content was
indexed N times; and the N-1 first iterations didn't even exclude all the
hashes they had to exclude.

The reason for this is to avoid surprises like the indexer journal client stuck
in limbo for a while.

Related to T2821
Related to T2814

Related to D4638

This detected some paper cuts within cli tests for example.

The main goal is to decrease friction when actually deploying indexer related
services (backend, indexers, ...).

The pg backends tests should still be reasonably fast as it's using the
swh.core.db.pytest_plugin (which truncate tables in between tests).

Related to T2821

According to the value_sanitizer docstring, this takes 2 parameters, first is
the object type, the second is the actual dict value to sanitize. As a somewhat
default identity function, this discards the object type and returns directly
the dict value unchanged.

[1] https://forge.softwareheritage.org/source/swh-journal/browse/master/swh/journal/writer/kafka.py$97-100

It will become mandatory in a future version.

This always happens when writing to Kafka, as the Kafka writer sets it
to None at the same it it injects the 'tool' data.

This was not caught by tests because they use the in-mem writer;
which did not call unique_key() at all in swh-journal<=v0.5.1
(but future versions will).

This fixes the indexer debian package build.

It's simplifying reading and more consistent with other similar tests

... instead of OriginVisit.

OriginVisit model object no longer hold status information so the current
filtering happening on the journal client side could not work.

Related to T2814
Related to P882

The minimum configuration is provided either by the --config-file
or the --broker parameters

Related to T2814

swh.storage and swh.objstorage, as well as swh.indexer itself, have deprecated
using an explicit `args` in their factories for a while; we can drop them now.

vcversioner was already removed months ago.

test_origin_intrinsic_metadata_add__deadlock: use more values, to make the test less likely to unexpectedly pass.

postgresql kindly returns the results in the order the test expected... most of the time.

Commits in the last month made these endpoints more consistent with
the other ones (though not completely), so these tests don't need to
be skipped anymore.

By removing the False behavior, which we didn't use in practice,
and was removed from the indexers in the previous commit.

The main motivation is to make _add endpoints write to Kafka in a
future commit, as Kafka's semantics are closer to conflict_update=True
than conflict_update=False.