This is needed by the vault.

Extend the APIs for Revisions and Releases to honor the field by default,
unless the new `ignore_displayname` argument is set.

It was made flaky by d4ddd415.

This opens up the possibility of eventually ignoring the `name` and
`email` fields stored in database in favor of parsing them again from
the fullname field (and therefore to update our parsing logic without
having to affect stored data).

This allows us to populate sensible name and email values out of the new
displayname field, without having to store them.

I don't expect directory_get_raw_manifest to be used, but it is needed for tests,
so why not.

It will be replaced by what is currently called 'offset_bytes'

This only keeps 'offset_bytes' to store the timezone, to support swh-model
v5.0.0.

However, this keeps writing 'offset' and 'negative_utc' to the postgresql
database, just in case we need to roll back this change.
But they are not read anymore.

They should be a rare occurence, so adding these indices allows us to count
and enumerate them without expensive full table scans.

[2022-01-17T16:03:27.448Z] /var/lib/jenkins/workspace/DSTO/tests-on-diff@2/docs/index.rst:25:hardcoded link 'https://archive.softwareheritage.org/api/' could be replaced by an extlink (try using ':swh_web:`api/`' instead)

Assuming all contents passed to content_missing() have (at least) a missing algo,
the function used to iterate over the size of the arg squared
in the worst case (when all contents are found).

With this commit, it starts with bucketing them by hash, so it does not
need to iterate over *all* found contents for each content passed as arg.

This is twice as fast, according to
https://forge.softwareheritage.org/T3577#72791

Instead of grouping ids in queries in arbitrary batches (which forces
the server node to coordinate with other nodes to complete the query),
this sends queries with one id each, directly to the right node.

This is the 'concurrent' algorithm from https://forge.softwareheritage.org/T3577#72791
which gives a >=2x speed-up on directories, and a >=8x speed-up on revisions.

This also drops:
- spurious copyright headers to those files if present.
- fix a type issue revealed by the new mypy

Related to T3812

Using `int()` on `date.timestamp()` rounded it up (toward zero),
but the semantics of `model.Timestamp` is that the actual time is
`ts.seconds + ts.microseconds/1000000`, so all negative dates were
shifted one second up.

In particular, this causes dates from
`1969-12-31T23:59:59.000001` to `1969-12-31T23:59:59.999999`
(inclusive) to smash into dates from
`1970-01-01T00:00:00.000001` to `1970-01-01T00:00:00.999999`,
which is how I discovered the issue.

Now that we have packaged tenacity 6.2 for debian buster and use it
in production, we can remove the workarounds to support tenacity < 5.

Directory entries are now checked for name duplicates in swh-model
so we must ensure the CrashyEntry class is properly initialized.

Closes T3776

The idea is that we check the BaseModel validity at journal
deserialization time so that we still have access to the raw object from
kafka for complete reporting (object id plus raw message from kafka).

This uses a new ModelObjectDeserializer class that is responsible for
deserializing the kafka message (still using kafka_to_value) then
immediately create the BaseModel object from that dict. Its `convert`
method is then passed as `value_deserializer` argument of the
`JournalClient`.

Then, for each deserialized object from kafka, if it's a HashableObject,
check its validity by comparing the computed hash with its id.

If it's invalid, report the error in logs, and if configured, register the
invalid object in via the `reporter` callback.

In the cli code, a `Redis.set()` is used a such a callback (if configured).
So it simply stores invalid objects using the object id a key (typically its
swhid), and the raw kafka message value as value.

Related to T3693.

allowing to use this dict independently of the fix_objects() function.

keep the the fix_objects() function for bw compat for now.

allows to choose replayed object types from the cli.

* merged origin and artifact metadata
* added metametadata
* uses structures instead of dict
* removed raw_extrinsic_metadata_get_latest

It enables to return in an efficient way the list of unique snapshot
identifiers resulting from the visits of an origin.

Previously it was required to query all visits of an origin then query
all visit statuses for each visit to extract such information.

Introduced method enables to extract origin snaphots information in
a single datase query.

Related to T3631

Some revisions in the archive do not have committer date so workaround
it to avoid errors when walking on such revisions when using the class
CommitterDateRevisionsWalker.

content_missing_by_sha1_git only checks the index and not the main table.

This is incorrect, because contents should not be considered written
before an entry is written to the main table, even if an entry
exists in one of the indexes.

This will be done in three steps to avoid any disruption:

1. (this step) add support for decoding the new name, but keep encoding as the old one
2. start encoding as the new name
3. remove support for decoding the old name

The size of individual revisions and releases is essentially unbounded.
This means that, when the buffer storage is used as a way of limiting
memory use for an ingestion process, it is still possible to go beyond
the expected memory use when adding a batch of revisions or releases
with large messages or other metadata.

The duration of the database operations for revision_add or release_add is also
commensurate to the size of the objects added in a batch, so
using the buffer proxy to limit the time individual database operations
takes was not effective.

Adding a threshold on estimated sizes for batches of revision and
release objects makes this overuse of memory and of database transaction
time much less likely.

The size of individual revisions is essentially unbounded. This means
that, when the buffer storage is used as a way of limiting memory use
for an ingestion process, it is still possible to go beyond the expected
memory use when adding a batch of revisions with extensive histories.

The duration of the database operation for revision_add is also
commensurate to the number of revision parents added in a batch, so
using the buffer proxy to limit the time individual database operations
takes was not effective.

Adding a threshold on cumulated number of revision parents per batch
makes this overuse of memory and of database transaction time much less
likely.

The size of individual directories is essentially unbounded. This means
that, when the buffer storage is used as a way of limiting memory use
for an ingestion process, it is still possible to go beyond the expected
memory use when adding a batch of (very) large directories.

The duration of the database operation for directory_add is also
commensurate to the number of directory entries added in a batch, so
using the buffer proxy to limit the time individual database operations
takes was not effective.

Adding a threshold on cumulated number of directory entries per batch
makes this overuse of memory and of database transaction time much less
likely.