They are closer semantics as 'html_url' is the main page of the repository,
so it is the best to identify it; and 'clone_url' is the URL that should
be given to 'git clone', as documented by https://schema.org/codeRepository

Additionally, that property was missing so far; but a future commit will
need to use it to identify fork relationships (node ids are required to
representation relationships between documents as we cannot use blank
nodes for that)

They have been moved in a swh-core pytest plugin to share them with
other swh packages that might need it.

It makes rdflib complain, and is invalid anyway

Sentry uses repr() by default, which does not look good in a UI

persist_index_computations deduplicated row entries based on the entire
content of the row; but postgresql enforces the 'id' should be unique.

This was not an issue in older version of swh-indexer, because all
operations were deterministic, given a specific directory as input.

The recent switch to rdflib introduced non-determinism, so different
outputs may be returned for the same directory id; causing the
deduplication to not be good enough to avoid duplicate ids.

With this commit, deduplication is now done on 'id', as expected.

As a side-effect, persist_index_computations is now more efficient
because:

1. it runs in linear time instead of quadratic in the number of
   metadata items
2. it only compares dir ids, instead of the content of indexed metadata
   (which is arbitrarily large JSON-like data)

RawExtrinsicMetadata contain a swh:1:ori: identifier of the origin,
which the indexer needs to resolve, by querying its storage replica.

Because RawExtrinsicMetadata are created by loaders, they are often
created shortly after the origin is created by the corresponding lister,
so the origin may not be known to the storage replica used by the
indexer, causing this function to crash.

Waiting 10s seems to be good enough when run on my computer with
production data and moma's replica; so I set it to 60s just to be safe.

... and they are processed in the same batch.

The last one received takes precedence, as it is likely to be more
up-to-date

They cause noisy logs

Related to T4477

This reverts commit 85b675fd.

Support for these old objects is fixed in swh-model v6.4.1.

Hash dicts are now prefered by swh-objstorage, in order to support
individual hash collisions.

Fix issue [1] which prevents indexer journal client from consuming the journal.

[1] https://sentry.softwareheritage.org/share/issue/1d3de3b47c234408889bff5c4f4b0d20/

Related to T4459#90001

Atom documents from old deposits usually use these instead of codemeta's

Motivation:

1. It makes it easier to visualize what is actually happening when modifying
   the graph, by working explicitly on triples instead of a JSON-LD (a tree
   serialization of the graph).

2. Remove the need for the hacky `merge_values()` function (and possibly
   `merge_documents()` in a future commit)

3. It also catches malformed data exactly where it is added in the document
   (the call to rdflib.Graph.add()) instead of at the end of the mapping
   when running compaction/expansion.

Downsides:

1. Tests are clunkier, because they relied on deterministic order of
   unordered lists; but rdflib does not guarantee it

2. Code is longer

3. Extra dependency (which we will need at some point if we want to
   import from RDF datasets, anyway)

Their order is nondeterministic, it just happens to work with
the way we use PyLD.

This parameter was only used to execute extra code before
`normalize_translation` is called. This caused some duplication, and
will not work when switching to a non-JSON-LD internal representation.

Removing it also makes the code of mappings more consistent, by removing
specific field handling from their implementation of the `translate`
method itself.

They cause postgresql to crash because it does not allow them in text fields.

They are seemingly only present accidentally in source documents;
so stripping them does not really impact the quality of metadata.

When capture_exceptions=False, the indexer's caller reports the exception
to Sentry itself.

However, because tags were added by indexers within a scope internal
to the indexers; the scope was closed before returning to the caller,
so these tags were actually not sent to Sentry.

Without these tags, it is often impossible to find what object
caused a given crash without guesswork based on the object's
content and swh-graph.

Invalid URLs are a common source of crashes