Docstring is mandatory for a celery task function as its value
is inserted in the required description column of the task_type
table in the scheduler database.

Celery task function name is also used as task name (with underscores
replaced by dashes) so ensure task function names match the task types
registered in production scheduler database.

So that we can get rid of indexer task types being created by
swh-schedulers' sql init scripts.

SingleFileIntrinsicMapping.detect_metadata_files was comparing lowercase
versions of filenames with SingleFileIntrinsicMapping.filename variable
value to detect metadata files.

But as SingleFileIntrinsicMapping.filename holds the canonical name of a
metadata file it can contain uppercase characters.

So ensure to compare lowercase versions of both filenames to avoid metadata
files being undetected.

This fixes indexing of Python intrinsic metadata.

It now needs types-click which is indeed a dependency of
swh.core[testing].

Refs. #4733

Related to swh/meta#4960

Related to swh/meta#4959

vlorentz authored 2 years ago and vlorentz committed 2 years ago

(except QueryCancelled)

also return 503/TransientRemoteException instead of 500/RemoteException,
which should have no effect on clients.

This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Also include them the proper sphinx way.

Related to T4496

1. they are internal to the DB so they do not belong in Kafka
2. on unrelated errors, they cause swh.journal to crash because it does
   not know how to handle integers in the output of unique_key()

The deduplication code assumed `remd.target` matches the id of results,
but this is no longer true, as we started using REMD objects whose
`origin` context was used as result id, when `remd.target` is a
directory (221d48e2).

This avoids having a transaction inserting row A then B, while another
inserts row B then A; which (probably) leads to deadlocks like this:

```
DeadlockDetected: deadlock detected
DETAIL:  Process 1842336 waits for ShareLock on transaction 1051957280; blocked by process 64261.
Process 64261 waits for ShareLock on transaction 1051957281; blocked by process 1842336.
HINT:  See server log for query details.
CONTEXT:  while inserting index tuple (1972253,5) in relation "origin_extrinsic_metadata"
SQL statement "insert into origin_extrinsic_metadata (id, metadata, indexer_configuration_id, from_remd_id, metadata_tsvector, mappings)
```

https://sentry.softwareheritage.org/share/issue/52b06caae89f4235a758887fd6817656/

This was already mitigating by sorting before inserting in temporary
tables, then expecting postgresql to read from temporary tables in the
same order rows where inserted. This is often true, but not guaranteed.

No test for this, because I do not see a way to replicate this more than
existing deadlock tests do.

REMD from deposits target a directory, with an origin in its context,
so this workaround allows indexing deposits easily, without significantly
changing swh-search.

Some snapshots are really large. Rather than fetching them entirely only to
discard most of the branches, this commit only fetches some branches (to
check existence + to use less queries on small snapshots), then requests
specific branches as needed (usually only 2).

This should improve performance and reduce timeout exceptions from the
storage.

This code was flushing kafka messages and waiting for the brokers on
every message, instead of just doing it once per batch.

Codemeta reexports schema:url, schema:dateCreated, ... with
`"@type": "@id"` and `"type": "schema:Date"` so that

```
{
    "@context": "https://doi.org/10.5063/schema/codemeta-2.0",
    "url": "http://example.org",
    "dateCreated": "2022-10-26"
}
```

expands to:

```
{
    "http://schema.org/url": {
        "@type": "@id",
        "@value": "http://example.org"
    },
    "dateCreated": {
        "@type": "http://schema.org/Date",
        "@value": "2022-10-26"
    }
}
```

However, our translation tried to translate directly to a partially expanded
form, like this:

```
{
    "@context": "https://doi.org/10.5063/schema/codemeta-2.0",
    "url": {
        "@value": "http://example.org"
    },
    "dateCreated": {
        "@value": "2022-10-26"
    }
}
```

which prevents the compaction and expansion algorithms from adding a
type themselves, causing the document to be compacted to:

```
{
    "@context": "https://doi.org/10.5063/schema/codemeta-2.0",
    "schema:url": "http://example.org"
    "schema:dateCreated": "2022-10-26"
}
```

or expanded to:

```
{
    "http://schema.org/url": {
        "@value": "http://example.org"
    },
    "http://schema.org/dateCreated": {
        "@value": "2022-10-26"
    }
}
```

which are not what we want.

This commit replaces the hack for `@type` with the right solution that
works for all properties.

This is hopefully the definitive workaround for the PyLD issue.

It will allow invalidating cache after changes to mappings,
without changing the puppet manifest every time.

Without this, some Sentry issues were tagged with the wrong object,
which can be very confusing

- pre-commit from 4.1.0 to 4.3.0,
- codespell from 2.2.1 to 2.2.2,
- black from 22.3.0 to 22.10.0 and
- flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies.

Also change flake8's repo config to github (the gitlab mirror
being outdated).