This aligns the behavior of the cli with the other swh modules'
cli (e.g. scheduler, deposit, ...).

Refs. swh/infra/sysadm-environment#5344

There are kafka messages where the 'type' attribute may be missing.
In this case, query the storage for the related OriginVisit to get
the visit type. If it's not available, keep the visit_types empty.

To enable `swh alter removal` to fully remove origins from the archive,
we need to be able to remove origins from ElasticSearch.

As ElasticSearch indexes a document per origin, referenced by the origin
URL, removing references to an origin is as simple as removing the
associated document.

Closes #4652

Fix a "typo" in paths returned by get_output_mapping()...

Add missing links in editable install to parser source files generated
by tree-sitter.

Since the Translator can build the extension on the fly, this should
work ok for strict editable install.

Note that this also fixes the building of the sdist package, including
generated files to build the tree-sitter parser extension.

Thanks to lunar for this solution.

They are usually less interesting than original projects.

Set objects cannot be serialized by the SWH RPC layer so use a
list instead to store visit types as in the elasticsearch backend
in order for the search memory backend to be used without errors
behind a remote proxy.

Closes #4653.

It seems a regression was introduced in latest pytest release in a
way that xunit-style teardown_function is no longer called.

So turn it into an autouse fixture instead to workaround that issue.

Previously, origin_search was only returning a list of dict containing
a single url field. From now on, the list of visit types and a boolean
indicating if an origin was visited by SWH are also included in each
returned origin dict.

The purpose of that change is to enable the webapp to exploit the list
of visit types associated to an origin when searching for origins.

Related to swh-web#4786.

Related to swh/meta#5075.

Convert README from markdown to ReST to make it embeddable in
docs/index.rst

There is no easy way to get that list dynamically as the query language
parser code in C is generated when building the swh-search Python package
and not at runtime.

So prefer to remove that harcoded list of visit types as it must updated
each time a new visit type is added in the archive or searching for such
visit type will fail.

Remove test cases inheritance of unittest.TestCase in order to benefit
from all pytest features like fixtures and parametrize.

Migrate tests setup and teardown to pytest style.

Remove no longer needed hypothesis use.

The way editable installs are handled by setuptools has changed since
release >= 64 and the custom develop command in setup.py is no longer
called. As a consequence, the tree-sitter parser code generation was
no longer executed when performing an editable install and runtime
error was raised when executing the swh-search backend.

So use proper way to handle editable install with latest setuptools and
ensure backward compatibility with older versions.

It now needs types-click which is indeed a dependency of
swh.core[testing].

Flask 2.3 removed the deprecated before_first_request handler.

So ensure to initialize the search backend once after it was
instantiated instead.

Related to swh/meta#4960

Related to swh/meta#4959

This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Also include them the proper sphinx way.

Related to T4496

- pre-commit from 4.1.0 to 4.3.0,
- codespell from 2.2.1 to 2.2.2,
- black from 22.3.0 to 22.10.0 and
- flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies.

Also change flake8's repo config to github (the gitlab mirror
being outdated).

Add visit types for new listers and loaders plus those in development.

This will allow getting the content of documents in ElasticSearch
in order to debug

Apply fix as described in the package build log on unstable:

UserWarning: Distutils was imported before Setuptools, but importing
Setuptools also replaces the `distutils` module in `sys.modules`.
This may lead to undesirable behaviors or errors. To avoid these issues,
avoid using distutils directly, ensure that setuptools is installed in
the traditional way (e.g. not an editable install), and/or make sure that
setuptools is always imported before distutils.

It can be useful to use a remote search server using the memory backend,
for instance in the docker environment to avoid spawning a costly
elasticsearch instance.

As this was never tested, there was an issue when trying to start
such remote search server so fix it and add tests.

Origin visit types must be explicitely declared in order to search
for them, add new ones from loaders in development.

In addition to origin_intrinsic_metadata.

If both sources exist for a given origin, they will be merged by ElasticSearch
as usual.

It will soon be used to store and query extrinsic metadata as well

This causes crashes in future document when the first document with
a 'version' field has a value that Elasticsearch understands as a double.

'document' is a typed dict, which mypy does not seem to allow here

In that case, the search will match all origins and return them in a
paginated way.

It also enables to consider all origins when filtering them according
to different criterions (visit type for instance).

These small changes will enable to improve the search features available
in swh-web.

Without this, the following query fails in the archive [1].

While without specifying anything [2], it works.

The following commit should fix the issue with the [1] link.

[1] https://webapp.staging.swh.network/browse/search/?q=repo1.maven.org&with_visit=true&with_content=true&visit_type=maven

[2] https://webapp.staging.swh.network/browse/search/?q=repo1.maven.org&with_visit=true&with_content=true