Convert README from markdown to ReST to make it embeddable in
docs/index.rst

There is no easy way to get that list dynamically as the query language
parser code in C is generated when building the swh-search Python package
and not at runtime.

So prefer to remove that harcoded list of visit types as it must updated
each time a new visit type is added in the archive or searching for such
visit type will fail.

Remove test cases inheritance of unittest.TestCase in order to benefit
from all pytest features like fixtures and parametrize.

Migrate tests setup and teardown to pytest style.

Remove no longer needed hypothesis use.

The way editable installs are handled by setuptools has changed since
release >= 64 and the custom develop command in setup.py is no longer
called. As a consequence, the tree-sitter parser code generation was
no longer executed when performing an editable install and runtime
error was raised when executing the swh-search backend.

So use proper way to handle editable install with latest setuptools and
ensure backward compatibility with older versions.

It now needs types-click which is indeed a dependency of
swh.core[testing].

Flask 2.3 removed the deprecated before_first_request handler.

So ensure to initialize the search backend once after it was
instantiated instead.

Related to swh/meta#4960

Related to swh/meta#4959

This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Also include them the proper sphinx way.

Related to T4496

- pre-commit from 4.1.0 to 4.3.0,
- codespell from 2.2.1 to 2.2.2,
- black from 22.3.0 to 22.10.0 and
- flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies.

Also change flake8's repo config to github (the gitlab mirror
being outdated).

Add visit types for new listers and loaders plus those in development.

This will allow getting the content of documents in ElasticSearch
in order to debug

Apply fix as described in the package build log on unstable:

UserWarning: Distutils was imported before Setuptools, but importing
Setuptools also replaces the `distutils` module in `sys.modules`.
This may lead to undesirable behaviors or errors. To avoid these issues,
avoid using distutils directly, ensure that setuptools is installed in
the traditional way (e.g. not an editable install), and/or make sure that
setuptools is always imported before distutils.

It can be useful to use a remote search server using the memory backend,
for instance in the docker environment to avoid spawning a costly
elasticsearch instance.

As this was never tested, there was an issue when trying to start
such remote search server so fix it and add tests.

Origin visit types must be explicitely declared in order to search
for them, add new ones from loaders in development.

In addition to origin_intrinsic_metadata.

If both sources exist for a given origin, they will be merged by ElasticSearch
as usual.

It will soon be used to store and query extrinsic metadata as well

This causes crashes in future document when the first document with
a 'version' field has a value that Elasticsearch understands as a double.

'document' is a typed dict, which mypy does not seem to allow here

In that case, the search will match all origins and return them in a
paginated way.

It also enables to consider all origins when filtering them according
to different criterions (visit type for instance).

These small changes will enable to improve the search features available
in swh-web.

Without this, the following query fails in the archive [1].

While without specifying anything [2], it works.

The following commit should fix the issue with the [1] link.

[1] https://webapp.staging.swh.network/browse/search/?q=repo1.maven.org&with_visit=true&with_content=true&visit_type=maven

[2] https://webapp.staging.swh.network/browse/search/?q=repo1.maven.org&with_visit=true&with_content=true

elasticsearch 7.17.3 uses JDK >= 18 which changed the value of
"java.security.manager" property to "disallow".

As a consequence a "java.lang.UnsupportedOperationException" is
now raised when bootstrapping elasticsearch server used in tests.

So explicitely set the "java.security.manager" property to "allow"
to avoid that exception raising when elasticsearch setups its custom
security manager.

Closes T4192

That hook can be frustrating as it can discard a long commit message
if it finds a typo in it so better removing it.

Related to T3922

black is considered stable since release 22.1.0 and the version
we are currently using is quite outdated and not compatible with
click 8.1.0, so it is time to bump it to its latest stable release.

Please note that E501 pycodestyle warning related to line length
is replaced by B950 one from flake8-bugbear as recommended by black.
https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#line-length

Related to T3922

pytest-postgresql 3.1.3 and pytest-redis 2.4.0 added support for
pytest >= 7 so we can now drop the pytest pinning.

In addition to test_search.py, which do not include search QL tests

It is caused by a client error; and raising it as 500 prevents the RPC client
from re-raising it using the right class.

Due to test modules being copied in subdirectories of the
build directory by setuptools, it makes pytest fail by raising
ImportPathMismatchError exceptions when invoked from root
directory of the module.

So ignore the build folder to discover tests.

When giving a date (with no time), it captures trailing whitespaces,
so the date in queries like 'last_visit >= 2020-01-01 and ...' is
parsed as '2020-01-01 ', which makes ES error.

Non-visited origins don't have a 'has_visits' field at all, so comparing
it to `false` never returns results.