Add an optional progress callback to `from_disk` method. It can
returns the number of computed entries for each top entries traversed.
This is useful for CLI, in particular to display progress information
for SWH Scanner.

Related to swh/meta#5075.

Right now, the discovery process offered by `filter_known_objects`
returns all results after the discovery is complete. The new callback
provides a way to get information "in real time" which is useful for at
least a couple of planned use case in the SWH scanner:
- displaying progress information while processing
- update a graphical UI in real time.

This simple callback fits this need without too much troubles.

For some reason, mypy complained about the existing type hint in this
file for unclear reason. So I fixed them.

The web Client is no longer using async so we no longer needs it.

When building package documentation outside tox by calling make in the
docs directory, the include of Makefile.sphinx inside the docs Makefile
was failing as its relative path was invalid.

So adapt this relative path according if the SWH_PACKAGE_DOC_TOX_BUILD
environment variable is set or not.

Convert README from markdown to ReST to make it embeddable in
docs/index.rst

instead of a mix-in class.

A future commit will add a method implemented by both with different signatures that mypy cannot unify yet.

`dir_filter` only filters directories. `swh-scanner` needs to
accurately filter out ignored files before making expensive requests
to the web API. We introduce a more general `path_filter` that allows
us to differentiate between files and folders.

`dir_filter` is now deprecated and will be removed once the remaining
users in other packages are migrated over to the new API.

`accept_all_directories` is also deprecated, because it only implies
accepting *directories* even though its behavior also accepts
non-directory entries when used with `path_filter`.

Use a list instead of a tuple to keep mypy happy with latest
hypothesis version.

Antoine Lambert authored 1 year ago and vlorentz committed 1 year ago

As with other fields containing sha1_git values, display hexadecimal
representation of parent revision ids.

This separate package was introduced recently and is needed for our CLIs
to pass type checking.

This allows using the "system" tox, if it's recent enough, instead of
always provisioning an internal .tox venv with tox 4.

Instead of going back to py3, pass through the environment name, so that
it can be called with an arbitrary interpreter version.

When parsing the configuration, tox would complain about the unfollowed
line continuation (which is what happens when the testenv was qualified
with neither full nor minimal).

Moving {posargs} to be unqualified allows the line continuation
character to always have something behind it.

This adds several helper methods returning SWHIDs to model objects,
namely:

- SkippedContent.swhid()
- DirectoryEntry.swhid()
- SnapshotBranch.swhid()
- Release.target_swhid()
- Revision.directory_swhid() and Release.parent_swhids()
- OriginVisitStatus.origin_swhid() and
  OriginVisitStatus.snapshot_swhid()

Better using latest mypy release.

Related to swh/meta#4960

See: https://tox.wiki/en/latest/upgrading.html#updating-usage-with-e

Related to swh/meta#4959

Previously when looking up data by key in an ImmutableDict, the inner
tuple storing keys and values was iterated until finding the requested
key.

This is not really efficient when the ImmutableDict contains a lot of
entries, typically for an origin snapshot containing a lot of branches.

So use an inner dictionary to speedup look up by key operations and
improve loader performances.

This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Also include them the proper sphinx way.

Related to T4496

There were two issues that was preventing to browse some SWHIDs given as
examples in that documentation:

- Some sphinx links were broken in rDMODe1c3fe80731226618616117dfd67a95f3d365645

- A SWHID with ';' in its path qualifier was correctly percent escaped but
  when used as URL argument an extra percent escaping is required as HTTP
  server will unescape URL arguments and thus break SWHID percent escaping.

Closes T4721

from_disk.Content object created for a symlink was missing path info
so ensure to add it for consistency with from_disk.Content object
created for a regular file.

- pre-commit from 4.1.0 to 4.3.0,
- codespell from 2.2.1 to 2.2.2,
- black from 22.3.0 to 22.10.0 and
- flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies.

Also change flake8's repo config to github (the gitlab mirror
being outdated).

When using attr < 21.3.0, adding field transformer breaks attrs
integration with hypothesis, because attributes transformed with
such function are not casted to generated AttrsClass, but remains
just an list of attributes. This causes error in hypothesis by
raising an AttributeError.

As we use attr 21.2.0 in production and when building debian buster
package, add a workaround for that issue as explained here:
https://github.com/python-attrs/attrs/issues/821.

Previously the MerkleNode.collect method was returning a dict whose keys
are node types and values dict of {<node_hash>: <node_data>}.

In order to give more flexibility to client code for the processing of
collected nodes, prefer to simply return a set of MerkleNode.

As a consequence, MerkleNode objects need to be hashable by Python so
the __hash__ method has also been implemented.

Closes T4633

It exists use cases where sha512 checksums need to be computed (content
integrity checks for instances) so add sha512 in the list of hashing
algorithms supported by the MultiHash class.

The pattern were validated from $PWD and later applied on path relative
to `root_path`. So we shuffle a bit of code to test them againt
root_path. We make the absolute pattern relative in the same go.

This code is coming from swh-scanner and should probably get an
overhaul, how ever for now we start with making it no broken.

This reduce the number of function call and should be faster.

The mashup of blind optimisation in the previous changeset yield some
interesting results in total.

It would be insightful to measure them individually, but that would
take more time than we currently have.

When testing all the validator changes on our previous "benchmark" we
see quite interesting improvement.

    swh loader run mercurial https://foss.heptapod.net/mercurial/mercurial-devel directory=/data/repos/mercurial-devel

= Median time of 3 run =
base:   17 minutes 48 seconds
before: 11 minutes 50 seconds
after:  11 minutes 11 seconds

On a profile of the same run, the `to_model` call of the from_disk's `Directory` class took the following percentage:
base:   43%
before: 15%
after:  11%