When loading a sub-path of a repository from a dump file it has been
observed it is less error-prone to dump the whole repository as some
partial dumps fail to be loaded by svnadmin or svnrdump can end up
with error.

In that case, we try to produce a dump file for the whole repository
and we stop to dump revisions once the last one modifying the repository
sub-path was dumped (revisions not modifying sub-path are then filtered
out during the loading process).

The performances when communicating with a remote subversion repository
are much better with the svn protocol than with http(s).

So when a subversion repository URL starts with http(s), check if the
one using svn protocol works and use that URL to communicate with the
remote repository instead.

Some subversion repositories in the wild require non trivial
authentication provider so ensure to use all available providers
when attempting to instantiate the subversion client.

Bump development tools: mypy, codespell, isort, ...

Move all tools configuration in pyproject.toml.

Remove no longer needed mypy overrides.

This regression was introduced in e75d0bd5.

Previous implementation detected some false positive cases which
were leading to hash mismatches when checking revisions.

If an external definition targets an URL whose starts with 'svn+ssh://<user>@',
the checkout or export operation is blocking due to SSH password prompt triggered
by such URL.

So ensure to automatically set the requested password to 'anonymous' to avoid
blocking when attempting to export such external.

The path attribute of a swh.model.from_disk.Directory object no longer
has a trailing slash which makes that test fail.

BaseLoader.load now returns a dict with an extra error field when
a loading fails.

codespell is not happy with thirdparty word and it is better to use
example.com domain for sample URLs.

It enables to specify from which revision to recompute hashes in
order to speedup debugging.

Latest tenacity release adds some internal changes that broke the
mocking of sleep calls in tests.

Fix it by directly mocking time.sleep (was not working previously).

Previously, the state of a directory where the svn:externals property is
unset was partially removed which could lead under certain conditions to
external paths not removed while they should have.

SWH data model allows an origin to have multiple visit types, in particular
a svn origin can have visit types 'svn' and 'svn-export'.

We must ensure to retrieve the latest snapshot for a svn visit type in the
svn loader implementation as it can break incremental loading of a svn origin
having both visit types mentioned above.

Related to swh/meta#5092.

Using command "swh loader run svn-no-dump <svn_url>" enables to load
a subversion repository without producing a dump file.

It is quite helpful for debugging loading issues.

It is required since the release of swh-loader-core v5.18.0.

Pick a free port for the svn server to be executed.

Ensure svn server can accept connections before using the fixture
in tests.

Removing an external path from the reconstructed filesystem can
be tedious due to numerous edge cases where externals are defined
in different directories but some of their paths overlap.

Previous code was not properly handling all cases so rework it
to ensure it does and add a new test with a new edge case.

Since BaseModel now have a object_type attribute, we no longer needs
these.

Everything is a Content now. So we no longer needs this complexity.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

When a repository URL gets scheme redirected, typically from using HTTP
to HTTPS, ensure proper computation of the repository root directory to
avoid loading issues afterwards.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

Some externals can erroneously target the same path and when a commit
fixes that the loader could not restore the content or directory properly
due to the cache used to speedup externals processing, so ensure to
handle that edge case.

When the SvnExportLoader must export multiple subsets of a subversion tree
using the same root svn URL, the same origin URL should not be used in order
to disambiguate what is archived.

So allow to specify the root svn URL in an optional svn_url parameter of the
loader while a different origin URL is provided as first positional parameter
of the loader.

Related to swh/infra/sysadm-environment#5263.

By specifying the sub-paths (directories or files) to export in the optional
svn_paths parameter of the loader, only that subset of paths will be loaded
into the archive.

Related to swh/infra/sysadm-environment#5263.

Remove use of --import-mode=importlib pytest option and use
new option consider_namespace_packages to fix tests execution
with latest pytest release.

Related to swh/devel/swh-loader-core@c9b51f8b.

Related to swh/meta#5075.

A utility function was renamed in swh-loader-core.

The subversion loader now uses Python 3.10 in production so we can
use the key parameter of bisect_right introduced in that release.

It exist some rare cases where a path with externals can be set, then
removed then set again through a copyfrom operation.

Previous implementation did not consider this case and could count the
occurrence of an exernal path multiple times, preventing its removal
when unsetting svn:externals property.

Python 3.7 support is no longer required for loaders so use copytree
function from shutil from now on (as dirs_exist_ok parameter was
added in Python 3.8).

Use the raw external definition to check parsing as test parameters
identifier to get a more meaningful pytest output.

When an external is defined using legacy format (svn < 1.5), the official
subversion client automatically uses the peg_rev parameter of the export
operation so ensure to have the same behavior in the loader to avoid hash
mismatches in reconstructed file systems.