BaseLoader.load now returns a dict with an extra error field when
a loading fails.

codespell is not happy with thirdparty word and it is better to use
example.com domain for sample URLs.

It enables to specify from which revision to recompute hashes in
order to speedup debugging.

Latest tenacity release adds some internal changes that broke the
mocking of sleep calls in tests.

Fix it by directly mocking time.sleep (was not working previously).

Previously, the state of a directory where the svn:externals property is
unset was partially removed which could lead under certain conditions to
external paths not removed while they should have.

SWH data model allows an origin to have multiple visit types, in particular
a svn origin can have visit types 'svn' and 'svn-export'.

We must ensure to retrieve the latest snapshot for a svn visit type in the
svn loader implementation as it can break incremental loading of a svn origin
having both visit types mentioned above.

Related to swh/meta#5092.

Using command "swh loader run svn-no-dump <svn_url>" enables to load
a subversion repository without producing a dump file.

It is quite helpful for debugging loading issues.

It is required since the release of swh-loader-core v5.18.0.

Pick a free port for the svn server to be executed.

Ensure svn server can accept connections before using the fixture
in tests.

Removing an external path from the reconstructed filesystem can
be tedious due to numerous edge cases where externals are defined
in different directories but some of their paths overlap.

Previous code was not properly handling all cases so rework it
to ensure it does and add a new test with a new edge case.

Since BaseModel now have a object_type attribute, we no longer needs
these.

Everything is a Content now. So we no longer needs this complexity.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

When a repository URL gets scheme redirected, typically from using HTTP
to HTTPS, ensure proper computation of the repository root directory to
avoid loading issues afterwards.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

Some externals can erroneously target the same path and when a commit
fixes that the loader could not restore the content or directory properly
due to the cache used to speedup externals processing, so ensure to
handle that edge case.

When the SvnExportLoader must export multiple subsets of a subversion tree
using the same root svn URL, the same origin URL should not be used in order
to disambiguate what is archived.

So allow to specify the root svn URL in an optional svn_url parameter of the
loader while a different origin URL is provided as first positional parameter
of the loader.

Related to swh/infra/sysadm-environment#5263.

By specifying the sub-paths (directories or files) to export in the optional
svn_paths parameter of the loader, only that subset of paths will be loaded
into the archive.

Related to swh/infra/sysadm-environment#5263.

Remove use of --import-mode=importlib pytest option and use
new option consider_namespace_packages to fix tests execution
with latest pytest release.

Related to swh-loader-core@c9b51f8b.

Related to swh/meta#5075.

A utility function was renamed in swh-loader-core.

The subversion loader now uses Python 3.10 in production so we can
use the key parameter of bisect_right introduced in that release.

It exist some rare cases where a path with externals can be set, then
removed then set again through a copyfrom operation.

Previous implementation did not consider this case and could count the
occurrence of an exernal path multiple times, preventing its removal
when unsetting svn:externals property.

Python 3.7 support is no longer required for loaders so use copytree
function from shutil from now on (as dirs_exist_ok parameter was
added in Python 3.8).

Use the raw external definition to check parsing as test parameters
identifier to get a more meaningful pytest output.

When an external is defined using legacy format (svn < 1.5), the official
subversion client automatically uses the peg_rev parameter of the export
operation so ensure to have the same behavior in the loader to avoid hash
mismatches in reconstructed file systems.

It simplifies code, improves readability and facilitates the adding
of new data related to a parsed external definition.

When a path is copied using a copyfrom operation, externals set on the paths
being copied must also be set on the copied paths.

Previous implementation was using the latest externals values set on the
paths being copied but those could differ from the ones set at revision
copyfrom_rev so ensure to set correct externals on the copied paths.

Renaming the celery task broke the subversion loading in production as the
load-svn task type is already registered in scheduler database and targets
the previous celery task name.

Related to swh-environment#3925.

Convert README from markdown to ReST to make it embeddable in
docs/index.rst