Removing an external path from the reconstructed filesystem can
be tedious due to numerous edge cases where externals are defined
in different directories but some of their paths overlap.

Previous code was not properly handling all cases so rework it
to ensure it does and add a new test with a new edge case.

Since BaseModel now have a object_type attribute, we no longer needs
these.

Everything is a Content now. So we no longer needs this complexity.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

When a repository URL gets scheme redirected, typically from using HTTP
to HTTPS, ensure proper computation of the repository root directory to
avoid loading issues afterwards.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

Some externals can erroneously target the same path and when a commit
fixes that the loader could not restore the content or directory properly
due to the cache used to speedup externals processing, so ensure to
handle that edge case.

When the SvnExportLoader must export multiple subsets of a subversion tree
using the same root svn URL, the same origin URL should not be used in order
to disambiguate what is archived.

So allow to specify the root svn URL in an optional svn_url parameter of the
loader while a different origin URL is provided as first positional parameter
of the loader.

Related to swh/infra/sysadm-environment#5263.

By specifying the sub-paths (directories or files) to export in the optional
svn_paths parameter of the loader, only that subset of paths will be loaded
into the archive.

Related to swh/infra/sysadm-environment#5263.

Remove use of --import-mode=importlib pytest option and use
new option consider_namespace_packages to fix tests execution
with latest pytest release.

Related to swh-loader-core@c9b51f8b.

Related to swh/meta#5075.

A utility function was renamed in swh-loader-core.

The subversion loader now uses Python 3.10 in production so we can
use the key parameter of bisect_right introduced in that release.

It exist some rare cases where a path with externals can be set, then
removed then set again through a copyfrom operation.

Previous implementation did not consider this case and could count the
occurrence of an exernal path multiple times, preventing its removal
when unsetting svn:externals property.

Python 3.7 support is no longer required for loaders so use copytree
function from shutil from now on (as dirs_exist_ok parameter was
added in Python 3.8).

Use the raw external definition to check parsing as test parameters
identifier to get a more meaningful pytest output.

When an external is defined using legacy format (svn < 1.5), the official
subversion client automatically uses the peg_rev parameter of the export
operation so ensure to have the same behavior in the loader to avoid hash
mismatches in reconstructed file systems.

It simplifies code, improves readability and facilitates the adding
of new data related to a parsed external definition.

When a path is copied using a copyfrom operation, externals set on the paths
being copied must also be set on the copied paths.

Previous implementation was using the latest externals values set on the
paths being copied but those could differ from the ones set at revision
copyfrom_rev so ensure to set correct externals on the copied paths.

Renaming the celery task broke the subversion loading in production as the
load-svn task type is already registered in scheduler database and targets
the previous celery task name.

Related to swh-environment#3925.

Convert README from markdown to ReST to make it embeddable in
docs/index.rst

It was broken in c834321d likely
due to a side effect related to a local configuration.

This is what is actually done on production system (via custom
configured scheduler task type definitions).

This also adds a commit to fix the svn export related task and loader inconsistently
named.

Refs. swh/infra/sysadm-environment#4906

Otherwise, we'd lose the context in the snapshot.

Refs. swh/meta#4979

Refs. swh/meta#4979

Refs. swh/meta#4979

Some remote subversion repositories in the wild require to use anonymous
credentials to obtain a read access. Those credentials can be either
'anonymous/' or 'anonymous/anonymous' in most of the cases.

Previously, it was required to add the credentials in the origin URL using
basic authentication syntax. While such URL can be provided to the Save Code
Now service, it cannot when svn URLs are coming from a lister.

To workaround this, try to connect using these anonymous credentials in the
get_svn_repo function when a connection error happens.

This should also simplify the submission of Save Code Now requests when
submitting a subversion origin that requires anonymous credentials.

Previously when exporting a sub-path of a remote subversion repository
over the network, the full repository was exported and the local path
targeting the sub-path was returned. This is no really optimal in terms
of network bandwidth if the repository filesystem is large but it was
implemented like this to ensure all tests related to sub-paths export
were passing regardless the subversion loader class used: either SvnLoader
or SvnLoaderFromRemoteDump.

After some analysis, it turned out that it was possible to avoid exporting
the full repository but only the request sub-path when using the SvnLoader
class. So modify the SvnRepo class to ensure that behavior and save some
network bandwdith when dealing with a large repository.

These changes in the SvnRepo class induce some in the replay module to ensure
all tests still pass and it also enables to remove a no longer needed optional
parameter to the class constructor.