by replacing their timestamp with epoch, and relying on the raw_manifest
to store the timestamp instead

When calling the content_get_data method from the storage interface,
ensure to provide all content hashes as parameter to avoid an extra
request on the storage server to fetch missing hashes.

Antoine Lambert authored 3 months ago and Antoine Lambert committed 1 month ago

Previously the dumb git loader was only considering refs targeting
commits and tags but those can also occasionally target blobs and
trees, so ensure to support such refs.

Fixes #4756.

This was used at the time we were building debian packages for
swh components but we no longer do that.

Bump development tools: mypy, codespell, isort, ...

Move all tools configuration in pyproject.toml.

Remove no longer needed mypy overrides.

It can lead to test failures if commit signing is
globally enabled in git configuration.

BaseLoader.load now returns a dict with an extra error field when
a loading fails.

Latest tenacity release adds some internal changes that broke the
mocking of sleep calls in tests.

Fix it by directly mocking time.sleep (was not working previously).

as well as in GitCheckoutLoader.

Previous implementation was building an invalid pack file with REF_DELTA
object types as it was using the new object to deltify as the base of the
delta.

This was leading to errors and undefined behavior after building an index
for such a pack file as the deltified objects could not be properly resolved
by dulwich (observed by stsp while working on git loader improvements).

The bases for deltified objects are now objects that were previously loaded
into the archive.

Tag objects produced in that test are also ensured to be valid.

SWH data model allows an origin to have multiple visit types, in particular
a git origin can have visit types 'git' and 'git-checkout'.

We must ensure to retrieve the latest snapshot for a git visit type in the
git loader implementation as it can break incremental loading of a git origin
having both visit types mentioned above.

Indeed a 'git-checkout' visit type produces a snapshot with a single branch
while a 'git' visit type produces a snapshot containing all branches of the
loaded repository. Previously, if the latest snapshot retrieved was produced
by a 'git-checkout' visit type, the loader would refetch all branches and
associated git objects while most of them have already been archived.

Related to swh/meta#5092.

Side effect of swh.loader.core v5.18.0 release.

Some dumb git servers can send a HEAD file in a legacy format that
contains a commit id instead of the string: "ref: <ref_name>".

So handle that edge case to avoid an error when loading such repository.

As with the smart git loader, restrain the maximum size for a pack file
to download.

Move the code writing pack data bytes and checking size in an utility
class to avoid code duplication.

Add missing tests covering the cases where the pack size limit is reached.

When using the requests library to perform HTTP requests, if responses
need to be streamed the stream parameter must be set to True to ensure
content is downloaded by chunks.

Previously, a whole HTTP response was cached in memory which could lead
to OOM errors when dealing with a repository with large pack files.

Related to swh-loader-core@c9b51f8b.

Related to swh/meta#5075.

Git loading tasks can take a pretty long time, and it's not easy to diagnose if
it's stuck or if it's just taking a while.

Instead of only logging at the end of the task, print a log line after
each object type has been fully processed. Also print a log line every 3
minutes while objects are being processed.

The packfile fetching operation can take a long time. Send one log line
every minute while it progresses.

Instead of dumping the dulwich remote communication stream to stderr,
add a separate logger for remote messages, and handle the remote stream
as proper log entries.

This hooks into the right urllib3 and requests settings for both the
smart and dumb loader.

This sets the connect and read timeout for both the smart loader (via
urllib3/dulwich) and for the dumb loader (via requests).

This is useful to override the default settings of the requests Session,
e.g. certificate verification of connect/read timeouts.

This is useful to override the default settings of the dulwich urllib3
adapter, e.g. certificate verification of connect/read timeouts.

Newer versions of git create a ".rev" file next to the existing ".pack"
and ".idx", making the nb_files inconsistent.

A utility function was renamed in swh-loader-core.

If the submodules parameter of the loader is True but no .gitmodules
file is found in root directory of the repository, the repository
path is not yielded and thus its loading discarded.

It indicates if submodules should be retrieved after the git checkout
operation as some guix origins require it.

Related to #4751.