vlorentz authored 1 year ago and vlorentz committed 1 year ago

This is going to happen in the near future

This should mitigate issues observed in production/staging environment.

Those are mercurial related and should not end up in the directory
archived by that loader.

Related to swh/infra/sysadm-environment#5256.

Related to swh-loader-core@c9b51f8b.

Related to swh/meta#5075.

When requesting to clone a repository at a given revision, the working
directory was not updated to that revision after the clone completes.

A utility function was renamed in swh-loader-core.

Convert README from markdown to ReST to make it embeddable in
docs/index.rst

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

VCS type now needs to be provided to swh.loader.core.nar.Nar constructor.

See swh-loader-core@b6de012f.

We don't import it since 28772169

This also fixes the hg checkout related loader and task inconsistently named (which
prevented them from being scheduled)

Refs. swh/infra/sysadm-environment#4906

Otherwise, we'd lose the context in the snapshot.

Refs. swh/meta#4979

This also unifies with other loaders. Prior to this commit the visit was in status
failed, now it's a not found like any other loaders.

Refs. swh/meta#4979

This provides the class HgDirectoryLoader with only method `fetch_artifact`. It clones a
repository at a specific changeset (or tag) and ingests the DAG objects from the
resulting directory tree. It also checks the checksums (Nar or standard) if provided.

Refs. swh/meta#4979

When a mercurial origin is loaded, SWH stores the mapping between
the original hg revision ids and SWH revision ids as extids.
This mapping is then reused in subsequent loads to determine which
new revisions should be archived.

However it exist cases where previously seen hg revisions were
removed from the mercurial repository between two loads by SWH.

Previously the code filtering such no longer existing revisions
was not catching the right exception and thus an error was raised
when attempting to reload such mercurial origins.

Related to #4583

Related to swh/meta#4960

Related to swh/meta#4959

This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Also include them the proper sphinx way.

Related to T4496

- pre-commit from 4.1.0 to 4.3.0,
- codespell from 2.2.1 to 2.2.2,
- black from 22.3.0 to 22.10.0 and
- flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies.

Also change flake8's repo config to github (the gitlab mirror
being outdated).

Use helper fixture loading_task_creation_for_listed_origin_test from
swh-loader-core and remove redundant tests.

Billiard version 4 is known to cause trouble in the git-loader's test
suite because of specific requirements set by the celery package:

  pkg_resources.ContextualVersionConflict: (billiard 4.0.2
   (/home/stsp/.virtualenvs/swh/lib/python3.9/site-packages),
  Requirement.parse('billiard<4.0,>=3.6.4.0'), {'celery'})

Until that is fixed, pin billiard to known-working versions.

Recent changes in swh-scheduler add new parameters to the celery tasks
produced from swh.scheduler.model.ListedOrigin instances.

So ensure to handle any new parameters by not hardcoding the expected
ones in task signatures.

Remove unsafe use of unnamed task parameters.

Add new tests checking task parameters produced from ListedOrigin
instances do no raise error when attempting to create a hg loader.

Related to T4187

That hook can be frustrating as it can discard a long commit message
if it finds a typo in it so better removing it.