A change in the base loader will allow to increase it punctually if needed in debugging
mode.

Recent changes in swh-scheduler add new parameters to the celery tasks
produced from swh.scheduler.model.ListedOrigin instances.

So ensure to handle any new parameters by not hardcoding the expected
ones in task signatures.

Remove unsafe use of unnamed task parameters.

Add new tests checking task parameters produced from ListedOrigin
instances do no raise error when attempting to create a cvs loader.

Related to T4187

9474ab23 was a workaround for this role being
unknown to PyPI.

Removing the role fixes the issue. I also do not think it is very useful;
all loaders have this entry point.

Otherwise, the upload-to-pypi build complains about it [1]

[1] https://jenkins.softwareheritage.org/job/DLDCVS/job/pypi-upload/12/console

That hook can be frustrating as it can discard a long commit message
if it finds a typo in it so better removing it.

Fetching CVS repository data using rsync often fails with the following error
(especially with archived repositories hosted on sourceforge):

rsync error: some files/attrs were not transferred (see previous errors) (code 23)

It seems the only way to mitigate that issue is to retry the rsync command
until it succeeds.

So add a rsync_retry decorator and apply it to a new method in the loader
wrapping the call to subprocess.run executing the rsync command.

Also use rsync option to compress file data during the transfer.

Related to T3922

black is considered stable since release 22.1.0 and the version
we are currently using is quite outdated and not compatible with
click 8.1.0, so it is time to bump it to its latest stable release.

Please note that E501 pycodestyle warning related to line length
is replaced by B950 one from flake8-bugbear as recommended by black.
https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#line-length

Related to T3922

pytest-postgresql 3.1.3 and pytest-redis 2.4.0 added support for
pytest >= 7 so we can now drop the pytest pinning.

Due to test modules being copied in subdirectories of the
build directory by setuptools, it makes pytest fail by raising
ImportPathMismatchError exceptions when invoked from root
directory of the module.

So ignore the build folder to discover tests.

And use f-string when raising

To install the new hook:

  $ pre-commit install -t commit-msg

Related to T3916

Related to T3788

The CVS loader creates files on the local file system based on
paths which were read from a local copy of a CVS repository or
sent by a CVS server as part of its "cvs rlog" response.

Ensure that such paths will not be able to escape the temporary
directory which stores checked out versions of files.

This also drops spurious copyright headers to those files if present.

Related to T3812

The existing code was probably made out of the svn loader and got never changed. This
drops the inexistant parameters and keep only the one needed.

This also adds coverage to the module.

Related to T3788

Our expansion of the Log keyword was slightly wrong. We need to
trim trailing whitespace from the "prefix" line content which
preceeds the Log keyword when we write out line content which
followed the Log keyword. Update the Log expansion example given
in a comment to document this (see there for details; this behaviour
of CVS is hard to explain without illustration).

Found while testing conversion of the OpenBSD CVS repository.
Add a new test which uses an RCS file from this repository to
reproduce this problem.

CVS supports the definition of custom keywords. A common use case
for custom keywords is to use the project name as a keyword. This
avoids confusion when files are copied between projects using CVS,
in case files contain a keyword that is in use by both projects.
In other words, a file will retain its expanded custom keyword from
project A, allowing to trace the initial file version back to its
origin, after the file was copied into project B's CVS repository.

This feature is in active use by OpenBSD and NetBSD, for example.
Existing conversions of their CVS repositories to Git expand
the corresponding custom keywords as well, and so should we.
Historically, X11 and FreeBSD were also using custom keywords.

During conversion via rsync:// we copy the CVSROOT directory and the
desired CVS module from the rsync server. The file CVSROOT/config
contains directives which configure the use of custom keywords.
Parse this file and expand keywords accordingly when checking out
versions of files from our local copy of the CVS repository.

For now, we only support custom keywords which correspond to the
Id keyword since this is known to be in common use by projects.
The latest releases of CVS (1.12.x) have optional support for arbitrary
keyword aliases via custom keywords. Support for this could be added
later, should there be a need to do so. In any case, the pserver access
method already supports arbitrary custom keywords because such keywords
will be expanded by the CVS server when we check out files from it.

While here, optimize our use of rsync a bit.
Fetch only CVSROOT and the desired CVS module over rsync, rather
than fetching the entire CVS repository directory, which may contain
unrelated CVS modules that require disk space but will not be used.

CVS modules were imported with the a top-level directory which
matched the module name. For a CVS origin such as
rsync://cvs.savannah.gnu.org/sources/dino/dino
the top-level directory contained a single directory called "dino"
with all expected files and directories residing inside this directory.
E.g. the dino project's top-level README file would be stored at
the path "dino/README" instead of just "/README".

Import project files directly into the top-level directory, as expected.
Adjust test expectations accordingly.

Mention that cvs is a required dependency for running the tests.

Document that some protocol schemes are not fully covered by
the test suite (as suggested by vlorentz in D6678).

The CVS loader used to create one snapshot per loaded revision.
As pointed out by ardumont in D6745, this is wrong; Other loaders
create only one snapshot per visit.
Fix this issue and adjust tests expectations accordingly.

While here, show SHW IDs of loaded revisions and snapshots in regular
"info" log output, rather than only in "debug" log output. Previously,
only CVS-related data was shown at the "info" log level. Showing both
CVs and SWH data in log output is more informative.

Align our expansion of Log keywords with the behaviour of a real
CVS server. With this, such keywords expand the same way over
the pserver and rsync access methods.

This is the last change required to consistently ingest CVS's own
CVS repository over both pserver and rsync. Otherwise we get commit
hash mis-matches due to differently expanded Log keywords.

Summary: Suggested by ardumont in D6566

Reviewers: #reviewers, vlorentz

Reviewed By: #reviewers, vlorentz

Differential Revision: https://forge.softwareheritage.org/D6585

The function RcsKeywords.expand_keyword() is used to expand keywords
when fetching an origin over rsync. This function failed to process
multiple keywords on a single line, even though the existing code
already keeps looping in an attempt to expand multiple keywords.

For example, consider this line from a file in the ccvs CVS repository:

  #ident	"@(#)cvs/contrib/pcl-cvs:$Name:  $Id$"

Here, a regular CVS server expands both keywords on this line.

The Name keyword is special; It expands only if an explicit tag name was
given on the CVS command line. This keyword always expands to an empty
string for now, until perhaps one day the CVS loader learns about tags.

Our regular expression which attempts to match keywords on a line splits
the above example into two match groups:

  1: #ident	"@(#)cvs/contrib/pcl-cvs:$Name:  $
  2: Id$

The Name keyword was then expanded as expected, but the Id keyword was missed.
To fix this, attempt another match starting from the terminating character of
the previous match, such that we match the following two strings:

  1: #ident	"@(#)cvs/contrib/pcl-cvs:$Name:  $
  2: $Id$

Now our CVS loader expands both keywords like the CVS server does.
Add new test data to confirm that this works as intended.