This job is responsible to setup the webhooks on swh GitLab repositories
enabling to trigger Jenkins builds when pushing changes or creating merge
requests.

It parses Jenkins jobs builder YAML configuration to get the name of the
repos with CI setup and set webhook URLs in GitLab projects through HTTP
requests to the GitLab REST API.

It is triggered at the end of the swh-jenkins-jobs-builder job after jobs
have been reconfigured.

This job is responsible of updating the Jenkins jobs configuration
when executed. It will be triggered when pushing changes to the
swh-jenkins-jobs repository.

When triggered from a GitLab merge request, Jenkins jobs configuration
is not updated, only proper jobs generation is checked.

For each swh package, add a "gitlab builds" job that will be triggered when
pushing commits or creating merge requests on a swh package repository hosted
on a swh GitLab instance.

Those jobs are based on the use of Jenkins integration on the GitLab side
and on the use of the GitLab plugin on the Jenkins side.

Remove use of setup wizard as plugins have already been installed.

This also removes the need for authentication in that sample Jenkins
instance.

Prefer to mount volumes instead of copying docker folder in Dockerfile.

It enables to take modifications of jenkins plugins list configuration,
jenkins job builder configuration or jenkins service entrypoint into
account without having to rebuild the docker image.

Also sort the list of plugins in lexicographical order.

We were using an old version patched by douardda until then.
But the patched features related to phabricator are no longer
used in our job definitions, so better using upstream release
of jenkins-job-builder from now on.

This is needed for the latest swh-lister > v4.3.

Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4684#note_13483

It prevents cypress tests for hanging when the cache is shared.

Remove cy2 hack to avoid race conditions now each cypress runner use
a dedicated folder when running tests.

`swh-docker-dev` and `swh-mirror/run-tests` both uses Docker containers
bound to port 5080. One to test if the generated images work fine, the
other as part of the mirror stack being tested. To prevent issues,
we explicitly use a volatile resource lock in their pipeline
definitions.

See: https://www.jenkins.io/doc/pipeline/steps/lockable-resources/

In order to more easily lock a shared resource in the future, let’s
convert the swh-docker-dev job from a freestyle Jenkins job to
a declarative Pipeline job.

The mirror tests start by build Docker images. Scheduling it at midnight
will make it run at the same time as the job building the main Docker
images. This mean they will fight for access to the Docker daemon and
thus one of them is likely to fail.

Ideally, we would use the lockable resource plugin but as a stopgap
measure, let’s schedule the mirror tests 4 hours after the main image
building job.

As the parallel cypress runners use the same repository filesystem to execute
tests from, some race conditions related to file system operations might happen
which make the tests fail.

As a workaround make each cypress runner execute tests from a copy of the swh-web
repository filesystem (without root hidden folders).

Such copy takes a dozen of seconds but it makes tests execution much more stable
on Jenkins by avoiding the race conditions mentionned above.

swh-mirror tests are quite involved as they deploy a replicated Software
Heritage Archive in the form of a Docker swarm.

This new `swh-mirror/run-tests` job assumes that Jenkins running has
access to a Docker engine. It needs:

1. to be able to build images;
2. to be configured as part of a swarm;
3. to have at least one node with the following labels:

    org.softwareheritage.mirror.volumes.storage-db=true
    org.softwareheritage.mirror.volumes.web-db=true
    org.softwareheritage.mirror.volumes.objstorage=true
    org.softwareheritage.mirror.volumes.redis=true

The pipeline will first create up-to-date Docker images used by the
swarm (using Jenkins host) and then run `pytest` in a Docker container
using `swh-jenkins/tox`.

As it depends on network resources, the test can drag for quite long.
The timeout is currently set at one hour. In most cases the pipeline
will be done in 15-20 minutes.

Care is taken both in the test code and within the Jenkins pipeline to
remove resources related to the Docker swarm (stack, containers, configs
and volumes) on teardown.

The job will need the changes proposed as D8634 to work.

This reverts commit a0fd3dc0 as that fix
does not work since the issue it aimed to fix can still happen when
running cypress tests in parallel on Jenkins.

Sometimes when multiple cypress processes load the same support file
in parallel, a race condition is encountered as file copy operations
are performed. As a result, the content of the support file is not
loaded and cypress tests will fail.

So use a specific copy of the support file for each cypress runner
to mitigate that issue.

Main apache repository is no longer reachable from our infra so we
need to use a mirror to fetch cassandra package now.

The jobs associated to this pipeline are only triggered when commiting
on the master branch or when pushing a phabricator diff based on a
commit of the master branch.

So in order to save time, data transfer and avoid possible clone issues,
fetch only the master branch when cloning as all remote branches are
fetched by default.

Related to T4641

We are only interested in fetching the master branch of each swh repository
to run the checks and tests of that job so pass option --single-branch to
git clone command.

Related to T4641

Recent changes in that build job (running all checks for each swh
package for instance) makes it exceed the 90 minutes timeout so
increase it by 30 minutes.

In order to gain some time when executing Jenkins jobs for swh-web,
execute cypress tests in parallel using 4 runners thanks to the
sorry-cypress tool.

To do so, we use the docker features of Jenkins pipeline.
One sidecar container is created to execute the sorry-cypress server
and parallel tests are executed in another container spawning 4
cypress processes. The communication between the two containers
is done through a dedicated docker network.

Regarding code coverage, we now need to merge the outputs produced
by the 4 runners before generating reports.

This change brings a 2x speedup on overall swh-web build jobs execution
(when a single job is executed, speedup is lesser when multiple jobs
are executed in parallel but still significative).

Also restrict the maximum number of concurrent builds per job to 3
in order to avoid executing too many cypress processes at the same
time as they consume quite a lot of resources on the Jenkins node.

Related to T3423

Add a max_concurrent option to swh pipeline jobs (master, Phabricator diff)
in order to limit the number of concurrent builds per job.

The number of concurrent builds is not limited by default.

Related to T3423

Also execute code checks (flake8, codespell, mypy, ...) for all
swh modules in this Jenkins job.

It will enable to catch possible issues with the tools above and
fix them early.

It enables to upload JavaScript source maps generated by webpack to
help debugging swh-web JavaScript errors in Sentry Web UI.

The Sentry webpack plugin must be used in webpack configuration to
do so.

Currently, the only way to test Jenkins configuration changes using
docker is to push changes to a remote feature branch in swh-jenkins-jobs
repository and make jenkins clone that branch to test the changes.

This is not really convenient to quickly test changes so prefer to
mount the local swh-jenkins-jobs repository as a docker volume
and clone that repository from Jenkins filesystem.

This reverts commit e1dd4f97.

Related to T4135

Related to T4135

Now that static analysis is no longer run in parallel and the number of
mypy checks increased, we can sometimes hit the current 25 minutes timeout
so better bumping it to 30 minutes.

We have a recurring data race in flake8 tasks, probably caused by 'tox -e mypy'
creating and deleting files while flake8 scans them; so we should not run
them at the same time.

While we are at it; also run radon sequentially. flake8 and radon each take less
than 10s, so we should not notice much of a difference.

Using --depth makes `git describe` return nonsensical values, which in
turn makes setuptools-scm generate a v0.1.dev1 version number. This
makes pip unable to resolve version-constrained dependencies on swh
modules, which makes it install them from pip again.

Without it, the error message is not clear from the get go and we need to tamper with
build configuration to actually have access to the error message.

See [1] with the --verbose flag versus [2] without it.

[1]
```
17:56:44  [?25h[34mINFO    [0m Response from https://upload.pypi.org/legacy/:
17:56:44           400 The description failed to render for 'text/x-rst'. See
17:56:44           https://pypi.org/help/#description-content-type for more information.
17:56:44  [34mINFO    [0m <html>
17:56:44            <head>
17:56:44             <title>400 The description failed to render for 'text/x-rst'. See
17:56:44           https://pypi.org/help/#description-content-type for more
17:56:44           information.</title>
17:56:44            </head>
17:56:44            <body>
17:56:44             <h1>400 The description failed to render for 'text/x-rst'. See
17:56:44           https://pypi.org/help/#description-content-type for more
17:56:44           information.</h1>
17:56:44             The server could not comply with the request since it is either
17:56:44           malformed or otherwise incorrect.<br/><br/>
17:56:44           The description failed to render for &#x27;text/x-rst&#x27;. See
17:56:44           https://pypi.org/help/#description-content-type for more information.
17:56:44
17:56:44
17:56:44            </body>
17:56:44           </html>
```
from https://jenkins.softwareheritage.org/job/DLDCVS/job/pypi-upload/12/console

[2]
```
17:44:19  [?25h[33mWARNING [0m Error during upload. Retry with the --verbose option for more details.
```
https://jenkins.softwareheritage.org/job/DLDCVS/job/pypi-upload/11/console