Pass it as a hidden parameter instead as jenkins now requires to manually
validate each groovy script for security concerns.

As a script is identified by the hash of its content, proceeding like
this avoid to revalidate the webhooks setting script each time the token
is rotated.

Jenkins docker image bumped debian to bookworm which now forbids
by default to install Python package globally with pip.

Prior to this, we could not build for example the software-stories image (which does not
have requirements.txt).

`git push -f` is not actually about tags so the previous implementation failed.
This now drops explicitely the previous tag prior to push the correct (rebased) tag.

Currently, it can still happen that concurrent build is triggered and docker images for
the same applications are built in parallel. This results in an infinite loop for one of
the docker images. [1]

This tries to overcome that issue by writing early on in the pipeline an empty tag (with
the next version to be built). That way, if concurrent issue arises, the next build
should take the next version.

[1] Possible scenario is swh.core is tagged (or any other highly connected module like
swh.storage) triggers the rebuild of the application swh.loader.git. In the mean time,
swh.loader.git is itself tagged, then possibly the 2 images are built closely enough
that one will get stuck trying to push the exact same tag.

That job started experiencing issues as Python 3.7 reached EOL so
use a more recent Python version (3.10 as in elastic infra) to run
the whole SWH tests suite.

This feature requires the use of the "Throttle Concurrent Builds" Jenkins
plugin and must be explicitely configured in groovy pipelines.

Otherwise, in some edge cases, it could happen a same dependency is needed to be built.
One of those could end up in the infinite loop of not being able to push the same
tag (since one succeeded).

It could be fixed differently but it would require a more convoluted fix. So we'll go
the simple path instead for now.

It happened recently when core and storage needed to be rebuilt and the scrubber
application got trapped in that infinite loop.

The path was wrong.

This delegates this step to the script swh-apps/scripts/git_tag_and_push.sh.

The escaping of \? did not work.

It can happen too many writes happen to the same repository and the current fallback is
too simple an approach to work. So add a loop to try up until it works as this should
finish at some point.

A `--pull` was missing so the base image (python 3.10 mostly)
is never refreshed

I messed up the previous commit

in the fallback push tag instruction. That will avoid further issues like [1] [2]
preventing the main build-docker-images to succeed.

[1]
```
11:08:06  + git tag -a swh-lister-20230704.1 -m swh-lister-20230704.1: Bump new release git push origin master swh-lister-20230704.1
11:08:06  fatal: too many params
```

[2] https://jenkins.softwareheritage.org/job/swh-apps/job/build-docker-image/167/consoleFull

1. It's failing if the remote branch does not exist.
2. The end of the job explicitely pushes the branch in the origin so it's not
needed in the end.

Ref. swh/infra/sysadm-environment#4724

Ref. swh/infra/sysadm-environment#4724

Ref. swh/infra/sysadm-environment#4724

Due to concurrency build [1], it can happen that pushing a new tag fails [2].
So this will try to clean up the first tag, pull and tag again the latest
commit, prior to push when the first try failed.

[1]
https://jenkins.softwareheritage.org/job/swh-apps/job/build-docker-image/147/console

[2]
```
14:22:07  + git pull --rebase
14:22:08  Current branch master is up to date.
14:22:08  + git tag -a swh-loader-git-20230628.2 -m swh-loader-git-20230628.2: Bump new release
14:22:08  + git push origin master swh-loader-git-20230628.2
14:22:08  remote: error: cannot lock ref 'refs/heads/master': is at f2f6f82c0ec231e37f36a3bed24835afd24291b2 but expected e243762e9de668829c7a45bf8b1dd35025593525
14:22:09  To gitlab.softwareheritage.org:swh/infra/swh-apps.git
14:22:09   * [new tag]         swh-loader-git-20230628.2 -> swh-loader-git-20230628.2
14:22:09   ! [remote rejected] master -> master (failed to update ref)
14:22:09  error: failed to push some refs to 'gitlab.softwareheritage.org:swh/infra/swh-apps.git'
```

Ref. swh/infra/sysadm-environment#4724

At its latest commit (so the computation of the next version does not conflict
at push time).

Ref. swh/infra/sysadm-environment#4724

This adds the behavior to update an existing MR.

If no MR exists, then it will be opened. Otherwise, the existing commits will
get rebased on the MR's branch and pushed to that MR's branch.

Ref. swh/infra/sysadm-environment#4724

This also explicits the pull policy.

Prior to this commit, that main instruction failed [1] (with a warning too) in
the job, resulting in an empty MR [2].

[1]
```
11:04:05  + git pull
11:04:05  hint: Pulling without specifying how to reconcile divergent branches is
11:04:05  hint: discouraged. You can squelch this message by running one of the following
11:04:05  hint: commands sometime before your next pull:
11:04:05  hint:
11:04:05  hint:   git config pull.rebase false  # merge (the default strategy)
11:04:05  hint:   git config pull.rebase true   # rebase
11:04:05  hint:   git config pull.ff only       # fast-forward only
11:04:05  hint:
11:04:05  hint: You can replace "git config" with "git config --global" to set a default
11:04:05  hint: preference for all repositories. You can also pass --rebase, --no-rebase,
11:04:05  hint: or --ff-only on the command line to override the configured default per
11:04:05  hint: invocation.
11:04:05  git@gitlab.softwareheritage.org: Permission denied (publickey,keyboard-interactive).
11:04:05  fatal: Could not read from remote repository.
11:04:05
11:04:05  Please make sure you have the correct access rights
11:04:05  and the repository exists.
```

[2] swh-charts!78

Refs. swh/infra/sysadm-environment#4724

It's installed just after the pypi upload stage.

It's not waiting for the end of this stage to finish as nothing is really
waiting for it. So debian packaging build should trigger in parallel. A quiet
period of 10s has been inserted for that new job to let pypi.org some time to
index the new version.

Refs. swh/infra/sysadm-environment#4724

Ref. swh/infra/sysadm-environment#4724

And add the post clean up workspace instructions at the end of the job run.

Ref. swh/infra/sysadm-environment#4724

- The git clone url was wrong with an extra ...:/swh/ instead of ...:swh/.
- The conditional on the "else" dry run was also always executed in case of
  issues which was not the initial intent.
- The actual git push instruction was missing the ssh key context

Ref. swh/infra/sysadm-environment#4724

Prior to this commit, only the local (docker agent) mode was used. It turns
out that on built-in node, the path changes, while acceptable in local mode,
is not done the same way. This unifies such behavior by wrapping instructions
to happen in absolute path names. And now both job implementations are happy.

Ref. swh/infra/sysadm-environment#4724

This opens a new job build-docker-images to allow impacted docker images to (re)build
after a new release.

For now, the job is triggered manually.

In effect, it compulses the list of impacted images to rebuild (out of module and
version tuple). Then, in parallel (or optionally sequentially), build those docker
images (lifting the existing build-docker-image job).

Finally, in swh-charts, commits a change about the new image versions to use and pushes
as a MR. That MR is to be reviewed by sysadm. They can merge it in staging and check the
behavior is appropriate. Once it is ok, they can merge it in production too. If it's not
ok, they can revert the commit.

The job can be triggered as a dry-run job so no commit nor docker images are pushed in
their respective repository or registry.

Ref. swh/infra/sysadm-environment#4724

This is used to generate distro-independent wheels that can actually be uploaded.

This allows migration to any PEP 517-compliant build backend, in the
future. direct calls to setup.py have been deprecated for a while now.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

The wheel and twine packages are only available for Python 3.11 in the
swh-jenkins/tox docker image so ensure to use that Python version.

(we've not been using it for years at this point)

This introduces a new variable node which allows to either run it on the built-in
node (like actually with the production jenkins) or on the 'docker' agent node (which
should be configured either prior to choose it).

This is actually useful for running in a local jenkins and hacking around those jobs.
This will also allow further developement on similar jobs which requires to run docker.
For example, the future build-docker-images.

This also drops the instruction `updateGitlabCommitStatus` which is not needed in that
job (since that won't be opened in a mr context).

Ref. swh/infra/sysadm-environment#4724

With a potential docker agent.

Some docker jobs can run directly on the host machines but some cannot. The ones
requiring some mount points to be passed along for example. In that case, the docker
agent is preferred.

Ref. swh/infra/sysadm-environment#4724