This should allow us to move to some more distributed jenkins testing, eventually

The debian build was never actually done.

For some reason, likely due to the fact that it is a hidden one, that parameter
was not correctly transmitted to the job setting gitlab webhooks and the jenkins
jobs could not be triggered anymore.

Nevertheless, the webhooks job also has the same hidden parameter with the rotated
token as default value so we can safely remove the jenkins_token parameter for the
jenkins-jobs-builder job.

Pass it as a hidden parameter instead as jenkins now requires to manually
validate each groovy script for security concerns.

As a script is identified by the hash of its content, proceeding like
this avoid to revalidate the webhooks setting script each time the token
is rotated.

Jenkins docker image bumped debian to bookworm which now forbids
by default to install Python package globally with pip.

Prior to this, we could not build for example the software-stories image (which does not
have requirements.txt).

`git push -f` is not actually about tags so the previous implementation failed.
This now drops explicitely the previous tag prior to push the correct (rebased) tag.

Currently, it can still happen that concurrent build is triggered and docker images for
the same applications are built in parallel. This results in an infinite loop for one of
the docker images. [1]

This tries to overcome that issue by writing early on in the pipeline an empty tag (with
the next version to be built). That way, if concurrent issue arises, the next build
should take the next version.

[1] Possible scenario is swh.core is tagged (or any other highly connected module like
swh.storage) triggers the rebuild of the application swh.loader.git. In the mean time,
swh.loader.git is itself tagged, then possibly the 2 images are built closely enough
that one will get stuck trying to push the exact same tag.

That job started experiencing issues as Python 3.7 reached EOL so
use a more recent Python version (3.10 as in elastic infra) to run
the whole SWH tests suite.

This feature requires the use of the "Throttle Concurrent Builds" Jenkins
plugin and must be explicitely configured in groovy pipelines.

Otherwise, in some edge cases, it could happen a same dependency is needed to be built.
One of those could end up in the infinite loop of not being able to push the same
tag (since one succeeded).

It could be fixed differently but it would require a more convoluted fix. So we'll go
the simple path instead for now.

It happened recently when core and storage needed to be rebuilt and the scrubber
application got trapped in that infinite loop.

The path was wrong.

This delegates this step to the script swh-apps/scripts/git_tag_and_push.sh.

The escaping of \? did not work.

It can happen too many writes happen to the same repository and the current fallback is
too simple an approach to work. So add a loop to try up until it works as this should
finish at some point.

A `--pull` was missing so the base image (python 3.10 mostly)
is never refreshed

I messed up the previous commit

in the fallback push tag instruction. That will avoid further issues like [1] [2]
preventing the main build-docker-images to succeed.

[1]
```
11:08:06  + git tag -a swh-lister-20230704.1 -m swh-lister-20230704.1: Bump new release git push origin master swh-lister-20230704.1
11:08:06  fatal: too many params
```

[2] https://jenkins.softwareheritage.org/job/swh-apps/job/build-docker-image/167/consoleFull

1. It's failing if the remote branch does not exist.
2. The end of the job explicitely pushes the branch in the origin so it's not
needed in the end.

Ref. swh/infra/sysadm-environment#4724

Ref. swh/infra/sysadm-environment#4724

Ref. swh/infra/sysadm-environment#4724

Due to concurrency build [1], it can happen that pushing a new tag fails [2].
So this will try to clean up the first tag, pull and tag again the latest
commit, prior to push when the first try failed.

[1]
https://jenkins.softwareheritage.org/job/swh-apps/job/build-docker-image/147/console

[2]
```
14:22:07  + git pull --rebase
14:22:08  Current branch master is up to date.
14:22:08  + git tag -a swh-loader-git-20230628.2 -m swh-loader-git-20230628.2: Bump new release
14:22:08  + git push origin master swh-loader-git-20230628.2
14:22:08  remote: error: cannot lock ref 'refs/heads/master': is at f2f6f82c0ec231e37f36a3bed24835afd24291b2 but expected e243762e9de668829c7a45bf8b1dd35025593525
14:22:09  To gitlab.softwareheritage.org:swh/infra/swh-apps.git
14:22:09   * [new tag]         swh-loader-git-20230628.2 -> swh-loader-git-20230628.2
14:22:09   ! [remote rejected] master -> master (failed to update ref)
14:22:09  error: failed to push some refs to 'gitlab.softwareheritage.org:swh/infra/swh-apps.git'
```

Ref. swh/infra/sysadm-environment#4724

At its latest commit (so the computation of the next version does not conflict
at push time).

Ref. swh/infra/sysadm-environment#4724

This adds the behavior to update an existing MR.

If no MR exists, then it will be opened. Otherwise, the existing commits will
get rebased on the MR's branch and pushed to that MR's branch.

Ref. swh/infra/sysadm-environment#4724

This also explicits the pull policy.

Prior to this commit, that main instruction failed [1] (with a warning too) in
the job, resulting in an empty MR [2].

[1]
```
11:04:05  + git pull
11:04:05  hint: Pulling without specifying how to reconcile divergent branches is
11:04:05  hint: discouraged. You can squelch this message by running one of the following
11:04:05  hint: commands sometime before your next pull:
11:04:05  hint:
11:04:05  hint:   git config pull.rebase false  # merge (the default strategy)
11:04:05  hint:   git config pull.rebase true   # rebase
11:04:05  hint:   git config pull.ff only       # fast-forward only
11:04:05  hint:
11:04:05  hint: You can replace "git config" with "git config --global" to set a default
11:04:05  hint: preference for all repositories. You can also pass --rebase, --no-rebase,
11:04:05  hint: or --ff-only on the command line to override the configured default per
11:04:05  hint: invocation.
11:04:05  git@gitlab.softwareheritage.org: Permission denied (publickey,keyboard-interactive).
11:04:05  fatal: Could not read from remote repository.
11:04:05
11:04:05  Please make sure you have the correct access rights
11:04:05  and the repository exists.
```

[2] swh-charts!78

Refs. swh/infra/sysadm-environment#4724

It's installed just after the pypi upload stage.

It's not waiting for the end of this stage to finish as nothing is really
waiting for it. So debian packaging build should trigger in parallel. A quiet
period of 10s has been inserted for that new job to let pypi.org some time to
index the new version.

Refs. swh/infra/sysadm-environment#4724

Ref. swh/infra/sysadm-environment#4724

And add the post clean up workspace instructions at the end of the job run.

Ref. swh/infra/sysadm-environment#4724

- The git clone url was wrong with an extra ...:/swh/ instead of ...:swh/.
- The conditional on the "else" dry run was also always executed in case of
  issues which was not the initial intent.
- The actual git push instruction was missing the ssh key context

Ref. swh/infra/sysadm-environment#4724

Prior to this commit, only the local (docker agent) mode was used. It turns
out that on built-in node, the path changes, while acceptable in local mode,
is not done the same way. This unifies such behavior by wrapping instructions
to happen in absolute path names. And now both job implementations are happy.

Ref. swh/infra/sysadm-environment#4724