- Jan 03, 2023
-
-
Antoine Lambert authored
This job is responsible of updating the Jenkins jobs configuration when executed. It will be triggered when pushing changes to the swh-jenkins-jobs repository. When triggered from a GitLab merge request, Jenkins jobs configuration is not updated, only proper jobs generation is checked.
-
Antoine Lambert authored
For each swh package, add a "gitlab builds" job that will be triggered when pushing commits or creating merge requests on a swh package repository hosted on a swh GitLab instance. Those jobs are based on the use of Jenkins integration on the GitLab side and on the use of the GitLab plugin on the Jenkins side.
-
Antoine Lambert authored
Remove use of setup wizard as plugins have already been installed. This also removes the need for authentication in that sample Jenkins instance.
-
Antoine Lambert authored
Prefer to mount volumes instead of copying docker folder in Dockerfile. It enables to take modifications of jenkins plugins list configuration, jenkins job builder configuration or jenkins service entrypoint into account without having to rebuild the docker image.
-
Antoine Lambert authored
Also sort the list of plugins in lexicographical order.
-
Antoine Lambert authored
We were using an old version patched by douardda until then. But the patched features related to phabricator are no longer used in our job definitions, so better using upstream release of jenkins-job-builder from now on.
-
- Jan 02, 2023
-
-
Antoine R. Dumont authored
This is needed for the latest swh-lister > v4.3. Related to https://gitlab.softwareheritage.org/infra/sysadm-environment/-/issues/4684#note_13483
-
- Nov 28, 2022
-
-
Antoine Lambert authored
It prevents cypress tests for hanging when the cache is shared.
-
- Nov 21, 2022
-
-
Antoine Lambert authored
Remove cy2 hack to avoid race conditions now each cypress runner use a dedicated folder when running tests.
-
- Nov 14, 2022
-
-
Jérémy Bobbio (Lunar) authored
`swh-docker-dev` and `swh-mirror/run-tests` both uses Docker containers bound to port 5080. One to test if the generated images work fine, the other as part of the mirror stack being tested. To prevent issues, we explicitly use a volatile resource lock in their pipeline definitions. See: https://www.jenkins.io/doc/pipeline/steps/lockable-resources/
-
Jérémy Bobbio (Lunar) authored
In order to more easily lock a shared resource in the future, let’s convert the swh-docker-dev job from a freestyle Jenkins job to a declarative Pipeline job.
-
- Nov 10, 2022
-
-
Jérémy Bobbio (Lunar) authored
The mirror tests start by build Docker images. Scheduling it at midnight will make it run at the same time as the job building the main Docker images. This mean they will fight for access to the Docker daemon and thus one of them is likely to fail. Ideally, we would use the lockable resource plugin but as a stopgap measure, let’s schedule the mirror tests 4 hours after the main image building job.
-
- Nov 08, 2022
-
-
Antoine Lambert authored
As the parallel cypress runners use the same repository filesystem to execute tests from, some race conditions related to file system operations might happen which make the tests fail. As a workaround make each cypress runner execute tests from a copy of the swh-web repository filesystem (without root hidden folders). Such copy takes a dozen of seconds but it makes tests execution much more stable on Jenkins by avoiding the race conditions mentionned above.
-
- Nov 07, 2022
-
-
Jérémy Bobbio (Lunar) authored
swh-mirror tests are quite involved as they deploy a replicated Software Heritage Archive in the form of a Docker swarm. This new `swh-mirror/run-tests` job assumes that Jenkins running has access to a Docker engine. It needs: 1. to be able to build images; 2. to be configured as part of a swarm; 3. to have at least one node with the following labels: org.softwareheritage.mirror.volumes.storage-db=true org.softwareheritage.mirror.volumes.web-db=true org.softwareheritage.mirror.volumes.objstorage=true org.softwareheritage.mirror.volumes.redis=true The pipeline will first create up-to-date Docker images used by the swarm (using Jenkins host) and then run `pytest` in a Docker container using `swh-jenkins/tox`. As it depends on network resources, the test can drag for quite long. The timeout is currently set at one hour. In most cases the pipeline will be done in 15-20 minutes. Care is taken both in the test code and within the Jenkins pipeline to remove resources related to the Docker swarm (stack, containers, configs and volumes) on teardown. The job will need the changes proposed as D8634 to work.
-
- Oct 31, 2022
-
-
Antoine Lambert authored
This reverts commit a0fd3dc0 as that fix does not work since the issue it aimed to fix can still happen when running cypress tests in parallel on Jenkins.
-
- Oct 26, 2022
-
-
Antoine Lambert authored
Sometimes when multiple cypress processes load the same support file in parallel, a race condition is encountered as file copy operations are performed. As a result, the content of the support file is not loaded and cypress tests will fail. So use a specific copy of the support file for each cypress runner to mitigate that issue.
-
- Oct 19, 2022
-
-
Antoine Lambert authored
Main apache repository is no longer reachable from our infra so we need to use a mirror to fetch cassandra package now.
-
Antoine Lambert authored
The jobs associated to this pipeline are only triggered when commiting on the master branch or when pushing a phabricator diff based on a commit of the master branch. So in order to save time, data transfer and avoid possible clone issues, fetch only the master branch when cloning as all remote branches are fetched by default. Related to T4641
-
- Oct 18, 2022
-
-
Antoine Lambert authored
We are only interested in fetching the master branch of each swh repository to run the checks and tests of that job so pass option --single-branch to git clone command. Related to T4641
-
- Sep 29, 2022
-
-
Nicolas Dandrimont authored
-
- Sep 15, 2022
-
-
Antoine Lambert authored
Recent changes in that build job (running all checks for each swh package for instance) makes it exceed the 90 minutes timeout so increase it by 30 minutes.
-
- Aug 31, 2022
-
-
Antoine Lambert authored
In order to gain some time when executing Jenkins jobs for swh-web, execute cypress tests in parallel using 4 runners thanks to the sorry-cypress tool. To do so, we use the docker features of Jenkins pipeline. One sidecar container is created to execute the sorry-cypress server and parallel tests are executed in another container spawning 4 cypress processes. The communication between the two containers is done through a dedicated docker network. Regarding code coverage, we now need to merge the outputs produced by the 4 runners before generating reports. This change brings a 2x speedup on overall swh-web build jobs execution (when a single job is executed, speedup is lesser when multiple jobs are executed in parallel but still significative). Also restrict the maximum number of concurrent builds per job to 3 in order to avoid executing too many cypress processes at the same time as they consume quite a lot of resources on the Jenkins node. Related to T3423
-
Antoine Lambert authored
Add a max_concurrent option to swh pipeline jobs (master, Phabricator diff) in order to limit the number of concurrent builds per job. The number of concurrent builds is not limited by default. Related to T3423
-
- Aug 26, 2022
-
-
Antoine Lambert authored
-
- Aug 19, 2022
-
-
Antoine Lambert authored
Also execute code checks (flake8, codespell, mypy, ...) for all swh modules in this Jenkins job. It will enable to catch possible issues with the tools above and fix them early.
-
- Aug 11, 2022
-
-
Antoine Lambert authored
It enables to upload JavaScript source maps generated by webpack to help debugging swh-web JavaScript errors in Sentry Web UI. The Sentry webpack plugin must be used in webpack configuration to do so.
-
- Aug 10, 2022
-
-
Antoine Lambert authored
Currently, the only way to test Jenkins configuration changes using docker is to push changes to a remote feature branch in swh-jenkins-jobs repository and make jenkins clone that branch to test the changes. This is not really convenient to quickly test changes so prefer to mount the local swh-jenkins-jobs repository as a docker volume and clone that repository from Jenkins filesystem.
-
- Jul 20, 2022
-
-
Antoine R. Dumont authored
This reverts commit e1dd4f97. Related to T4135
-
Antoine R. Dumont authored
Related to T4135
-
- Jun 30, 2022
-
-
Antoine R. Dumont authored
-
Vincent Sellier authored
-
- Jun 28, 2022
-
-
Antoine R. Dumont authored
-
- Jun 03, 2022
-
-
Antoine R. Dumont authored
-
Antoine Lambert authored
Now that static analysis is no longer run in parallel and the number of mypy checks increased, we can sometimes hit the current 25 minutes timeout so better bumping it to 30 minutes.
-
- May 23, 2022
-
-
vlorentz authored
We have a recurring data race in flake8 tasks, probably caused by 'tox -e mypy' creating and deleting files while flake8 scans them; so we should not run them at the same time. While we are at it; also run radon sequentially. flake8 and radon each take less than 10s, so we should not notice much of a difference.
-
- May 04, 2022
-
-
Nicolas Dandrimont authored
Using --depth makes `git describe` return nonsensical values, which in turn makes setuptools-scm generate a v0.1.dev1 version number. This makes pip unable to resolve version-constrained dependencies on swh modules, which makes it install them from pip again.
-
- Apr 26, 2022
-
-
Nicolas Dandrimont authored
-
- Apr 25, 2022
-
-
Antoine R. Dumont authored
Without it, the error message is not clear from the get go and we need to tamper with build configuration to actually have access to the error message. See [1] with the --verbose flag versus [2] without it. [1] ``` 17:56:44 [?25h[34mINFO [0m Response from https://upload.pypi.org/legacy/: 17:56:44 400 The description failed to render for 'text/x-rst'. See 17:56:44 https://pypi.org/help/#description-content-type for more information. 17:56:44 [34mINFO [0m <html> 17:56:44 <head> 17:56:44 <title>400 The description failed to render for 'text/x-rst'. See 17:56:44 https://pypi.org/help/#description-content-type for more 17:56:44 information.</title> 17:56:44 </head> 17:56:44 <body> 17:56:44 <h1>400 The description failed to render for 'text/x-rst'. See 17:56:44 https://pypi.org/help/#description-content-type for more 17:56:44 information.</h1> 17:56:44 The server could not comply with the request since it is either 17:56:44 malformed or otherwise incorrect.<br/><br/> 17:56:44 The description failed to render for 'text/x-rst'. See 17:56:44 https://pypi.org/help/#description-content-type for more information. 17:56:44 17:56:44 17:56:44 </body> 17:56:44 </html> ``` from https://jenkins.softwareheritage.org/job/DLDCVS/job/pypi-upload/12/console [2] ``` 17:44:19 [?25h[33mWARNING [0m Error during upload. Retry with the --verbose option for more details. ``` https://jenkins.softwareheritage.org/job/DLDCVS/job/pypi-upload/11/console
-
- Apr 22, 2022
-
-
vlorentz authored
-
- Apr 13, 2022
-
-
Antoine R. Dumont authored
That's up to the repository's Makefile to determine that. That way, it currently fixes the build and push tag logic target to actually work. But bonus, it will allow to keep the building logic of image(s) within the scope of the repository which knows what to do (so building multiple images becomes possible with that very build). So now the makefile must respect the following conventions (still explained in the template): - build: locally build the image out of the repository TAG, IMAGE_LABEL (to ease cleanup part) must be provided - test: test the locally built image TAG must be provided - clean: clean up locally built image (parametric on the jenkins build) IMAGE_LABEL must be provided - push: publish to the registry TAG, REGISTRY_URL must be provided
-