Skip to content
Snippets Groups Projects
  1. Jan 03, 2023
  2. Jan 02, 2023
  3. Nov 28, 2022
  4. Nov 21, 2022
  5. Nov 14, 2022
  6. Nov 10, 2022
    • Jérémy Bobbio (Lunar)'s avatar
      Stop scheduling mirror tests while building Docker images · 948efe64
      Jérémy Bobbio (Lunar) authored
      The mirror tests start by build Docker images. Scheduling it at midnight
      will make it run at the same time as the job building the main Docker
      images. This mean they will fight for access to the Docker daemon and
      thus one of them is likely to fail.
      
      Ideally, we would use the lockable resource plugin but as a stopgap
      measure, let’s schedule the mirror tests 4 hours after the main image
      building job.
      948efe64
  7. Nov 08, 2022
    • Antoine Lambert's avatar
      templates/stage-cypress-tests: Prevent race conditions between runners · 13f95d29
      Antoine Lambert authored
      As the parallel cypress runners use the same repository filesystem to execute
      tests from, some race conditions related to file system operations might happen
      which make the tests fail.
      
      As a workaround make each cypress runner execute tests from a copy of the swh-web
      repository filesystem (without root hidden folders).
      
      Such copy takes a dozen of seconds but it makes tests execution much more stable
      on Jenkins by avoiding the race conditions mentionned above.
      13f95d29
  8. Nov 07, 2022
    • Jérémy Bobbio (Lunar)'s avatar
      Add a job running swh-mirror tests · 3f35eb07
      Jérémy Bobbio (Lunar) authored
      swh-mirror tests are quite involved as they deploy a replicated Software
      Heritage Archive in the form of a Docker swarm.
      
      This new `swh-mirror/run-tests` job assumes that Jenkins running has
      access to a Docker engine. It needs:
      
      1. to be able to build images;
      2. to be configured as part of a swarm;
      3. to have at least one node with the following labels:
      
          org.softwareheritage.mirror.volumes.storage-db=true
          org.softwareheritage.mirror.volumes.web-db=true
          org.softwareheritage.mirror.volumes.objstorage=true
          org.softwareheritage.mirror.volumes.redis=true
      
      The pipeline will first create up-to-date Docker images used by the
      swarm (using Jenkins host) and then run `pytest` in a Docker container
      using `swh-jenkins/tox`.
      
      As it depends on network resources, the test can drag for quite long.
      The timeout is currently set at one hour. In most cases the pipeline
      will be done in 15-20 minutes.
      
      Care is taken both in the test code and within the Jenkins pipeline to
      remove resources related to the Docker swarm (stack, containers, configs
      and volumes) on teardown.
      
      The job will need the changes proposed as D8634 to work.
      3f35eb07
  9. Oct 31, 2022
  10. Oct 26, 2022
    • Antoine Lambert's avatar
      templates/stage-cypress-tests: Prevent race condition with support file · a0fd3dc0
      Antoine Lambert authored
      Sometimes when multiple cypress processes load the same support file
      in parallel, a race condition is encountered as file copy operations
      are performed. As a result, the content of the support file is not
      loaded and cypress tests will fail.
      
      So use a specific copy of the support file for each cypress runner
      to mitigate that issue.
      a0fd3dc0
  11. Oct 19, 2022
  12. Oct 18, 2022
  13. Sep 29, 2022
  14. Sep 15, 2022
  15. Aug 31, 2022
    • Antoine Lambert's avatar
      templates/cypress: Run tests in parallel using sorry-cypress · ddbafe95
      Antoine Lambert authored
      In order to gain some time when executing Jenkins jobs for swh-web,
      execute cypress tests in parallel using 4 runners thanks to the
      sorry-cypress tool.
      
      To do so, we use the docker features of Jenkins pipeline.
      One sidecar container is created to execute the sorry-cypress server
      and parallel tests are executed in another container spawning 4
      cypress processes. The communication between the two containers
      is done through a dedicated docker network.
      
      Regarding code coverage, we now need to merge the outputs produced
      by the 4 runners before generating reports.
      
      This change brings a 2x speedup on overall swh-web build jobs execution
      (when a single job is executed, speedup is lesser when multiple jobs
      are executed in parallel but still significative).
      
      Also restrict the maximum number of concurrent builds per job to 3
      in order to avoid executing too many cypress processes at the same
      time as they consume quite a lot of resources on the Jenkins node.
      
      Related to T3423
      ddbafe95
    • Antoine Lambert's avatar
      jobs/swh-pipeline: Enable to limit number of concurrent builds per job · b00f888c
      Antoine Lambert authored
      Add a max_concurrent option to swh pipeline jobs (master, Phabricator diff)
      in order to limit the number of concurrent builds per job.
      
      The number of concurrent builds is not limited by default.
      
      Related to T3423
      b00f888c
  16. Aug 26, 2022
  17. Aug 19, 2022
  18. Aug 11, 2022
  19. Aug 10, 2022
    • Antoine Lambert's avatar
      docker: Mount and clone local swh-jenkins-jobs repo for conf testing · 7b19d5be
      Antoine Lambert authored
      Currently, the only way to test Jenkins configuration changes using
      docker is to push changes to a remote feature branch in swh-jenkins-jobs
      repository and make jenkins clone that branch to test the changes.
      
      This is not really convenient to quickly test changes so prefer to
      mount the local swh-jenkins-jobs repository as a docker volume
      and clone that repository from Jenkins filesystem.
      7b19d5be
  20. Jul 20, 2022
  21. Jun 30, 2022
  22. Jun 28, 2022
  23. Jun 03, 2022
  24. May 23, 2022
    • vlorentz's avatar
      Run flake8/mypy/radon sequentially · 166de856
      vlorentz authored
      We have a recurring data race in flake8 tasks, probably caused by 'tox -e mypy'
      creating and deleting files while flake8 scans them; so we should not run
      them at the same time.
      
      While we are at it; also run radon sequentially. flake8 and radon each take less
      than 10s, so we should not notice much of a difference.
      166de856
  25. May 04, 2022
  26. Apr 26, 2022
  27. Apr 25, 2022
  28. Apr 22, 2022
  29. Apr 13, 2022
    • Antoine R. Dumont's avatar
      Simplify docker image build to not override the docker image to build · 723923ce
      Antoine R. Dumont authored
      That's up to the repository's Makefile to determine that. That way, it currently fixes
      the build and push tag logic target to actually work. But bonus, it will allow to keep
      the building logic of image(s) within the scope of the repository which knows what to
      do (so building multiple images becomes possible with that very build).
      
      So now the makefile must respect the following conventions (still explained in the
      template):
      - build: locally build the image out of the repository
      TAG, IMAGE_LABEL (to ease cleanup part) must be provided
      - test: test the locally built image
      TAG must be provided
      - clean: clean up locally built image (parametric on the jenkins build)
      IMAGE_LABEL must be provided
      - push: publish to the registry
      TAG, REGISTRY_URL must be provided
      Verified
      723923ce
Loading