This code was used at the time to get extra info (notably stack trace) about a save code now task but this no longer works since a while and it is redundant with sentry so better removing it.
Antoine Lambert (4145db56) at 28 Mar 11:03
save_code_now: Remove use of elasticsearch to get extra save task info
... and 162 more commits
How about this then ?
Quick start
-----------
First, retrieve Software Heritage development environment to get the
Docker configuration::
~$ git clone https://gitlab.softwareheritage.org/swh/devel/swh-environment.git
~$ git -C swh-environment clone https://gitlab.softwareheritage.org/swh/devel/docker.git
~$ cd swh-environment/docker
.. note::
If you intend to hack on Sofware Heritage souce code and test your changes with docker,
you should rather follow the instructions in section :ref:`checkout-source-code` to
install the full Software Heritage development environment that includes Docker configuration.
Antoine Lambert (97fd9824) at 28 Mar 10:13
devel/getting-started: Update instructions to clone docker environment
... and 182 more commits
This matches what swh-search returns and expects. This could make test_lookup_origin() (in test_archive.py) fail when the bad luck (?) made hypothesis generate a test set where swh-search's in-memory backend tries to merge the visit_types with an existing one (thus trying to add a pair set objects).
Adapt the definition of OriginInfo (more or less the local version of swh-search's OriginDict) accordingly.
Ack, and we also need to update the docker images used for these tests as they are running quite outdated versions of our packages (using legacy debian buster packages that we no longer maintain).
As new mirrors will be deployed in the future, better having the mirror stack regularly tested.
So I ended up with swh/infra/ci-cd/swh-jenkins-jobs!223 (diffs) that fixes the Jenkins job for swh-mirror
but unfortunately the kafka credentials are no longer valid as the broker changed.
Run job in the Jenkins docker agent.
Add support for testing merge requests.
Ensure a docker swarm cluster with required labels is created prior running tests.
Update kafka broker URL.
Nevertheless, credentials for the kafka broker on staging must be updated as the following error is reported (see https://jenkins.softwareheritage.org/job/DFP/job/run-tests/526/consoleFull):
ERROR:swh.objstorage.replayer.replay:Failed operation get_object on 7e3d4d9741e269a4889eb2e3b8361ee1021f1648 after 3 retries; last exception: ReplayError(7e3d4d9741e269a4889eb2e3b8361ee1021f1648, APIError(\'<html>\\r\\n<head><title>403 Forbidden</title></head>\\r\\n<body>\\r\\n<center><h1>403 Forbidden</h1></center>\\r\\n<hr><center>nginx</center>\\r\\n</body>\\r\\n</html>\\r\\n\', <Response [403]>), [\' File "/usr/lib/python3/dist-packages/swh/objstorage/replayer/replay.py", line 183, in get_object\\n obj = objstorage.get(obj_id)\\n\', \' File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 190, in meth_\\n return self._post(meth._endpoint_path, post_data)\\n\', \' File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 285, in _post\\n return self._decode_response(response)\\n\', \' File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 372, in _decode_response\\n self.raise_for_status(response)\\n\', \' File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 362, in raise_for_status\\n raise exception from None\\n\'])\n' (test_graph_replayer.py:101)
Antoine Lambert (08713edc) at 27 Mar 16:15
jobs/swh-mirror: Update job configuration and fix its execution
Antoine Lambert (20a2f332) at 27 Mar 16:10
jobs/swh-mirror: Update job configuration to fix its execution
... and 182 more commits
Antoine Lambert (5aad6070) at 27 Mar 15:42
Debug mirror tests
Antoine Lambert (496f1969) at 27 Mar 14:59
objstorage_checker: Use check method from object storage interface
Instead of reinventing the wheel, prefer to use the check method from the object storage interface for verifying content presence and integrity.
Depends on !65 and swh-objstorage!178.
Related to #4694.
Antoine Lambert (c401fa6b) at 27 Mar 14:56
objstorage_checker: Use check method from object storage interface
... and 49 more commits
Antoine Lambert (481e31e1) at 27 Mar 14:29
Well I guess that it must have already been setup at some point on Jenkins as those tests used to be up and running.
But indeed it is currently hanging after that MR applied: https://jenkins.softwareheritage.org/blue/organizations/jenkins/DFP%2Frun-tests/detail/run-tests/517/pipeline, will try to make them work locally.
Antoine Lambert (a7420031) at 27 Mar 13:29
tests: Simplify setup after pytest 8.1 release
Since the release of pytest 8.1, some pytest options are no longer needed and editable install can be used when running tests using tox.