Skip to content

docker: significant rework and test improvements

David Douard requested to merge douardda/swh-environment:docker-tests into master

This monster MR aims at improving the docker test environment and add more test scenarios (aka integration tests).

It is probably not to be reviewed from the diffs but rather have a look at how the result looks like...

Among all the things in there:

  • allow mirror and replica archives to completely run in parallel with the main instance instead of replacing the main web/api etc services with mirror/replica ones,
  • put everything related to mirror/replicas in conf/{mirror,replica} directories
  • make the mirror and replica environments "full-battery included" (have dedicated swh-web, storage, objstorage (for the mirror) scheduler, vault, etc. stacks)
  • configure the "mirror skin" for swh-web for these 2 environments sot it's visible one is not browsing the main one (with beautiful logos)
  • (re-)add healthchecks in compose files: these are necessary to make different "compose up" command run flawlessly (hopefully)
  • extract (main) vault and deposit in dedicated compose files; they are not needed for most test scenarios
  • add a scrubber stack (untested for now)
  • remove the cmak service
  • add a kafka-rest service (confluent cp kafka-rest) providing easy access to kafka cluster status (topics, consumers, producers, etc.) Used to ensure replayers have done their job in mirror tests.

Tests:

  • add a test scenario for the mirror stack, including testing the vault service in a mirror stack
  • add support for more than one loaded origin (and partially test a mercurial loading in the mirror tests)
  • improve execution time for the test_git_loader by using the /known/ API endpoint (instead of querying each object one at a time)

Debatable (well, even more than the other ones) changes:

  • move pg 16
  • move to Python 3.11

Note: with the addition of the new mirror test, all docker compose related fixtures have been changed to "module" scoped fixtures. This is required because testing the vault or testing the mirror require different compose setups. But it comes at a significant execution time cost.

What is still missing:

  • proper hg (and all other supported loaders really) scenario
  • scrubber scenarios (<- this is my primary goal for all this)
  • swh-search tests
  • indexers tests
  • graphql tests
  • you-name-it tests...
Edited by David Douard

Merge request reports