docker: Speedup compose environment initialization significantly
Instead of creating kafka topics prior starting the kafka consumer services, prefer to delegate these tasks to the journal clients used in these services.
Proceeding like this brings a huge speedup to initialize the compose environment (from more than one minute to 15 seconds on my machine).
Depends on swh-journal!263 (merged)
Before:
(swh) ✔ ~/swh/swh-environment/docker [master|⚑ 21]
11:41 $ time docker compose up -d
[+] Running 50/50
⠿ Network docker_default Created 0.1s
⠿ Volume "docker_storage-data" Created 0.0s
⠿ Volume "docker_kafka-data" Created 0.0s
⠿ Volume "docker_redis-data" Created 0.0s
⠿ Volume "docker_objstorage-data" Created 0.0s
⠿ Container docker-swh-lister-maven-nginx-1 Started 4.8s
⠿ Container docker-swh-objstorage-1 Started 3.2s
⠿ Container docker-amqp-1 Started 3.0s
⠿ Container docker-zookeeper-1 Started 3.1s
⠿ Container docker-prometheus-rabbitmq-exporter-1 Started 4.1s
⠿ Container docker-cmak-1 Started 3.8s
⠿ Container docker-swh-idx-storage-db-1 Started 3.7s
⠿ Container docker-swh-deposit-db-1 Started 2.8s
⠿ Container docker-nginx-1 Started 2.7s
⠿ Container docker-swh-storage-db-1 Started 4.3s
⠿ Container docker-swh-vault-db-1 Started 3.2s
⠿ Container docker-swh-web-db-1 Started 3.8s
⠿ Container docker-mailhog-1 Started 4.5s
⠿ Container docker-redis-1 Healthy 6.3s
⠿ Container docker-memcache-1 Started 4.8s
⠿ Container docker-swh-search-1 Started 5.0s
⠿ Container docker-swh-scheduler-db-1 Started 3.6s
⠿ Container docker-prometheus-statsd-exporter-1 Started 5.0s
⠿ Container docker-swh-idx-storage-1 Started 5.2s
⠿ Container docker-prometheus-1 Started 6.4s
⠿ Container docker-swh-counters-1 Healthy 26.7s
⠿ Container docker-swh-scheduler-1 Started 5.3s
⠿ Container docker-swh-scheduler-update-metrics-1 Started 5.7s
⠿ Container docker-kafka-1 Healthy 65.7s
⠿ Container docker-grafana-1 Started 8.3s
⠿ Container docker-swh-scheduler-runner-1 Started 6.7s
⠿ Container docker-swh-deposit-1 Started 6.8s
⠿ Container docker-swh-scheduler-listener-1 Started 8.3s
⠿ Container docker-swh-scheduler-runner-priority-1 Started 8.4s
⠿ Container docker-swh-counters-journal-client-1 Started 66.3s
⠿ Container docker-swh-search-journal-client-indexed-1 Started 67.3s
⠿ Container docker-swh-scheduler-journal-client-1 Started 67.0s
⠿ Container docker-swh-search-journal-client-objects-1 Started 67.0s
⠿ Container docker-swh-lister-1 Started 9.4s
⠿ Container docker-swh-storage-1 Started 4.9s
⠿ Container docker-swh-loader-1 Started 7.6s
⠿ Container docker-swh-web-1 Started 8.0s
⠿ Container docker-swh-indexer-worker-celery-1 Started 8.4s
⠿ Container docker-swh-graphql-1 Started 7.1s
⠿ Container docker-swh-loader-deposit-1 Started 7.9s
⠿ Container docker-swh-indexer-worker-journal-1 Started 64.9s
⠿ Container docker-swh-vault-1 Started 7.2s
⠿ Container docker-swh-web-cron-1 Started 9.4s
⠿ Container docker-swh-scheduler-schedule-recurrent-1 Started 9.2s
⠿ Container docker-swh-vault-worker-1 Started 8.9s
real 1m7,686s
user 0m1,345s
sys 0m0,401sAfter:
(swh) ✔ ~/swh/swh-environment/docker [docker-compose-speedup-init L|⚑ 21]
11:47 $ time docker compose up -d
[+] Running 50/50
⠿ Network docker_default Created 0.1s
⠿ Volume "docker_redis-data" Created 0.0s
⠿ Volume "docker_objstorage-data" Created 0.0s
⠿ Volume "docker_storage-data" Created 0.0s
⠿ Volume "docker_kafka-data" Created 0.0s
⠿ Container docker-swh-scheduler-db-1 Started 3.7s
⠿ Container docker-redis-1 Started 3.5s
⠿ Container docker-swh-storage-db-1 Started 3.7s
⠿ Container docker-mailhog-1 Started 3.3s
⠿ Container docker-prometheus-statsd-exporter-1 Started 5.1s
⠿ Container docker-swh-web-db-1 Started 3.3s
⠿ Container docker-zookeeper-1 Started 3.9s
⠿ Container docker-nginx-1 Started 3.5s
⠿ Container docker-swh-search-1 Started 5.3s
⠿ Container docker-swh-objstorage-1 Started 5.2s
⠿ Container docker-swh-vault-db-1 Started 5.7s
⠿ Container docker-cmak-1 Started 3.0s
⠿ Container docker-memcache-1 Started 3.2s
⠿ Container docker-swh-deposit-db-1 Started 5.0s
⠿ Container docker-prometheus-rabbitmq-exporter-1 Started 4.5s
⠿ Container docker-amqp-1 Started 3.6s
⠿ Container docker-swh-idx-storage-db-1 Started 5.5s
⠿ Container docker-swh-lister-maven-nginx-1 Started 5.7s
⠿ Container docker-prometheus-1 Started 7.0s
⠿ Container docker-swh-counters-1 Started 5.9s
⠿ Container docker-swh-scheduler-update-metrics-1 Started 5.5s
⠿ Container docker-swh-scheduler-1 Started 5.7s
⠿ Container docker-swh-idx-storage-1 Started 7.5s
⠿ Container docker-grafana-1 Started 12.0s
⠿ Container docker-kafka-1 Started 6.1s
⠿ Container docker-swh-deposit-1 Started 8.5s
⠿ Container docker-swh-scheduler-listener-1 Started 8.1s
⠿ Container docker-swh-scheduler-runner-1 Started 8.1s
⠿ Container docker-swh-scheduler-runner-priority-1 Started 7.9s
⠿ Container docker-swh-search-journal-client-objects-1 Started 8.5s
⠿ Container docker-swh-scheduler-journal-client-1 Started 8.6s
⠿ Container docker-swh-search-journal-client-indexed-1 Started 8.8s
⠿ Container docker-swh-counters-journal-client-1 Started 9.9s
⠿ Container docker-swh-lister-1 Started 10.7s
⠿ Container docker-swh-storage-1 Started 7.0s
⠿ Container docker-swh-vault-1 Started 11.3s
⠿ Container docker-swh-indexer-worker-celery-1 Started 11.1s
⠿ Container docker-swh-loader-1 Started 10.9s
⠿ Container docker-swh-web-1 Started 11.9s
⠿ Container docker-swh-graphql-1 Started 12.0s
⠿ Container docker-swh-loader-deposit-1 Started 10.7s
⠿ Container docker-swh-indexer-worker-journal-1 Started 9.3s
⠿ Container docker-swh-web-cron-1 Started 13.2s
⠿ Container docker-swh-vault-worker-1 Started 13.2s
⠿ Container docker-swh-scheduler-schedule-recurrent-1 Started 13.1s
real 0m15,750s
user 0m0,316s
sys 0m0,126s