diff --git a/docs/sysadm/mirror-operations/deploy.rst b/docs/sysadm/mirror-operations/deploy.rst
index 004df5d97125e0f8b2218ce6e61b575be724d51c..89a154f2927d87f522fa46d54cde564203cd52a0 100644
--- a/docs/sysadm/mirror-operations/deploy.rst
+++ b/docs/sysadm/mirror-operations/deploy.rst
@@ -14,9 +14,9 @@ stack:
- An instance of the object storage (:ref:`swh-objstorage`);
- A large storage system (zfs or cloud storage) as the objstorage backend;
- An instance of the frontend (:ref:`swh-web`);
-- [Optional] An instance of the search engine backend (:ref:`swh-search`);
-- [Optional] An elasticsearch instance as swh-search backend;
-- [Optional] The vault service and its support tooling (RabbitMQ,
+- An instance of the search engine backend (:ref:`swh-search`);
+- An elasticsearch instance as swh-search backend;
+- The vault service and its support tooling (RabbitMQ,
:ref:`swh-scheduler`, :ref:`swh-vault`, ...);
- The replayer services:
@@ -29,6 +29,11 @@ Each service consists in an HTTP-based RPC served by a `gunicorn
<https://gunicorn.org/>`_ `WSGI
<https://fr.wikipedia.org/wiki/Web_Server_Gateway_Interface>`_ server.
+.. Note:: It is **not** recommended to try to deploy each |swh| service
+ individually. You should rather start from the example docker-based
+ deployment project linked below.
+
+
Docker-based deployment
-----------------------
diff --git a/docs/sysadm/mirror-operations/docker.rst b/docs/sysadm/mirror-operations/docker.rst
index 8ff64bd388559a841bbd253311bfe161bf66f74a..9e24731d63d74d9af28828eb3cc6636788708418 100644
--- a/docs/sysadm/mirror-operations/docker.rst
+++ b/docs/sysadm/mirror-operations/docker.rst
@@ -78,11 +78,10 @@ Node labels
Note that the provided :file:`base-services.yaml` file has label-based
placement constraints for several services.
-The ``db-storage``, ``db-web``, ``objstorage`` and ``redis`` containers, which
-depend on the availability of specific volumes (respectively
-``<STACK>_storage-db``, ``<STACK>_web-db`` and ``<STACK>_objstorage``) are
-pinned to specific nodes using labels named
-``org.softwareheritage.mirror.volumes.<base volume name>`` (e.g.
+The ``elasticsearch``, ``scheduler-db``, ``storage-db``, ``vault-db``,
+``web-db``, ``objstorage`` and ``redis`` containers, which depend on the
+availability of specific volumes, are pinned to specific nodes using labels
+named ``org.softwareheritage.mirror.volumes.<base volume name>`` (e.g.
``org.softwareheritage.mirror.volumes.objstorage``).
When you create a local volume for a given container, you should add the relevant label
@@ -99,8 +98,8 @@ requirements, for the services to start.
The monitoring services, ``prometheus``, ``prometheus-statsd-exporter`` and
``grafana`` also have placement constraints based on the label
-``org.softwareheritage.mirror.monitoring``. So make sure to add this label to
-one (and only one) node of the cluster:
+``org.softwareheritage.mirror.monitoring`` (and they also use volumes). So make
+sure to add this label to one (and only one) node of the cluster:
.. code-block:: console
@@ -120,19 +119,29 @@ inspect`` command:
Labels that need to be defined are:
+- ``org.softwareheritage.mirror.monitoring=true``: node that will host
+ the monitoring services.
+
- ``org.softwareheritage.mirror.volumes.objstorage=true``: node that will host
- the objstorage service, on which the ``swh_objstorage`` volume must be
- defined.
+ the objstorage service.
+
+- ``org.softwareheritage.mirror.volumes.elasticsearch=true``: node that will
+ host the elasticsearch service.
- ``org.softwareheritage.mirror.volumes.redis=true``: node that will host the
- redis service on which the ``swh_redis`` volume must be defined.
+ redis service.
- ``org.softwareheritage.mirror.volumes.storage-db=true``: node that will host
- the swh-storage Postgresql database, on which the ``swh_storage-db`` volume must
- be defined.
+ the swh-storage Postgresql database.
+
+- ``org.softwareheritage.mirror.volumes.scheduler-db=true``: node that will
+ host the swh-scheduler Postgresql database.
+
+- ``org.softwareheritage.mirror.volumes.vault-db=true``: node that will host
+ the swh-vault Postgresql database.
- ``org.softwareheritage.mirror.volumes.web-db=true``: node that will host the
- swh-web Postgresql database, on which the ``swh_web-db`` must be defined.
+ swh-web Postgresql database.
Managing secrets
@@ -179,7 +188,7 @@ environment variable:
.. code-block:: console
- swh:~/swh-mirror$ export SWH_IMAGE_TAG=20211022-121751
+ swh:~/swh-mirror$ export SWH_IMAGE_TAG=20240417-190717
Make sure you have node labels attributed properly. Then you can spawn the
base services using the following command:
@@ -219,19 +228,19 @@ base services using the following command:
swh:~/swh-mirror$ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
- ptlhzue025zm swh_content-replayer replicated 0/0 softwareheritage/replayer:20220225-101454
+ ptlhzue025zm swh_content-replayer replicated 0/0 softwareheritage/replayer:20240417-190717
ycyanvhh0jnt swh_db-storage replicated 1/1 (max 1 per node) postgres:13
qlaf9tcyimz7 swh_db-web replicated 1/1 (max 1 per node) postgres:13
aouw9j8uovr2 swh_grafana replicated 1/1 (max 1 per node) grafana/grafana:latest
- uwqe13udgyqt swh_graph-replayer replicated 0/0 softwareheritage/replayer:20220225-101454
+ uwqe13udgyqt swh_graph-replayer replicated 0/0 softwareheritage/replayer:20240417-190717
mepbxllcxctu swh_memcache replicated 1/1 memcached:latest
kfzirv0h298h swh_nginx global 3/3 nginx:latest *:5081->5081/tcp
- t7med8frg9pr swh_objstorage replicated 2/2 softwareheritage/base:20220225-101454
+ t7med8frg9pr swh_objstorage replicated 2/2 softwareheritage/base:20240417-190717
5s34wzo29ukl swh_prometheus replicated 1/1 (max 1 per node) prom/prometheus:latest
rwom7r3yv5ql swh_prometheus-statsd-exporter replicated 1/1 (max 1 per node) prom/statsd-exporter:latest
wuwydthechea swh_redis replicated 1/1 (max 1 per node) redis:6.2.6
- jztolbmjp1vi swh_storage replicated 2/2 softwareheritage/base:20220225-101454
- xxc4c66x0uj1 swh_web replicated 1/1 softwareheritage/web:20220225-101454
+ jztolbmjp1vi swh_storage replicated 2/2 softwareheritage/base:20240417-190717
+ xxc4c66x0uj1 swh_web replicated 1/1 softwareheritage/web:20240417-190717
This will start a series of containers with:
@@ -247,6 +256,8 @@ This will start a series of containers with:
- a swh_content-replayer service (initially set to 0 replica, see below)
- a swh_graph-replayer service (initially set to 0 replica, see below)
- a redis for the replication error logs,
+- a set of services for the vault,
+- a set of services for the search (including a single node elasticsearch)
using the pinned version of the docker images.
@@ -436,7 +447,7 @@ You can check everything is running with:
swh:~/swh-mirror$ docker service ps swh_graph-replayer
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
- ioyt34ok118a swh_graph-replayer.1 softwareheritage/replayer:20220225-101454 node1 Running Running 17 minutes ago
+ ioyt34ok118a swh_graph-replayer.1 softwareheritage/replayer:20240417-190717 node1 Running Running 17 minutes ago
If everything is OK, you should have your mirror filling. Check docker logs:
diff --git a/docs/sysadm/mirror-operations/index.rst b/docs/sysadm/mirror-operations/index.rst
index f34af06405d346e993aed5474029ba168a21ff64..fbe9835abb318fb1d9650f1df97aabcd5c52c93e 100644
--- a/docs/sysadm/mirror-operations/index.rst
+++ b/docs/sysadm/mirror-operations/index.rst
@@ -24,6 +24,8 @@ proper mirror should also allow to:
- navigate the archive copy using a Web browser and/or the Web API (typically
using the :ref:`the web application <swh-web>`),
+- have minimal search capabilities (typically using :ref:`the swh-search
+ service <swh-search>` with an Elasticsearch backend),
- retrieve data from the copy of the archive (typically using the :ref:`the
vault service <swh-vault>`)
@@ -41,9 +43,23 @@ blob objects (file content) from the |swh| :ref:`object storage <swh-objstorage>
General view of the |swh| mirroring architecture.
+.. Note:: This general view is very simplified and does not show all the
+ services involved in hosting and operating a mirror.
+
See the :ref:`planning-a-mirror` for a complete description of the requirements
to host a mirror.
+.. Note:: Hosting a complete mirror is a complex task, involving the deployment
+ of dozens of inter related (micro-)services. It should be planned and
+ operated carefully, using state-of-art ops practices (cloud-based, or
+ using container orchestration tools on an elastic execution platform
+ like kubernetes_, `docker swarm`_, or using tools like Ansible_ or
+ `Salt Stack`_).
+
+.. Important:: It is **strongly** recommended to start with a simple `docker
+ swarm`_ based deployment (this can be done on a single machine) as
+ described in :ref:`mirror_deploy`.
+
Mirroring the Graph Storage
~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -110,7 +126,11 @@ Installation
When using the |swh| software stack to deploy a mirror, a number of |swh|
software components must be installed (cf. architecture diagram above).
-A `docker-swarm <https://docs.docker.com/engine/swarm/>`_ based deployment
+.. Note:: It is **not** recommended to try to deploy each |swh| service
+ individually. You should rather start from the example docker-based
+ deployment project :ref:`described here <mirror_docker>`.
+
+A `docker swarm`_ based deployment
solution is provided as a working example of the mirror stack,
see :ref:`mirror_deploy`.
@@ -119,6 +139,10 @@ production-like deployment.
.. _Kafka: https://kafka.apache.org/
.. _msgpack: https://msgpack.org
+.. _`docker swarm`: https://docs.docker.com/engine/swarm/
+.. _Ansible: https://docs.ansible.com/ansible/latest/index.html
+.. _kubernetes: https://kubernetes.io/fr/
+.. _`Salt Stack`: https://saltproject.io/
You may want to read: