Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • ardumont/swh-docs
  • anlambert/swh-docs
  • douardda/swh-docs
  • vlorentz/swh-docs
  • vsellier/swh-docs
  • lunar/swh-docs
  • cmatrix/swh-docs
  • bchauvet/swh-docs
  • guillaume/swh-docs
  • HarshvMahawar/swh-docs
  • swh/devel/swh-docs
  • olasd/swh-docs
  • pabs/swh-docs
  • rboyer/swh-docs
  • marmoute/swh-docs
  • varasterix/swh-docs
16 results
Show changes
Commits on Source (2)
Hide whitespace changes
Inline Side-by-side
docs/sysadm/mirror-operations/deploy.rst
View file @ 6376a8f6
......@@ -14,9 +14,9 @@ stack:
- An instance of the object storage (:ref:`swh-objstorage`);
- A large storage system (zfs or cloud storage) as the objstorage backend;
- An instance of the frontend (:ref:`swh-web`);
- [Optional] An instance of the search engine backend (:ref:`swh-search`);
- [Optional] An elasticsearch instance as swh-search backend;
- [Optional] The vault service and its support tooling (RabbitMQ,
- An instance of the search engine backend (:ref:`swh-search`);
- An elasticsearch instance as swh-search backend;
- The vault service and its support tooling (RabbitMQ,
:ref:`swh-scheduler`, :ref:`swh-vault`, ...);
- The replayer services:
......@@ -29,6 +29,11 @@ Each service consists in an HTTP-based RPC served by a `gunicorn
<https://gunicorn.org/>`_ `WSGI
<https://fr.wikipedia.org/wiki/Web_Server_Gateway_Interface>`_ server.
.. Note:: It is **not** recommended to try to deploy each |swh| service
individually. You should rather start from the example docker-based
deployment project linked below.
Docker-based deployment
-----------------------
......
docs/sysadm/mirror-operations/docker.rst
View file @ 6376a8f6
......@@ -78,11 +78,10 @@ Node labels
Note that the provided :file:`base-services.yaml` file has label-based
placement constraints for several services.
The ``db-storage``, ``db-web``, ``objstorage`` and ``redis`` containers, which
depend on the availability of specific volumes (respectively
``<STACK>_storage-db``, ``<STACK>_web-db`` and ``<STACK>_objstorage``) are
pinned to specific nodes using labels named
``org.softwareheritage.mirror.volumes.<base volume name>`` (e.g.
The ``elasticsearch``, ``scheduler-db``, ``storage-db``, ``vault-db``,
``web-db``, ``objstorage`` and ``redis`` containers, which depend on the
availability of specific volumes, are pinned to specific nodes using labels
named ``org.softwareheritage.mirror.volumes.<base volume name>`` (e.g.
``org.softwareheritage.mirror.volumes.objstorage``).
When you create a local volume for a given container, you should add the relevant label
......@@ -99,8 +98,8 @@ requirements, for the services to start.
The monitoring services, ``prometheus``, ``prometheus-statsd-exporter`` and
``grafana`` also have placement constraints based on the label
``org.softwareheritage.mirror.monitoring``. So make sure to add this label to
one (and only one) node of the cluster:
``org.softwareheritage.mirror.monitoring`` (and they also use volumes). So make
sure to add this label to one (and only one) node of the cluster:
.. code-block:: console
......@@ -120,19 +119,29 @@ inspect`` command:
Labels that need to be defined are:
- ``org.softwareheritage.mirror.monitoring=true``: node that will host
the monitoring services.
- ``org.softwareheritage.mirror.volumes.objstorage=true``: node that will host
the objstorage service, on which the ``swh_objstorage`` volume must be
defined.
the objstorage service.
- ``org.softwareheritage.mirror.volumes.elasticsearch=true``: node that will
host the elasticsearch service.
- ``org.softwareheritage.mirror.volumes.redis=true``: node that will host the
redis service on which the ``swh_redis`` volume must be defined.
redis service.
- ``org.softwareheritage.mirror.volumes.storage-db=true``: node that will host
the swh-storage Postgresql database, on which the ``swh_storage-db`` volume must
be defined.
the swh-storage Postgresql database.
- ``org.softwareheritage.mirror.volumes.scheduler-db=true``: node that will
host the swh-scheduler Postgresql database.
- ``org.softwareheritage.mirror.volumes.vault-db=true``: node that will host
the swh-vault Postgresql database.
- ``org.softwareheritage.mirror.volumes.web-db=true``: node that will host the
swh-web Postgresql database, on which the ``swh_web-db`` must be defined.
swh-web Postgresql database.
Managing secrets
......@@ -179,7 +188,7 @@ environment variable:
.. code-block:: console
swh:~/swh-mirror$ export SWH_IMAGE_TAG=20211022-121751
swh:~/swh-mirror$ export SWH_IMAGE_TAG=20240417-190717
Make sure you have node labels attributed properly. Then you can spawn the
base services using the following command:
......@@ -219,19 +228,19 @@ base services using the following command:
swh:~/swh-mirror$ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
ptlhzue025zm swh_content-replayer replicated 0/0 softwareheritage/replayer:20220225-101454
ptlhzue025zm swh_content-replayer replicated 0/0 softwareheritage/replayer:20240417-190717
ycyanvhh0jnt swh_db-storage replicated 1/1 (max 1 per node) postgres:13
qlaf9tcyimz7 swh_db-web replicated 1/1 (max 1 per node) postgres:13
aouw9j8uovr2 swh_grafana replicated 1/1 (max 1 per node) grafana/grafana:latest
uwqe13udgyqt swh_graph-replayer replicated 0/0 softwareheritage/replayer:20220225-101454
uwqe13udgyqt swh_graph-replayer replicated 0/0 softwareheritage/replayer:20240417-190717
mepbxllcxctu swh_memcache replicated 1/1 memcached:latest
kfzirv0h298h swh_nginx global 3/3 nginx:latest *:5081->5081/tcp
t7med8frg9pr swh_objstorage replicated 2/2 softwareheritage/base:20220225-101454
t7med8frg9pr swh_objstorage replicated 2/2 softwareheritage/base:20240417-190717
5s34wzo29ukl swh_prometheus replicated 1/1 (max 1 per node) prom/prometheus:latest
rwom7r3yv5ql swh_prometheus-statsd-exporter replicated 1/1 (max 1 per node) prom/statsd-exporter:latest
wuwydthechea swh_redis replicated 1/1 (max 1 per node) redis:6.2.6
jztolbmjp1vi swh_storage replicated 2/2 softwareheritage/base:20220225-101454
xxc4c66x0uj1 swh_web replicated 1/1 softwareheritage/web:20220225-101454
jztolbmjp1vi swh_storage replicated 2/2 softwareheritage/base:20240417-190717
xxc4c66x0uj1 swh_web replicated 1/1 softwareheritage/web:20240417-190717
This will start a series of containers with:
......@@ -247,6 +256,8 @@ This will start a series of containers with:
- a swh_content-replayer service (initially set to 0 replica, see below)
- a swh_graph-replayer service (initially set to 0 replica, see below)
- a redis for the replication error logs,
- a set of services for the vault,
- a set of services for the search (including a single node elasticsearch)
using the pinned version of the docker images.
......@@ -436,7 +447,7 @@ You can check everything is running with:
swh:~/swh-mirror$ docker service ps swh_graph-replayer
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
ioyt34ok118a swh_graph-replayer.1 softwareheritage/replayer:20220225-101454 node1 Running Running 17 minutes ago
ioyt34ok118a swh_graph-replayer.1 softwareheritage/replayer:20240417-190717 node1 Running Running 17 minutes ago
If everything is OK, you should have your mirror filling. Check docker logs:
......
docs/sysadm/mirror-operations/index.rst
View file @ 6376a8f6
......@@ -24,6 +24,8 @@ proper mirror should also allow to:
- navigate the archive copy using a Web browser and/or the Web API (typically
using the :ref:`the web application <swh-web>`),
- have minimal search capabilities (typically using :ref:`the swh-search
service <swh-search>` with an Elasticsearch backend),
- retrieve data from the copy of the archive (typically using the :ref:`the
vault service <swh-vault>`)
......@@ -41,9 +43,23 @@ blob objects (file content) from the |swh| :ref:`object storage <swh-objstorage>
General view of the |swh| mirroring architecture.
.. Note:: This general view is very simplified and does not show all the
services involved in hosting and operating a mirror.
See the :ref:`planning-a-mirror` for a complete description of the requirements
to host a mirror.
.. Note:: Hosting a complete mirror is a complex task, involving the deployment
of dozens of inter related (micro-)services. It should be planned and
operated carefully, using state-of-art ops practices (cloud-based, or
using container orchestration tools on an elastic execution platform
like kubernetes_, `docker swarm`_, or using tools like Ansible_ or
`Salt Stack`_).
.. Important:: It is **strongly** recommended to start with a simple `docker
swarm`_ based deployment (this can be done on a single machine) as
described in :ref:`mirror_deploy`.
Mirroring the Graph Storage
~~~~~~~~~~~~~~~~~~~~~~~~~~~
......@@ -110,7 +126,11 @@ Installation
When using the |swh| software stack to deploy a mirror, a number of |swh|
software components must be installed (cf. architecture diagram above).
A `docker-swarm <https://docs.docker.com/engine/swarm/>`_ based deployment
.. Note:: It is **not** recommended to try to deploy each |swh| service
individually. You should rather start from the example docker-based
deployment project :ref:`described here <mirror_docker>`.
A `docker swarm`_ based deployment
solution is provided as a working example of the mirror stack,
see :ref:`mirror_deploy`.
......@@ -119,6 +139,10 @@ production-like deployment.
.. _Kafka: https://kafka.apache.org/
.. _msgpack: https://msgpack.org
.. _`docker swarm`: https://docs.docker.com/engine/swarm/
.. _Ansible: https://docs.ansible.com/ansible/latest/index.html
.. _kubernetes: https://kubernetes.io/fr/
.. _`Salt Stack`: https://saltproject.io/
You may want to read:
......