Enable the swh-search environment in staging

Actions:

• Deploy a dedicated elasticsearch node (1 node to start, we'll see if a cluster is needed)

• Deploy a rpc search service

• webapp.staging uses the search service

• Deploy a search journal client (read from kafka / Write to ES)

• Deploy a search journal client for indexed metadata topic [1]

• [1] Deployed but currently stuck due to #2876 (closed)

---

Migrated from T2817 (view on Phabricator)

marked this issue as related to #2590 (closed)

added Archive search Journal Staging environment System administrators priority:Normal labels

changed title from Enable the swh-search in staging to Enable the swh-search environment in staging

changed the description

added state:wip label

mentioned in commit swh/infra/puppet/puppet-swh-site@06327419

mentioned in commit swh/infra/puppet/puppet-environment@017e1cd6

mentioned in commit swh/infra/puppet/puppet-environment@de7c399e

mentioned in commit swh/infra/puppet/puppet-environment@8592fd52

mentioned in commit swh/infra/puppet/puppet-environment@2dd0ddba

mentioned in commit swh/infra/swh-sysadmin-provisioning@6273f496

changed the description

dedicated ES node for staging deployed (search-esnode0.internal.staging.swh.network) with swh/infra/swh-sysadmin-provisioning!45 (closed) and swh/infra/puppet/puppet-swh-site!270 (closed)

mentioned in commit swh/infra/puppet/puppet-swh-site@cd37882b

mentioned in commit swh/infra/swh-sysadmin-provisioning@5cff2306

mentioned in commit swh/infra/swh-sysadmin-provisioning@e95a5d80

mentioned in commit swh/infra/puppet/puppet-swh-site@97ce4d5d

We added a volume of 100Gib to the search-esnode0 through terraform (swh/infra/swh-sysadmin-provisioning!46 (closed)). So we could mount the /srv/elasticsearch as zfs volume.

root@search-esnode0:~# apt install linux-headers-`uname -r`  # otherwise, zfs-dkms install is not happy
root@search-esnode0:~# apt install zfs-dkms
## reboot

root@search-esnode0:~# systemctl stop kafka
root@search-esnode0:~# rm -rf /srv/kafka/logdir/*

root@search-esnode0:~# zpool create -f elasticsearch-volume -m /srv/kafka/logdir  /dev/vdb
root@search-esnode0:~# zfs set relatime=on elasticsearch-volume

root@search-esnode0:~# zpool list
NAME                   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
elasticsearch-volume  99.5G   184K  99.5G        -         -     0%     0%  1.00x    ONLINE  -

root@search-esnode0:~# puppet agent --test  # to rectify the permissions adequately
Info: Loading facts                                                                                                                                                                                                                    [27/463]
Info: Caching catalog for search-esnode0.internal.staging.swh.network
Info: Applying configuration version '1607081929'
Notice: /Stage[main]/Profile::Elasticsearch/File[/srv/elasticsearch]/owner: owner changed 'root' to 'elasticsearch'
Notice: /Stage[main]/Profile::Elasticsearch/File[/srv/elasticsearch]/group: group changed 'root' to 'elasticsearch'
Notice: /Stage[main]/Profile::Elasticsearch/File[/srv/elasticsearch]/mode: mode changed '0755' to '2755'
Notice: /Stage[main]/Profile::Elasticsearch/Service[elasticsearch]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Profile::Elasticsearch/Service[elasticsearch]: Unscheduling refresh on Service[elasticsearch]
Notice: Applied catalog in 15.19 seconds
root@search-esnode0:~#
root@search-esnode0:~# systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/elasticsearch.service.d
           └─elasticsearch.conf
   Active: active (running) since Fri 2020-12-04 11:39:08 UTC; 4s ago
     Docs: https://www.elastic.co

# reboot to be sure

# status: everything is fine!

mentioned in commit swh/infra/puppet/puppet-environment@725cfe44

mentioned in commit swh/infra/puppet/puppet-swh-site@73f427c8