Project 'infra/sysadm-environment' was moved to 'swh/infra/sysadm-environment'. Please update any links and bookmarks that may still have the old path.
Deploy visit-stats journal client on production
- Stop the loaders (so they won't fail and consume too much messages once the storage is stopped)
- Stop the storage service
- Upgrade the storage storage server to last version of swh-storage/swh-model/swh-journal
- Upgrade the storage database model
- Start the storage service
- Upgrade the replica db model so replication continues
- Upgrade and start the loaders
- Upgrade scheduler stack to last versions created during the sprint
- Stop swh-scheduler-runner to avoid blocking queries
- Upgrade the scheduler database model
- Restart scheduler service (gunicorn-swh-scheduler, swh-scheduler-runner, ...)
-
Create consumer group
swh.scheduler.journal_clientso it starts at the end of the topics - Deploy a new scheduler journal-client service on saatchi (which must start at the end of the topic)
- Launch a backfill of the origin_visit_status topic
Migrated from T2993 (view on Phabricator)
Activity
marked this issue as related to swh/devel/swh-scheduler#2444
added Scheduling utilities System administration priority:High labels
- Stop the workers:
$ clush -b -w @swh-workers 'puppet agent --disable "Deploy new storage version"; cd /etc/systemd/system/multi-user.target.wants; for unit in swh-worker@*; do systemctl disable $unit; done; systemctl stop swh-worker@*'- Upgrade saam backend to latest storage/model/scheduler versions:
# apt ... # dpkg -l python3-swh.model python3-swh.storage python3-swh.scheduler python3-swh.journal Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-=====================-=====================-============-=================================== ii python3-swh.journal 0.6.2-1~swh1~bpo10+1 all Software Heritage Journal utilities ii python3-swh.model 0.11.0-1~swh1~bpo10+1 all Software Heritage data model ii python3-swh.scheduler 0.9.2-1~swh1~bpo10+1 all Software Heritage Scheduler ii python3-swh.storage 0.21.0-1~swh1~bpo10+1 all Software Heritage storage utilities- Upgrade db from 164 to 166:
$ psql service=swh softwareheritage=> \conninfo You are connected to database "softwareheritage" as user "swhstorage" on host "belvedere.internal.softwareheritage.org" (address "192.168.100.210") at port "5432". SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off) softwareheritage=> select dbversion from dbversion order by dbversion desc limit 3; dbversion ---------------------------------------------------------- (166,"2021-01-26 09:28:10.658088+00","Work In Progress") (165,"2021-01-26 09:28:10.658088+00","Work In Progress") (164,"2020-11-04 14:00:07.29092+00","Work In Progress")- Ensure replication is fine
$ psql service=mirror-swh softwareheritage=> \conninfo You are connected to database "softwareheritage" as user "guest" on host "somerset.internal.softwareheritage.org" (address "192.168.100.103") at port "5432". SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off) softwareheritage=> select dbversion from dbversion order by dbversion desc limit 3; dbversion --------------------------------------------------------- (166,"2021-01-26 09:43:57.94437+00","Work In Progress") (165,"2021-01-26 09:43:10.95685+00","Work In Progress") (164,"2020-12-02 10:17:09.47653+00","Work In Progress")-
Upgrade swh-workers with latest swh versions.
-
Restart those
(break, deposit meeting ;)
-
- stop scheduler runner to avoid blocking queries:
systemctl stop swh-scheduler-runner- upgrade scheduler stack
# apt ...- Upgrade scheduler db model:
softwareheritage-scheduler=> \conninfo You are connected to database "softwareheritage-scheduler" as user "swhscheduler" on host "belvedere.internal.softwareheritage.org" (address "192.168.100.210") at port "5432". SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off) softwareheritage-scheduler=> select dbversion from dbversion order by dbversion desc limit 8; dbversion --------------------------------------------------------- (25,"2021-01-26 11:42:29.918304+00","Work In Progress") (24,"2021-01-26 11:42:15.917849+00","Work In Progress") (23,"2021-01-26 11:42:05.066391+00","Work In Progress") (20,"2021-01-26 11:41:02.493991+00","Work In Progress") (19,"2021-01-26 11:40:39.74381+00","Work In Progress") (18,"2021-01-26 11:40:32.628425+00","Work In Progress") (17,"2021-01-26 11:30:31.575419+00","Work In Progress") (16,"2020-06-22 18:00:33.852039+00","Work In Progress") (8 rows)- restart scheduler services
# systemctl restart gunicorn-swh-scheduler swh-scheduler-runner swh-scheduler-listener(break food time ¯_(ツ)_/¯)
added state:wip label
-
- Create consumer group swh.scheduler.journal_client so it starts at the end of the topics
root@kafka1# cd /opt/kafka/bin root@kafka1# export SERVER=kafka1.internal.softwareheritage.org:9092 root@kafka1:/opt/kafka/bin# ./kafka-consumer-groups.sh --bootstrap-server $SERVER --group swh.scheduler.journal_client \ > --topic swh.journal.objects.origin_visit_status --reset-offsets --to-latest --execute GROUP TOPIC PARTITION NEW-OFFSET swh.scheduler.journal_client swh.journal.objects.origin_visit_status 109 5087970 swh.scheduler.journal_client swh.journal.objects.origin_visit_status 26 5080073 swh.scheduler.journal_client swh.journal.objects.origin_visit_status 201 5079810 swh.scheduler.journal_client swh.journal.objects.origin_visit_status 238 5076767 swh.scheduler.journal_client swh.journal.objects.origin_visit_status 12 5080050 swh.scheduler.journal_client swh.journal.objects.origin_visit_status 231 5085073 swh.scheduler.journal_client swh.journal.objects.origin_visit_status 140 5078655 swh.scheduler.journal_client swh.journal.objects.origin_visit_status 35 5079412 swh.scheduler.journal_client swh.journal.objects.origin_visit_status 10 5080078 swh.scheduler.journal_client swh.journal.objects.origin_visit_status 115 5083860 swh.scheduler.journal_client swh.journal.objects.origin_visit_status 243 5080515 swh.scheduler.journal_client swh.journal.objects.origin_visit_status 99 5086378 swh.scheduler.journal_client swh.journal.objects.origin_visit_status 215 5080377 swh.scheduler.journal_client swh.journal.objects.origin_visit_status 208 5082540 swh.scheduler.journal_client swh.journal.objects.origin_visit_status 143 5082577 swh.scheduler.journal_client swh.journal.objects.origin_visit_status 178 5081556 swh.scheduler.journal_client swh.journal.objects.origin_visit_status 39 5079159 swh.scheduler.journal_client swh.journal.objects.origin_visit_status 63 5082302 swh.scheduler.journal_client swh.journal.objects.origin_visit_status 194 5081268 ... root@kafka1:/opt/kafka/bin# ./kafka-consumer-groups.sh --bootstrap-server $SERVER --list swh.search.journal_client KMOffsetCache-getty swh.indexer.journal_client swh.scheduler.journal_client root@kafka1:/opt/kafka/bin# ./kafka-consumer-groups.sh --bootstrap-server $SERVER --describe --group swh.scheduler.journal_client Consumer group 'swh.scheduler.journal_client' has no active members. GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID swh.scheduler.journal_client swh.journal.objects.origin_visit_status 134 5082134 5082144 10 - - - swh.scheduler.journal_client swh.journal.objects.origin_visit_status 167 5082487 5082499 12 - - - swh.scheduler.journal_client swh.journal.objects.origin_visit_status 200 5082963 5082976 13 - - - swh.scheduler.journal_client swh.journal.objects.origin_visit_status 233 5080371 5080384 13 - - - swh.scheduler.journal_client swh.journal.objects.origin_visit_status 18 5080655 5080667 12 - - - swh.scheduler.journal_client swh.journal.objects.origin_visit_status 51 5086240 5086252 12 - - - swh.scheduler.journal_client swh.journal.objects.origin_visit_status 84 5080830 5080839 9 - - - swh.scheduler.journal_client swh.journal.objects.origin_visit_status 117 5078954 5078967 13 - - - swh.scheduler.journal_client swh.journal.objects.origin_visit_status 150 5081221 5081237 16 - - - swh.scheduler.journal_client swh.journal.objects.origin_visit_status 183 5079272 5079282 10 - - - ...- Deploy a new scheduler journal-client service on saatchi (which must start at the end of the topic)
root@saatchi:~# puppet agent --enable; puppet agent --test Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Retrieving locales Info: Loading facts Info: Caching catalog for saatchi.internal.softwareheritage.org Info: Applying configuration version '1611667435' Notice: /Stage[main]/Profile::Swh::Deploy::Journal/File[/etc/softwareheritage/journal]/ensure: created Notice: /Stage[main]/Profile::Swh::Deploy::Scheduler::Journal_client/File[/etc/softwareheritage/scheduler/journal-client.yml]/ensure: defined content as '{md5}b007df0fe6dd8fc60558eeb60c71c83d' Info: /Stage[main]/Profile::Swh::Deploy::Scheduler::Journal_client/File[/etc/softwareheritage/scheduler/journal-client.yml]: Scheduling refresh of Service[swh-scheduler-journal-client] Notice: /Stage[main]/Profile::Swh::Deploy::Scheduler::Journal_client/Systemd::Unit_file[swh-scheduler-journal-client.service]/File[/etc/systemd/system/swh-scheduler-journal-client.service]/ensure: defined content as '{md5}7958a9db226f3b0d774df2ebb512f350' Info: /Stage[main]/Profile::Swh::Deploy::Scheduler::Journal_client/Systemd::Unit_file[swh-scheduler-journal-client.service]/File[/etc/systemd/system/swh-scheduler-journal-client.service]: Scheduling refresh of Class[Systemd::Systemctl::Daemon_reload] Info: Systemd::Unit_file[swh-scheduler-journal-client.service]: Scheduling refresh of Service[swh-scheduler-journal-client] Notice: /Stage[main]/Profile::Swh::Deploy::Scheduler::Journal_client/Service[swh-scheduler-journal-client]/ensure: ensure changed 'stopped' to 'running' Info: /Stage[main]/Profile::Swh::Deploy::Scheduler::Journal_client/Service[swh-scheduler-journal-client]: Unscheduling refresh on Service[swh-scheduler-journal-client] Info: Class[Systemd::Systemctl::Daemon_reload]: Scheduling refresh of Exec[systemctl-daemon-reload] Notice: /Stage[main]/Systemd::Systemctl::Daemon_reload/Exec[systemctl-daemon-reload]: Triggered 'refresh' from 1 event Notice: Applied catalog in 14.32 seconds root@saatchi:~#- Check the lag subsides thanks to the new deployed (scheduler) journal client consuming the topic:
root@kafka1:/opt/kafka/bin# ./kafka-consumer-groups.sh --bootstrap-server $SERVER --describe --group swh.scheduler.journal_client | head GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID swh.scheduler.journal_client swh.journal.objects.origin_visit_status 109 5088064 5088064 0 rdkafka-39edd9ab-33ce-4de2-9dad-9cac7fa0e0fe /192.168.100.104 rdkafka swh.scheduler.journal_client swh.journal.objects.origin_visit_status 26 5080170 5080170 0 rdkafka-39edd9ab-33ce-4de2-9dad-9cac7fa0e0fe /192.168.100.104 rdkafka swh.scheduler.journal_client swh.journal.objects.origin_visit_status 201 5079903 5079903 0 rdkafka-39edd9ab-33ce-4de2-9dad-9cac7fa0e0fe /192.168.100.104 rdkafka swh.scheduler.journal_client swh.journal.objects.origin_visit_status 238 5076872 5076872 0 rdkafka-39edd9ab-33ce-4de2-9dad-9cac7fa0e0fe /192.168.100.104 rdkafka swh.scheduler.journal_client swh.journal.objects.origin_visit_status 12 5080135 5080135 0 rdkafka-39edd9ab-33ce-4de2-9dad-9cac7fa0e0fe /192.168.100.104 rdkafka swh.scheduler.journal_client swh.journal.objects.origin_visit_status 231 5085161 5085161 0 rdkafka-39edd9ab-33ce-4de2-9dad-9cac7fa0e0fe /192.168.100.104 rdkafka swh.scheduler.journal_client swh.journal.objects.origin_visit_status 140 5078748 5078748 0 rdkafka-39edd9ab-33ce-4de2-9dad-9cac7fa0e0fe /192.168.100.104 rdkafka swh.scheduler.journal_client swh.journal.objects.origin_visit_status 35 5079492 5079493 1 rdkafka-39edd9ab-33ce-4de2-9dad-9cac7fa0e0fe /192.168.100.104 rdkafka ...- Check the new visit-stats table is populated along the way
$ psql service=swh-scheduler -c 'select now() , count(*) from origin_visit_stats'; now | count -------------------------------+------- 2021-01-26 13:27:37.053621+00 | 12682 (1 row) $ psql service=swh-scheduler -c 'select now() , count(*) from origin_visit_stats'; now | count ------------------------------+------- 2021-01-26 13:27:40.89528+00 | 12719 (1 row)
Please register or sign in to reply