backup: copy the dali snapshots to the azure's backup vm (!484) · Merge requests · Platform / Infrastructure / Puppet / puppet-swh-site

Vincent Sellier requested to merge generated-differential-D7235-source into generated-differential-D7235-target Feb 24, 2022

support the --no-sync-snap to only transfer the snapshot created by sanoid
support the target dataset override because it's not possible to put a / in the yaml key

Related to T3889

Test Plan

dali:
- only collected ressources not detected by octocatalog-diff

*** Running octocatalog-diff on host dali.internal.admin.swh.network
I, [2022-02-24T09:07:23.834782 #4106756]  INFO -- : Catalogs compiled for dali.internal.admin.swh.network
I, [2022-02-24T09:07:24.176202 #4106756]  INFO -- : Diffs computed for dali.internal.admin.swh.network
I, [2022-02-24T09:07:24.176239 #4106756]  INFO -- : No differences
*** End octocatalog-diff on dali.internal.admin.swh.network

backup01:

diff origin/production/backup01.euwest.azure.internal.softwareheritage.org current/backup01.euwest.azure.internal.softwareheritage.org
*******************************************
  Apt::Setting[list-backports] =>
   parameters =>
     content =>
      @@ -1,3 +1,3 @@
       # This file is managed by Puppet. DO NOT EDIT.
       # backports
      -deb http://deb.debian.org/debian/ bullseye-backports main
      +deb http://deb.debian.org/debian/ bullseye-backports main contrib non-free
*******************************************
  Apt::Setting[list-debian-security] =>
   parameters =>
     content =>
      @@ -1,3 +1,3 @@
       # This file is managed by Puppet. DO NOT EDIT.
       # debian-security
      -deb http://deb.debian.org/debian-security/ bullseye-security main
      +deb http://deb.debian.org/debian-security/ bullseye-security main contrib non-free
*******************************************
  Apt::Setting[list-debian-updates] =>
   parameters =>
     content =>
      @@ -1,3 +1,3 @@
       # This file is managed by Puppet. DO NOT EDIT.
       # debian-updates
      -deb http://deb.debian.org/debian/ bullseye-updates main
      +deb http://deb.debian.org/debian/ bullseye-updates main contrib non-free
*******************************************
  Apt::Setting[list-debian] =>
   parameters =>
     content =>
      @@ -1,3 +1,3 @@
       # This file is managed by Puppet. DO NOT EDIT.
       # debian
      -deb http://deb.debian.org/debian/ bullseye main
      +deb http://deb.debian.org/debian/ bullseye main contrib non-free
*******************************************
  Apt::Source[backports] =>
   parameters =>
     repos =>
      - main
      + main contrib non-free
*******************************************
  Apt::Source[debian-security] =>
   parameters =>
     repos =>
      - main
      + main contrib non-free
*******************************************
  Apt::Source[debian-updates] =>
   parameters =>
     repos =>
      - main
      + main contrib non-free
*******************************************
  Apt::Source[debian] =>
   parameters =>
     repos =>
      - main
      + main contrib non-free
*******************************************
  File[/etc/apt/sources.list.d/backports.list] =>
   parameters =>
     content =>
      @@ -1,3 +1,3 @@
       # This file is managed by Puppet. DO NOT EDIT.
       # backports
      -deb http://deb.debian.org/debian/ bullseye-backports main
      +deb http://deb.debian.org/debian/ bullseye-backports main contrib non-free
*******************************************
  File[/etc/apt/sources.list.d/debian-security.list] =>
   parameters =>
     content =>
      @@ -1,3 +1,3 @@
       # This file is managed by Puppet. DO NOT EDIT.
       # debian-security
      -deb http://deb.debian.org/debian-security/ bullseye-security main
      +deb http://deb.debian.org/debian-security/ bullseye-security main contrib non-free
*******************************************
  File[/etc/apt/sources.list.d/debian-updates.list] =>
   parameters =>
     content =>
      @@ -1,3 +1,3 @@
       # This file is managed by Puppet. DO NOT EDIT.
       # debian-updates
      -deb http://deb.debian.org/debian/ bullseye-updates main
      +deb http://deb.debian.org/debian/ bullseye-updates main contrib non-free
*******************************************
  File[/etc/apt/sources.list.d/debian.list] =>
   parameters =>
     content =>
      @@ -1,3 +1,3 @@
       # This file is managed by Puppet. DO NOT EDIT.
       # debian
      -deb http://deb.debian.org/debian/ bullseye main
      +deb http://deb.debian.org/debian/ bullseye main contrib non-free
*******************************************
+ File[/etc/systemd/system/syncoid-dali-postgresql.service] =>
   parameters =>
     "content": "# Managed by puppet class profile::sanoid::syncoid_destination\n...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": "Class[Systemd::Systemctl::Daemon_reload]",
     "owner": "root",
     "show_diff": true
*******************************************
+ File[/etc/systemd/system/syncoid-dali-postgresql.timer] =>
   parameters =>
     "content": "# Managed by puppet class profile::sanoid::syncoid_destination\n...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": [
       "Class[Systemd::Systemctl::Daemon_reload]",
       "Service[syncoid-dali-postgresql.timer]"
     ],
     "owner": "root",
     "show_diff": true
*******************************************
+ File[/etc/systemd/system/syncoid-dali-postgresql_wal.service] =>
   parameters =>
     "content": "# Managed by puppet class profile::sanoid::syncoid_destination\n...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": "Class[Systemd::Systemctl::Daemon_reload]",
     "owner": "root",
     "show_diff": true
*******************************************
+ File[/etc/systemd/system/syncoid-dali-postgresql_wal.timer] =>
   parameters =>
     "content": "# Managed by puppet class profile::sanoid::syncoid_destination\n...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": [
       "Class[Systemd::Systemctl::Daemon_reload]",
       "Service[syncoid-dali-postgresql_wal.timer]"
     ],
     "owner": "root",
     "show_diff": true
*******************************************
+ File[/root/.ssh/id_ed25519.syncoid_backup01-azure] =>
   parameters =>
     "content": "syncoid::ssh_key::backup01-azure",
     "ensure": "present",
     "group": "root",
     "mode": "0600",
     "owner": "root"
*******************************************
+ Package[linux-headers-cloud-amd64] =>
   parameters =>
     "ensure": "present"
*******************************************
+ Package[sanoid] =>
   parameters =>
     "ensure": "installed"
*******************************************
+ Package[zfs-dkms] =>
   parameters =>
     "ensure": "present"
*******************************************
+ Service[syncoid-dali-postgresql.timer] =>
   parameters =>
     "enable": true,
     "ensure": true,
     "provider": "systemd"
*******************************************
+ Service[syncoid-dali-postgresql_wal.timer] =>
   parameters =>
     "enable": true,
     "ensure": true,
     "provider": "systemd"
*******************************************
+ Systemd::Timer[syncoid-dali-postgresql.timer] =>
   parameters =>
     "active": true,
     "enable": true,
     "ensure": "present",
     "group": "root",
     "mode": "0444",
     "owner": "root",
     "path": "/etc/systemd/system",
     "service_content": "# Managed by puppet class profile::sanoid::syncoid_desti...
     "service_unit": "syncoid-dali-postgresql.service",
     "show_diff": true,
     "timer_content": "# Managed by puppet class profile::sanoid::syncoid_destina...
*******************************************
+ Systemd::Timer[syncoid-dali-postgresql_wal.timer] =>
   parameters =>
     "active": true,
     "enable": true,
     "ensure": "present",
     "group": "root",
     "mode": "0444",
     "owner": "root",
     "path": "/etc/systemd/system",
     "service_content": "# Managed by puppet class profile::sanoid::syncoid_desti...
     "service_unit": "syncoid-dali-postgresql_wal.service",
     "show_diff": true,
     "timer_content": "# Managed by puppet class profile::sanoid::syncoid_destina...
*******************************************
+ Systemd::Unit_file[syncoid-dali-postgresql.service] =>
   parameters =>
     "content": "# Managed by puppet class profile::sanoid::syncoid_destination\n...
     "ensure": "present",
     "group": "root",
     "mode": "0444",
     "owner": "root",
     "path": "/etc/systemd/system",
     "show_diff": true
*******************************************
+ Systemd::Unit_file[syncoid-dali-postgresql.timer] =>
   parameters =>
     "active": true,
     "content": "# Managed by puppet class profile::sanoid::syncoid_destination\n...
     "enable": true,
     "ensure": "present",
     "group": "root",
     "mode": "0444",
     "owner": "root",
     "path": "/etc/systemd/system",
     "show_diff": true
*******************************************
+ Systemd::Unit_file[syncoid-dali-postgresql_wal.service] =>
   parameters =>
     "content": "# Managed by puppet class profile::sanoid::syncoid_destination\n...
     "ensure": "present",
     "group": "root",
     "mode": "0444",
     "owner": "root",
     "path": "/etc/systemd/system",
     "show_diff": true
*******************************************
+ Systemd::Unit_file[syncoid-dali-postgresql_wal.timer] =>
   parameters =>
     "active": true,
     "content": "# Managed by puppet class profile::sanoid::syncoid_destination\n...
     "enable": true,
     "ensure": "present",
     "group": "root",
     "mode": "0444",
     "owner": "root",
     "path": "/etc/systemd/system",
     "show_diff": true
*******************************************
*** End octocatalog-diff on backup01.euwest.azure.internal.softwareheritage.org

db1.staging:
- just a new space due to the optional parameter

diff origin/production/db1.internal.staging.swh.network current/db1.internal.staging.swh.network
*******************************************
  File[/etc/systemd/system/syncoid-storage1-kafka.service] =>
   parameters =>
     content =>
      @@ -11,5 +11,5 @@
       User=root
       Group=root
      -ExecStart=syncoid --sshkey /root/.ssh/id_ed25519.syncoid_db1 root@storage1.internal.staging.swh.network:data/kafka data/sync/storage1/kafka
      +ExecStart=syncoid  --sshkey /root/.ssh/id_ed25519.syncoid_db1 root@storage1.internal.staging.swh.network:data/kafka data/sync/storage1/kafka
       Restart=on-failure
       RestartSec=1s
*******************************************
  File[/etc/systemd/system/syncoid-storage1-objects.service] =>
   parameters =>
     content =>
      @@ -11,5 +11,5 @@
       User=root
       Group=root
      -ExecStart=syncoid --sshkey /root/.ssh/id_ed25519.syncoid_db1 root@storage1.internal.staging.swh.network:data/objects data/sync/storage1/objects
      +ExecStart=syncoid  --sshkey /root/.ssh/id_ed25519.syncoid_db1 root@storage1.internal.staging.swh.network:data/objects data/sync/storage1/objects
       Restart=on-failure
       RestartSec=1s
*******************************************
  Systemd::Timer[syncoid-storage1-kafka.timer] =>
   parameters =>
     service_content =>
      @@ -11,5 +11,5 @@
       User=root
       Group=root
      -ExecStart=syncoid --sshkey /root/.ssh/id_ed25519.syncoid_db1 root@storage1.internal.staging.swh.network:data/kafka data/sync/storage1/kafka
      +ExecStart=syncoid  --sshkey /root/.ssh/id_ed25519.syncoid_db1 root@storage1.internal.staging.swh.network:data/kafka data/sync/storage1/kafka
       Restart=on-failure
       RestartSec=1s
*******************************************
  Systemd::Timer[syncoid-storage1-objects.timer] =>
   parameters =>
     service_content =>
      @@ -11,5 +11,5 @@
       User=root
       Group=root
      -ExecStart=syncoid --sshkey /root/.ssh/id_ed25519.syncoid_db1 root@storage1.internal.staging.swh.network:data/objects data/sync/storage1/objects
      +ExecStart=syncoid  --sshkey /root/.ssh/id_ed25519.syncoid_db1 root@storage1.internal.staging.swh.network:data/objects data/sync/storage1/objects
       Restart=on-failure
       RestartSec=1s
*******************************************
  Systemd::Unit_file[syncoid-storage1-kafka.service] =>
   parameters =>
     content =>
      @@ -11,5 +11,5 @@
       User=root
       Group=root
      -ExecStart=syncoid --sshkey /root/.ssh/id_ed25519.syncoid_db1 root@storage1.internal.staging.swh.network:data/kafka data/sync/storage1/kafka
      +ExecStart=syncoid  --sshkey /root/.ssh/id_ed25519.syncoid_db1 root@storage1.internal.staging.swh.network:data/kafka data/sync/storage1/kafka
       Restart=on-failure
       RestartSec=1s
*******************************************
  Systemd::Unit_file[syncoid-storage1-objects.service] =>
   parameters =>
     content =>
      @@ -11,5 +11,5 @@
       User=root
       Group=root
      -ExecStart=syncoid --sshkey /root/.ssh/id_ed25519.syncoid_db1 root@storage1.internal.staging.swh.network:data/objects data/sync/storage1/objects
      +ExecStart=syncoid  --sshkey /root/.ssh/id_ed25519.syncoid_db1 root@storage1.internal.staging.swh.network:data/objects data/sync/storage1/objects
       Restart=on-failure
       RestartSec=1s
*******************************************
*** End octocatalog-diff on db1.internal.staging.swh.network

Migrated from D7235 (view on Phabricator)

backup: copy the dali snapshots to the azure's backup vm

Test Plan

Merge request reports