Skip to content
Snippets Groups Projects
New look: Off

Borgmatic service alerts since bookworm migration

    • Closed Issue created by Guillaume Samson @guillaume

    Here are the servers where borg is installed:

    root@pergamon:~# clush -b -w @all "dpkg -l borg{backup,matic} 2> /dev/null | grep '^ii'"
clush: cassandra[01-13],esnode[1-3,7-9],getty,giverny,jenkins-docker[01-02],kafka[1-4],kelvingrove,kibana0,logstash0,ns0.euwest.azure,rancher-node-highmem[01-02],rancher-node-metal[01-03],rancher-node-production-rke2-mgmt[1-3],search-esnode[4-6],thanos-compact.euwest.azure,thyssen (44): exited with exit code 1
---------------
banco,branly,chaillot,counters1,moma,mucem,pergamon,rancher-node-metal04,saam,saatchi,tate,uffizi (12)
---------------
ii  borgbackup     1.2.4-1      amd64        deduplicating and compressing backup program
ii  borgmatic      1.7.7-1      all          automatically create, prune and verify backups with borgbackup

    The borgmatic package in bookworm install a systemd service and timer:

    root@uffizi:~# lsb_release -c
No LSB modules are available.
Codename:       bookworm
root@uffizi:~# dpkg -L borgmatic | awk '/borgmatic\.(timer|service)/'
/lib/systemd/system/borgmatic.service
/lib/systemd/system/borgmatic.timer
    root@dali:~# lsb_release -c
Codename:       bullseye
root@dali:~# dpkg -L borgmatic | awk '/borgmatic\.(timer|service)/'

    Previously borgmatic was managed by cron jobs whivh are still present today:

    root@dali:~# cat /etc/cron.d/puppet-borgmatic 
# Managed by puppet (module profile::cron), manual changes will be lost

# Cron snippet borgmatic-create
51 0,1,2,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23 * * * root borgmatic create
# Cron snippet borgmatic-full
51 3 * * * root borgmatic
    1. The cron task borgmatic-full and the systemd service seems redundant on bookworm servers.
    2. saam and rancher-node-metal04 are not declared in the borg authorized_keys:
    root@banco:~# awk -F '"' '{n=(split($2,a,"/"));print a[n]}'  /srv/borg/.ssh/authorized_keys
albertina.internal.softwareheritage.org
banco.internal.softwareheritage.org
bardo.internal.admin.swh.network
bojimans.internal.admin.swh.network
branly.internal.softwareheritage.org
chaillot.internal.softwareheritage.org
counters0.internal.staging.swh.network
counters1.internal.softwareheritage.org
dali.internal.admin.swh.network
db1.internal.staging.swh.network
massmoca.internal.softwareheritage.org
moma.internal.softwareheritage.org
mucem.internal.softwareheritage.org
pergamon.internal.softwareheritage.org
rp0.internal.staging.swh.network
rp1.internal.admin.swh.network
saatchi.internal.softwareheritage.org
scheduler0.internal.staging.swh.network
tate.internal.softwareheritage.org
uffizi.internal.softwareheritage.org

    so borgmatic (systemd service or cron task) got a permission denied error.
    3. bardo backups have not been cleaned up for a long time:

    root@bardo:~# borgmatic info
ssh://borg@banco.internal.softwareheritage.org/srv/borg/repositories/bardo.internal.admin.swh.network: Displaying archive summary information
Repository ID: 87e59d0c58d6d0855ce3b613b111dd6265773c5f3a8001e0d9d1cab8cb8fafc7
Location: ssh://borg@banco.internal.softwareheritage.org/srv/borg/repositories/bardo.internal.admin.swh.network
Encrypted: Yes (repokey BLAKE2b)
Cache: /var/lib/borg/.cache/borg/87e59d0c58d6d0855ce3b613b111dd6265773c5f3a8001e0d9d1cab8cb8fafc7
Security dir: /var/lib/borg/.config/borg/security/87e59d0c58d6d0855ce3b613b111dd6265773c5f3a8001e0d9d1cab8cb8fafc7
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
All archives:               30.78 TB             17.49 TB             28.55 GB
                       Unique chunks         Total chunks
Chunk index:                  628567           1402382808
    root@bardo:~# borgmatic list | ( head -2 ; printf "...\n" ; tail -2 )
ssh://borg@banco.internal.softwareheritage.org/srv/borg/repositories/bardo.internal.admin.swh.network: Listing archives
bardo.internal.admin.swh.network-2023-12-31T23:12:01.459433 Sun, 2023-12-31 23:12:03 [154d24f40fac1d6f4de025f320fdd4fe66fc71831c0d7f1847d37091ad7e9fc1]
...
bardo.internal.admin.swh.network-2025-03-17T13:12:02.436941 Mon, 2025-03-17 13:12:03 [c828823a2a64c9509f240a116f7d94ae3682d13018f3adb0d20cc25a5cbacaaa]

    so the prune operation systematically fails even with increasing the ssh alive interval 1 and count on both server and client

    1. https://borgbackup.readthedocs.io/en/stable/usage/serve.html#ssh-configuration ↩

    Designs

      Child items ...

      2. Activity

      • Guillaume Samson added activity::MRO label

        added activity::MRO label

      • Guillaume Samson assigned to @guillaume

        assigned to @guillaume

      • Guillaume Samson
        Guillaume Samson @guillaume ·
        Author Owner

        Backing up the bardo archive on banco and recreating an archive directory:

        root@banco:/srv/borg/repositories# ls -trld bardo.internal.admin.swh.network*
drwx------ 3 borg borg 4096 Mar 17 16:05 bardo.internal.admin.swh.network.back_20250317
drwx------ 3 borg borg 4096 Mar 17 16:34 bardo.internal.admin.swh.network

        Re initializing backups on bardo:

        root@bardo:~# borgmatic init --encryption repokey
root@bardo:~# borgmatic create
root@bardo:~# systemctl start borgmatic.service
      • Guillaume Samson
        Guillaume Samson @guillaume ·
        Author Owner

        Removing unconfigured borgmatic service on saam and rancher-node-metal04:

        apt purge borgbackup borgmatic
systemctl reset-failed
rm /etc/borgmatic/config.yaml
rmdir /etc/borgmatic
      • Guillaume Samson
        Guillaume Samson @guillaume ·
        Author Owner

        Here are the old server archives that are no longer backed up:

        root@banco:~# du -sh /srv/borg/repositories/* | grep -v "\($(awk -F '"' '{n=(split($2,a,"/"));print a[n]}'  /srv/borg/.ssh/authorized_keys | while read -r dom; do echo -n "$dom\|";done)END\)$"
29G     /srv/borg/repositories/bardo.internal.admin.swh.network.back_20250317
3.5G    /srv/borg/repositories/bardo.internal.softwareheritage.org
7.1G    /srv/borg/repositories/beaubourg.internal.softwareheritage.org
8.7G    /srv/borg/repositories/belvedere.internal.softwareheritage.org
1.2G    /srv/borg/repositories/bojimans.internal.admin.swh.network.internal.softwareheritage.org
2.0G    /srv/borg/repositories/bojimans.internal.softwareheritage.org
2.5G    /srv/borg/repositories/dbreplica0.euwest.azure.internal.softwareheritage.org
2.1G    /srv/borg/repositories/dbreplica1.euwest.azure.internal.softwareheritage.org
3.2G    /srv/borg/repositories/deposit.internal.staging.swh.network
861M    /srv/borg/repositories/grafana0.internal.softwareheritage.org
17G     /srv/borg/repositories/hypervisor3.internal.softwareheritage.org
1.8G    /srv/borg/repositories/louvre.internal.softwareheritage.org
1.1T    /srv/borg/repositories/met.internal.softwareheritage.org
5.4G    /srv/borg/repositories/migration.internal.softwareheritage.org
12G     /srv/borg/repositories/orsay.internal.softwareheritage.org
19G     /srv/borg/repositories/pompidou.internal.softwareheritage.org
84K     /srv/borg/repositories/rancher-node-metal04.internal.softwareheritage.org
3.2G    /srv/borg/repositories/rp1.internal.softwareheritage.org
4.1G    /srv/borg/repositories/saam.internal.softwareheritage.org
2.3G    /srv/borg/repositories/search0.internal.staging.swh.network
2.0G    /srv/borg/repositories/search1.internal.softwareheritage.org
3.9G    /srv/borg/repositories/somerset.internal.softwareheritage.org
1.2G    /srv/borg/repositories/storage0.euwest.azure.internal.softwareheritage.org
2.6G    /srv/borg/repositories/storage0.internal.staging.swh.network
1.4G    /srv/borg/repositories/storage01.euwest.azure.internal.softwareheritage.org
1.6G    /srv/borg/repositories/storage02.euwest.azure.internal.softwareheritage.org
7.2G    /srv/borg/repositories/storage1.internal.staging.swh.network
1.2G    /srv/borg/repositories/thanos.internal.softwareheritage.org
3.0G    /srv/borg/repositories/vangogh.euwest.azure.internal.softwareheritage.org
1.6G    /srv/borg/repositories/vault.internal.staging.swh.network
2.9G    /srv/borg/repositories/webapp.internal.staging.swh.network
3.1G    /srv/borg/repositories/webapp0.internal.softwareheritage.org
2.9G    /srv/borg/repositories/webapp1.internal.softwareheritage.org
      • Guillaume Samson
        Guillaume Samson @guillaume ·
        Author Owner

        Here are the date of last backup for the servers that are no longer backed up:

        root@banco:~# du -sh /srv/borg/repositories/* | grep -v "\($(awk -F '"' '{n=(split($2,a,"/"));print a[n]}'  /srv/borg/.ssh/authorized_keys | while read -r dom; do echo -n "$dom\|";done)END\)$" | awk '{print $2}' | xargs -I{} sh -c 'ls -trld {}/index.[0-9]*' | awk '{$1=$2=$3=$4=$5="";print}'
     Mar 17 14:14 /srv/borg/repositories/bardo.internal.admin.swh.network.back_20250317/index.97097
     Jul 13 2022 /srv/borg/repositories/bardo.internal.softwareheritage.org/index.56831
     Feb 23 2024 /srv/borg/repositories/beaubourg.internal.softwareheritage.org/index.172055
     Jan 25 2024 /srv/borg/repositories/belvedere.internal.softwareheritage.org/index.148677
     Jan 24 2022 /srv/borg/repositories/bojimans.internal.admin.swh.network.internal.softwareheritage.org/index.6
     Jul 13 2022 /srv/borg/repositories/bojimans.internal.softwareheritage.org/index.64968
     Feb 2 2022 /srv/borg/repositories/dbreplica0.euwest.azure.internal.softwareheritage.org/index.75956
     Jan 23 2023 /srv/borg/repositories/dbreplica1.euwest.azure.internal.softwareheritage.org/index.110713
     Nov 27 2023 /srv/borg/repositories/deposit.internal.staging.swh.network/index.113931
     Jan 14 2022 /srv/borg/repositories/grafana0.internal.softwareheritage.org/index.13
     Feb 13 12:13 /srv/borg/repositories/hypervisor3.internal.softwareheritage.org/index.189353
     Jun 29 2021 /srv/borg/repositories/louvre.internal.softwareheritage.org/index.54035
     Mar 21 2024 /srv/borg/repositories/met.internal.softwareheritage.org/index.93208
     Mar 8 2024 /srv/borg/repositories/migration.internal.softwareheritage.org/index.50401
     Nov 16 2020 /srv/borg/repositories/orsay.internal.softwareheritage.org/index.31756
     Feb 18 16:20 /srv/borg/repositories/pompidou.internal.softwareheritage.org/index.187393
     Jan 25 2024 /srv/borg/repositories/rancher-node-metal04.internal.softwareheritage.org/index.1
     Jul 13 2022 /srv/borg/repositories/rp1.internal.softwareheritage.org/index.55269
     Jan 26 2024 /srv/borg/repositories/saam.internal.softwareheritage.org/index.116724
     Oct 27 2023 /srv/borg/repositories/search0.internal.staging.swh.network/index.103969
     Jan 23 2024 /srv/borg/repositories/search1.internal.softwareheritage.org/index.112491
     Feb 20 2024 /srv/borg/repositories/somerset.internal.softwareheritage.org/index.150201
     Feb 5 2020 /srv/borg/repositories/storage0.euwest.azure.internal.softwareheritage.org/index.2997
     Nov 6 2020 /srv/borg/repositories/storage0.internal.staging.swh.network/index.2771
     Feb 1 2022 /srv/borg/repositories/storage01.euwest.azure.internal.softwareheritage.org/index.72596
     Feb 4 2021 /srv/borg/repositories/storage02.euwest.azure.internal.softwareheritage.org/index.36490
     Jan 9 2024 /srv/borg/repositories/storage1.internal.staging.swh.network/index.122891
     Jul 13 2022 /srv/borg/repositories/thanos.internal.softwareheritage.org/index.107
     Dec 10 2020 /srv/borg/repositories/vangogh.euwest.azure.internal.softwareheritage.org/index.33839
     Dec 10 2020 /srv/borg/repositories/vault.internal.staging.swh.network/index.6177
     Oct 20 2023 /srv/borg/repositories/webapp.internal.staging.swh.network/index.110092
     Jan 4 2021 /srv/borg/repositories/webapp0.internal.softwareheritage.org/index.36598
     Dec 14 2023 /srv/borg/repositories/webapp1.internal.softwareheritage.org/index.108835
      • Guillaume Samson
        Guillaume Samson @guillaume ·
        Author Owner

        After cleaning up the archive directory of servers that hadn't been backed up for over 6 months:

        • here are the current backups:
        root@banco:~# du -sh /srv/borg/repositories/* | grep "\($(awk -F '"' '{n=(split($2,a,"/"));print a[n]}'  /srv/borg/.ssh/authorized_keys | while read -r dom; do echo -n "$dom\|";done)END\)$"
24G     /srv/borg/repositories/albertina.internal.softwareheritage.org
12G     /srv/borg/repositories/banco.internal.softwareheritage.org
3.5G    /srv/borg/repositories/bardo.internal.admin.swh.network
3.1G    /srv/borg/repositories/bojimans.internal.admin.swh.network
18G     /srv/borg/repositories/branly.internal.softwareheritage.org
3.9G    /srv/borg/repositories/chaillot.internal.softwareheritage.org
3.1G    /srv/borg/repositories/counters0.internal.staging.swh.network
3.2G    /srv/borg/repositories/counters1.internal.softwareheritage.org
3.1G    /srv/borg/repositories/dali.internal.admin.swh.network
6.1G    /srv/borg/repositories/db1.internal.staging.swh.network
2.8G    /srv/borg/repositories/massmoca.internal.softwareheritage.org
12G     /srv/borg/repositories/moma.internal.softwareheritage.org
8.8G    /srv/borg/repositories/mucem.internal.softwareheritage.org
105G    /srv/borg/repositories/pergamon.internal.softwareheritage.org
4.6G    /srv/borg/repositories/rp0.internal.staging.swh.network
5.1G    /srv/borg/repositories/rp1.internal.admin.swh.network
5.3G    /srv/borg/repositories/saatchi.internal.softwareheritage.org
4.3G    /srv/borg/repositories/scheduler0.internal.staging.swh.network
21G     /srv/borg/repositories/tate.internal.softwareheritage.org
28G     /srv/borg/repositories/uffizi.internal.softwareheritage.org
        • here are the former (less than 6 months) backups:
        root@banco:~# du -sh /srv/borg/repositories/* | grep -v "\($(awk -F '"' '{n=(split($2,a,"/"));print a[n]}'  /srv/borg/.ssh/authorized_keys | while read -r dom; do echo -n "$dom\|";done)END\)$" | awk '{print $2}' | xargs -I{} sh -c 'ls -trld {}/index.[0-9]*' | awk '{$1=$2=$3=$4=$5="";print}'
     Mar 17 14:14 /srv/borg/repositories/bardo.internal.admin.swh.network.back_20250317/index.97097
     Feb 13 12:13 /srv/borg/repositories/hypervisor3.internal.softwareheritage.org/index.189353
     Feb 18 16:20 /srv/borg/repositories/pompidou.internal.softwareheritage.org/index.187393
        • Guillaume Samson
          Guillaume Samson @guillaume ·
          Author Owner

          After increasing ServerAlive* values, Borgmatic backup on bardo still failed.

          root@bardo:~# cat .ssh/config
Host banco.internal.softwareheritage.org
    ServerAliveInterval 10
    ServerAliveCountMax 360
          root@bardo:~# time borgmatic prune
ssh://borg@banco.internal.softwareheritage.org/srv/borg/repositories/bardo.internal.admin.swh.network: Error running actions for repository
Command 'borg prune --keep-hourly 24 --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --glob-archives bardo.internal.admin.swh.network-* ssh://borg@banco.internal.softwareheritage.org/srv/borg/repositories/bardo.internal.admin.swh.network' returned non-zero exit status 2.
/etc/borgmatic/config.yaml: Error running configuration file

summary:
/etc/borgmatic/config.yaml: Error running configuration file
ssh://borg@banco.internal.softwareheritage.org/srv/borg/repositories/bardo.internal.admin.swh.network: Error running actions for repository
Remote: client_loop: send disconnect: Broken pipe
Connection closed by remote host
Command 'borg prune --keep-hourly 24 --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --glob-archives bardo.internal.admin.swh.network-* ssh://borg@banco.internal.softwareheritage.org/srv/borg/repositories/bardo.internal.admin.swh.network' returned non-zero exit status 2.

Need some help? https://torsion.org/borgmatic/#issues

real    16m39.727s
user    0m59.185s
sys     0m3.656s

          Increasing ClientAlive values on banco sshd configuration.

          root@bardo:~# sshd -T | grep clientalive
clientaliveinterval 0
clientalivecountmax 3
          root@banco:~# vi /etc/ssh/sshd_config
root@banco:~# systemctl restart sshd.service
root@banco:~# sshd -T | grep clientalive
clientaliveinterval 10
clientalivecountmax 360

          Restarting borgmatic.service on bardo.

          root@bardo:~# systemctl start borgmatic.service 
root@bardo:~# systemctl status borgmatic.service 
○ borgmatic.service - borgmatic backup
     Loaded: loaded (/lib/systemd/system/borgmatic.service; static)
     Active: inactive (dead) since Tue 2025-03-25 08:35:55 UTC; 1min 36s ago
TriggeredBy: ● borgmatic.timer
       Docs: https://torsion.org/borgmatic/
    Process: 1672028 ExecStartPre=sleep 1m (code=exited, status=0/SUCCESS)
    Process: 1672053 ExecStart=systemd-inhibit --who=borgmatic --what=sleep:shutdown --why=Prevent interrupting scheduled backup /usr/bin/borgmatic --verbosity -1 --syslog-verbosity 1 (code=exited, status=0/SUCCESS)
   Main PID: 1672053 (code=exited, status=0/SUCCESS)
        CPU: 24.495s

Mar 25 08:35:30 bardo borgmatic[1672054]: INFO Creating archive at "ssh://borg@banco.internal.softwareheritage.org/srv/borg/repositories/bardo.internal.admin.swh.network::bardo.internal.admin.swh.network-2025-03-25T08:35:29.633529"
Mar 25 08:35:53 bardo borgmatic[1672054]: INFO ssh://borg@banco.internal.softwareheritage.org/srv/borg/repositories/bardo.internal.admin.swh.network: Running consistency checks
Mar 25 08:35:55 bardo borgmatic[1672054]: INFO Skipping repository check due to configured frequency; 22 days, 8:03:33.719337 until next check
Mar 25 08:35:55 bardo borgmatic[1672054]: INFO Skipping archives check due to configured frequency; 22 days, 8:03:33.719200 until next check
Mar 25 08:35:55 bardo borgmatic[1672054]: INFO
Mar 25 08:35:55 bardo borgmatic[1672054]: INFO summary:
Mar 25 08:35:55 bardo borgmatic[1672054]: INFO /etc/borgmatic/config.yaml: Successfully ran configuration file
Mar 25 08:35:55 bardo systemd[1]: borgmatic.service: Deactivated successfully.
Mar 25 08:35:55 bardo systemd[1]: Finished borgmatic.service - borgmatic backup.
Mar 25 08:35:55 bardo systemd[1]: borgmatic.service: Consumed 24.495s CPU time.
        • Guillaume Samson
          Guillaume Samson @guillaume ·
          Author Owner

          Borgmatic service on bardo did not failed this night:

          root@bardo:~# systemctl list-timers borgmatic.timer | awk 'NR<=2'
NEXT                        LEFT     LAST                        PASSED      UNIT            ACTIVATES
Thu 2025-03-27 01:16:01 UTC 17h left Wed 2025-03-26 02:42:34 UTC 5h 8min ago borgmatic.timer borgmatic.service
root@bardo:~# systemctl status borgmatic.service
○ borgmatic.service - borgmatic backup
     Loaded: loaded (/lib/systemd/system/borgmatic.service; static)
     Active: inactive (dead) since Wed 2025-03-26 02:44:45 UTC; 5h 6min ago
TriggeredBy: ● borgmatic.timer
       Docs: https://torsion.org/borgmatic/
    Process: 1735734 ExecStartPre=sleep 1m (code=exited, status=0/SUCCESS)
    Process: 1735759 ExecStart=systemd-inhibit --who=borgmatic --what=sleep:shutdown --why=Prevent interrupting scheduled backup /usr/bin/borgmatic --verbosity -1 --syslog-verbosity 1 (code=exited, status=0/SUCCESS)
   Main PID: 1735759 (code=exited, status=0/SUCCESS)
        CPU: 1min 7.875s

Mar 26 02:44:20 bardo borgmatic[1735760]: INFO Creating archive at "ssh://borg@banco.internal.softwareheritage.org/srv/borg/repositories/bardo.internal.admin.swh.network::bardo.internal.admin.swh.network-2025-03-26T02:44:19.893520"
Mar 26 02:44:44 bardo borgmatic[1735760]: INFO ssh://borg@banco.internal.softwareheritage.org/srv/borg/repositories/bardo.internal.admin.swh.network: Running consistency checks
Mar 26 02:44:45 bardo borgmatic[1735760]: INFO Skipping repository check due to configured frequency; 21 days, 13:54:43.512043 until next check
Mar 26 02:44:45 bardo borgmatic[1735760]: INFO Skipping archives check due to configured frequency; 21 days, 13:54:43.511928 until next check
Mar 26 02:44:45 bardo borgmatic[1735760]: INFO
Mar 26 02:44:45 bardo borgmatic[1735760]: INFO summary:
Mar 26 02:44:45 bardo borgmatic[1735760]: INFO /etc/borgmatic/config.yaml: Successfully ran configuration file
Mar 26 02:44:45 bardo systemd[1]: borgmatic.service: Deactivated successfully.
Mar 26 02:44:45 bardo systemd[1]: Finished borgmatic.service - borgmatic backup.
Mar 26 02:44:45 bardo systemd[1]: borgmatic.service: Consumed 1min 7.875s CPU time.

          No borgamtic services error this mormning: borg serve ssh-configuration.

          Puppet agent is disabled on banco to preserve the sshd_config: I will update puppet code to integrate these modifications.

        • Please register or sign in to reply
      • Guillaume Samson mentioned in commit swh/infra/puppet/puppet-swh-site@858b72bf

        mentioned in commit swh/infra/puppet/puppet-swh-site@858b72bf

      • Guillaume Samson mentioned in merge request swh/infra/puppet/puppet-swh-site!732 (merged)

        mentioned in merge request swh/infra/puppet/puppet-swh-site!732 (merged)

      • Guillaume Samson mentioned in commit swh/infra/puppet/puppet-swh-site@e84609e8

        mentioned in commit swh/infra/puppet/puppet-swh-site@e84609e8

      • Guillaume Samson
        Guillaume Samson @guillaume ·
        Author Owner

        No borgmatic services alert since Tuesday (except bardo which seems ok since the sshd_config update on banco).
        Closing.

      • Guillaume Samson closed

        closed

      Please register or sign in to reply