Skip to content

Display a progressbar per object type during backup

Jérémy Bobbio (Lunar) requested to merge lunar/swh-alter:23-backup-per-object-type into main

The time it takes to backup objects vary widely between an object type from the next. Lumping all of them all in a single progressbar results in very inaccurate ETA. Instead, we display a progress bar for each object we backup.

This required moving progress bar management to RecoveryBundleCreator.backup_swhids(), leading to a refactor of Remover.create_recovery_bundle(), iter_swhids_grouped_by_type() and its other users. The result feels more symmetric with RecoveryBundle.restore() which was already managing its progress bar.

We take the opportunity to use different chunk sizes depending on the type of objects we are backing up. As Content objects have to retrieve data from objstorage, they tend to be longer to backup. We use smaller chunks there so we can update the progress more often.

We also handle a hidden issue were RawExtrinsingMetadata would be wrongly numbered in a RecoveryBundle if they were added in multiple calls to RecoveryBundleCreator.backup_swhids(). Trying to do so will now raise an exception.

Finally, objects being backed up are now storted by SWHID to ease reproducibility.

Example output:

$ swh alter remove --identifier initial --recovery-bundle /tmp/initial.zip https://gitlab.softwareheritage.org/swh/devel/swh-alter.git  --dry-run=stop-before-removal
Removing the following origins:
 - swh:1:ori:563a9a2cd47a25caf1a8d13b2a20f20276c8c808
Finding removable objects…
Inventorying all reachable objects…  [-----------------------#------------]  done (859 objects found / 0 left to look up)                                                
Determining which objects can be safely removed…  [####################################]  100%          
Finding RawExtrinsicMetadata objects…  [####################################]  100%
Removal plan:
- Origin: 1
- Snapshot: 1
- Release: 9
- Revision: 117
- Directory: 369
- Content: 360
- … and more objects that are not addresseable by a SWHID (OriginVisit, OriginVisitStatus, ExtID).
Creating recovery bundle…
Backing up Content objects…  [####################################]  100%          
Backing up Directory objects…  [####################################]  100%          
Backing up Revision objects…  [####################################]  100%
Backing up Release objects…  [####################################]  100%
Backing up Snapshot objects…  [####################################]  100%
Backing up Origin objects…  [####################################]  100%
Recovery bundle created.
Recovery bundle decryption key: AGE-SECRET-KEY-XXX
Stopping before removal.

Closes #23 (closed)

Merge request reports