Skip to content

loader: Fix rsync failures by retrying associated commands

Antoine Lambert requested to merge generated-differential-D7586-source into generated-differential-D7586-target

Fetching CVS repository data using rsync often fails with the following error (especially with archived repositories hosted on sourceforge):

rsync error: some files/attrs were not transferred (see previous errors) (code 23)

It seems the only way to mitigate that issue is to retry the rsync command until it succeeds.

(swh) anlambert@carnavalet:/tmp$ rsync -avz rsync://a.cvs.sourceforge.net/cvsroot/ojirc/ /tmp/cvsroot/ojirc/
receiving incremental file list
rsync: change_dir "/ojirc" (in cvsroot) failed: No such file or directory (2)

sent 8 bytes  received 99 bytes  71.33 bytes/sec
total size is 0  speedup is 0.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1819) [Receiver=3.2.3]
(swh) anlambert@carnavalet:/tmp$ echo $?
23
(swh) anlambert@carnavalet:/tmp$ rsync -avz rsync://a.cvs.sourceforge.net/cvsroot/ojirc/ /tmp/cvsroot/ojirc/
receiving incremental file list

sent 34 bytes  received 2,050 bytes  833.60 bytes/sec
total size is 487,994  speedup is 234.16
(swh) anlambert@carnavalet:/tmp$ echo $?
0

So add a rsync_retry decorator and apply it to a new method in the loader wrapping the call to subprocess.run executing the rsync command.

Also use rsync option to compress file data during the transfer.

Fixes SWH-LOADER-CVS-D and SWH-LOADER-CVS-J.

Migrated from D7586 (view on Phabricator)

Merge request reports