Multiple loaders errored with `No such file or directory: '/tmp/...` on 2022-05-02 and 2022-05-03

Sentry shows these errors happening across multiple loaders and seemingly all production workers:

• bzr: 27 events, eg. No such file: '/tmp/swh.loader.bzr.from_diskn2da5v8j-2504976/.bzr/repository/pack-names': [Errno 2] No such file or directory: '/tmp/swh.loader.bzr.from_diskn2da5v8j-2504976/.bzr/repository/pack-names'
• svn: 7 events, eg. [Errno 2] No such file or directory: '/tmp/swh.loader.svn.5xfy0xda-2367722/swh.loader.svn.4ho5suil-2367722'
• deb: 1 event, eg. [Errno 2] No such file or directory: '/tmp/tmpxqa2iwek'
• npm: 45 events, eg. [Errno 2] No such file or directory: '/tmp/tmponioyenb'

This happened mostly in the breezy code (for the breezy loader, ofc) and in tempfile.mkdtemp, but also occasionally in shutil.rmtree.

changed the description

One possible suspect could be the tmp directory cleanup done by this crontab:

0-22/2 * * * root find /tmp -depth -mindepth 3 -maxdepth 3 -type d -ctime +2 -exec rm -rf {} \+

For example for the last bzr errors, it match the runs of the command: exemple1: May 28, 2023 6:43:01 AM UTC

sentry: https://sentry.softwareheritage.org/organizations/swh/issues/106816/events/cbb42658873a4397b6228e471db67720/?project=22
cron: http://kibana0.internal.softwareheritage.org:5601/app/discover#/doc/3f8dbf80-18cc-11e9-b8ce-cf95f437ce37/systemlogs-7.15.2-2023.05.28?id=qiY9Y4gB-8-c3mqA8kcu

example2: May 2, 2023 4:04:13 PM UTC

sentry: https://sentry.softwareheritage.org/organizations/swh/issues/106809/events/8912d0a55ea64445b37cc5cbe11055aa/?project=14
cron: http://kibana0.internal.softwareheritage.org:5601/app/discover#/doc/3f8dbf80-18cc-11e9-b8ce-cf95f437ce37/systemlogs-7.15.2-2023.05.27?id=mlvVXIgB-8-c3mqAT7rK

example3: May 27, 2023 12:51:01 PM UTC

sentry: https://sentry.softwareheritage.org/organizations/swh/issues/106816/events/f67cb8970f864e39b4035ca5d100c329/?project=22
cron: http://kibana0.internal.softwareheritage.org:5601/app/discover#/doc/3f8dbf80-18cc-11e9-b8ce-cf95f437ce37/systemlogs-7.15.2-2023.05.27?id=mlvVXIgB-8-c3mqAT7rK

svn is not so closed to the run of the crontab but it could depend when the data is flush on disk

Isn't this cron tab the one used for "static" workers? IINM, it was a workaround for loaders with a high disk usage (which gets killed and leave noise in /tmp). That may no longer be needed for elastic workers.

If so, do we need it for "elastic" workers? If not, we may be able to drop it from the "elastic" one and see if the problem persists or not.

we should not need it for the elastic workers because the data is stored in a pv which is automagically deleted when the pod is destroyed.

Perhaps the problem could be present for long running workers not correctly cleaning their data between two repository loadings

I had something like that in mind. Thanks for confirming. So great, let's drop it!

(We'll see later for the edge case problems you mention, if that arise) What do you think?

If the cron deleted temporary directories that were being used by loaders, then doesn't it mean that some PM visits may be missing files, because they were deleted between the moment the loader extracted them and the moment the loader loaded them?

PM visits?

But yes, it means the loading is only partially done. What's missing is kinda dependent on how the loader is implemented... (and at what point it tried to read those temporary files back).

And, currently, for those origins in errors to be reloaded, we can rely on the scheduler to reschedule them (if they did not get deactivated because they reached the error threshold or some such). Another manual way is to lift the sentry api to have the listing of the errored origins and reschedule those (if they are still in the sentry history) [1]

[1] something i do once in a while e.g. in #4847 (closed)

PM visits?

package manager

And, currently, for those origins in errors to be reloaded, we can rely on the scheduler to reschedule them

Not only those in error. We need to re-run every visit that was active while the rf -r was running.

and invalidate their ExtIDs first