ingest antelink s3 contents in swh

---

Migrated from T347 (view on Phabricator)

added General priority:Normal state:wip labels

s3 downloaded contents can be injected in swh. worker01 is currently doing it.

changed title from Inject s3 contents in swh to ingest antelink s3 contents in swh

assigned to @ardumont

I forgot to update this, sorry.

This is done but for 34 files whose size are > 4,000,000,000 bytes. List is https://forge.softwareheritage.org/$51

Currently independently running a final check of what data has been properly imported from s3 before removing it from the source.

I'm only looking at databases, not at the object store.

Check done (source file : a s3 listing Guillaume provided me in december). 85670 unique sha1s are missing.

missing_s3_20160405.gz

Thanks. Missing data reinjected.

Latest missing files (24348 items)

missing_s3_20160407.gz

Like i said in an email, those match the corrupted files at ~99,5%, the remaining ones are files whose size > 4Gib. We are good for s3 bucket deletion.

removed state:wip label

closed