deposit: Handle download and aggregation of uploaded tarballs

Instead of using the deposit API raw private endpoint to get aggregated version of tarballs uploaded with a deposit, let the loader handle this possibly costly operation by using the new deposit API upload-urls private endpoint returning URLs for downloading uploaded tarballs.

Fixes swh-deposit#4657 (closed).

Depends on swh-deposit!455 (merged).

added 1 commit

• 947d3c1f - deposit: Handle download and aggregation of uploaded tarballs

Compare with previous version

Jenkins job DLDBASE/gitlab-builds #387 succeeded in 2 min 15 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

Jenkins job DLDBASE/gitlab-builds #388 succeeded in 1 min 58 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

Jenkins job DLDBASE/gitlab-builds #389 succeeded in 1 min 58 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

added 1 commit

• d6b7a831 - deposit: Handle download and aggregation of uploaded tarballs

Compare with previous version

changed the description

Jenkins job DLDBASE/gitlab-builds #390 succeeded in 1 min 48 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

mentioned in merge request swh-deposit!455 (merged)

added 4 commits

• d6b7a831...1ea3e051 - 3 commits from branch swh/devel:master
• 6a4e9829 - deposit: Handle download and aggregation of uploaded tarballs

Compare with previous version

Jenkins job DLDBASE/gitlab-builds #395 succeeded in 1 min 46 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

Seems like you've added a spurious cov.xml file

added 1 commit

• 54fab701 - deposit: Handle download and aggregation of uploaded tarballs

Compare with previous version

^ Remove spurious cov.xml file.

Jenkins job DLDBASE/gitlab-builds #397 succeeded in 1 min 50 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

lgtm (I recognize some code from the deposit, probably improved too ;)

I assume that, this will now use a bit disks than before (in /tmp) for aggregating tarballs but that should be ok already. Our (celery) workers (pods) have some specific mount to deal with space there (loader-svn is quite a gluton ;). We'll see.

approved this merge request

resolved all threads

resolved all threads