dumb: Fix streaming of HTTP responses

When using the requests library to perform HTTP requests, if responses need to be streamed the stream parameter must be set to True to ensure content is downloaded by chunks.

Previously, a whole HTTP response was cached in memory which could lead to OOM errors when dealing with a repository with large pack files.

I noticed the issue as I did not understand why the new visits of https://git.texlive.info/texlive have their final status to created. I tested locally and got the loader process OOM killed as that repository has gigantic pack files the loader needs to download.

Jenkins job DLDG/gitlab-builds #165 succeeded .
See Console Output and Coverage Report for more details.

Good catch! While we're at it, should we also make it honor the pack_size_bytes setting?

Done in a new commit in that MR, I also added tests to cover the cases where pack size limit is reached.

added 2 commits

• 64ac0204 - dumb: Fix streaming of HTTP responses
• b7d16897 - dumb: Synchronize fetch_pack behavior with smart loader

Compare with previous version

Jenkins job DLDG/gitlab-builds #166 succeeded .
See Console Output and Coverage Report for more details.

approved this merge request

Great, thanks!

approved this merge request

merged