Skip to content

dumb: Fix streaming of HTTP responses

Antoine Lambert requested to merge anlambert/swh-loader-git:dump-fix-http-stream-reponse into master

When using the requests library to perform HTTP requests, if responses need to be streamed the stream parameter must be set to True to ensure content is downloaded by chunks.

Previously, a whole HTTP response was cached in memory which could lead to OOM errors when dealing with a repository with large pack files.

I noticed the issue as I did not understand why the new visits of https://git.texlive.info/texlive have their final status to created. I tested locally and got the loader process OOM killed as that repository has gigantic pack files the loader needs to download.

Merge request reports