Skip to content

package/utils: Set accept-encoding to identity for download request

Antoine Lambert requested to merge anlambert/swh-loader-core:download-accept-encoding-identity into master

requests package set the value of the accept-encoding HTTP header to gzip, deflate by default and some servers (https://download.ocamlcore.org/ for instance) will then send a compressed version of the artifact to download with response header content-encoding usually set to gzip.

$ curl -H "Accept-Encoding: gzip, deflate" -I https://download.ocamlcore.org/promela/promela/0.4.2/promela-0.4.2.tar.gz
HTTP/2 200 
accept-ranges: bytes
age: 0
cache-control: public,max-age=0,must-revalidate
cache-status: "Netlify Edge"; fwd=miss
content-encoding: gzip
content-type: application/gzip
date: Thu, 04 Jan 2024 16:17:03 GMT
etag: "512c6b96f14abd65501fa9a48ad17d81-ssl-df"
server: Netlify
strict-transport-security: max-age=31536000
vary: Accept-Encoding
x-nf-request-id: 01HKAM73HWC1H3HPA3F29STFG5

Nevertheless, this conflicts with the code checking if the response bytes should be uncompressed as it should not when content-encoding header is equal to gzip and content-type is equal to application/*gzip.

As artifacts to download are usually tarballs already compressed with gzip, set the accept-encoding request header to identity in order to force the server to send raw artifact bytes without the content-encoding header set.

$ curl -H "Accept-Encoding: identity" -I https://download.ocamlcore.org/promela/promela/0.4.2/promela-0.4.2.tar.gz
HTTP/2 200 
accept-ranges: bytes
age: 0
cache-control: public,max-age=0,must-revalidate
cache-status: "Netlify Edge"; fwd=miss
content-type: application/gzip
date: Thu, 04 Jan 2024 16:17:30 GMT
etag: "512c6b96f14abd65501fa9a48ad17d81-ssl"
server: Netlify
strict-transport-security: max-age=31536000
x-nf-request-id: 01HKAM7XF0GPR840BGH7YW1N41
content-length: 50287

This fixes numerous hash mismatch errors related to opam loading reported in SWH-LOADER-CORE-1SK.

Merge request reports