Serve 2023-09-06 provenances indexes stored on mam on the web

To distribute the provenance index to our partners, we need to (temporarily) serve some fairly large files stored on mam (/data/ddouard/upload/provenance-heads/2023-09-06/) on the web.

$ ls -lh /data/ddouard/upload/provenance-heads/2023-09-06/
total 7,6T
-rw-r--r-- 1 vlorentz vlorentz 4,7T Dec 14 17:09 contents_in_frontier_directories.csv.zst
-rw-r--r-- 1 vlorentz vlorentz 489G Dec 14 17:32 contents_in_revisions_without_frontiers.csv.zst
-rw-r--r-- 1 vlorentz vlorentz 379G Dec 14 17:50 directory_frontier.csv.zst
-rw-r--r-- 1 vlorentz vlorentz  18G Dec 14 17:51 directory_frontier.deduplicated.csv.zst
-rw-r--r-- 1 vlorentz vlorentz 345G Dec 14 18:08 earliest_revrel_for_cntdir.csv.zst
-rw-r--r-- 1 vlorentz vlorentz 1,3T Jan 13 22:18 frontier_directories_in_revisions.csv.zst
-rw-r--r-- 1 vlorentz vlorentz 359G Jan  9 09:29 frontier_directories_in_revisions.csv.zst.bad
-rw-r--r-- 1 vlorentz vlorentz  86G Dec 14 18:12 revrel_by_author_date.csv.zst

As mam doesn't have a public IP address, we'll need to install a web server and manage a reverse proxy (being careful that no buffering happens, and that partial requests are properly served).

• deploy web server on mam
• pick reverse proxy
• deploy reverse proxy config

cc. @douardda @vlorentz

changed milestone to %Provenance in production [Roadmap - Tooling and infrastructure]

assigned to @olasd

changed the description

Deployed a nginx instance on mam (via puppet). Tuned it to have a bit more performance (by enabling sendfile)

Configured a reverse proxy on moma's varnish, disabling all caching (and buffering) by using a pipe in the VCL.

https://provenance-export.softwareheritage.org/2023-09-06/ is now available.

closed

mentioned in issue #5620 (closed)