deployment/provenance: Adapt template to manage grpc or rpc service

Review changes
Download
Patches
Plain diff

Closed deployment/provenance: Adapt template to manage grpc or rpc service

mr/adapt-provenance-deployment into production

Overview 14
Commits 0
Pipelines 0
Changes 0

Closed Antoine R. Dumont requested to merge mr/adapt-provenance-deployment into production 3 weeks ago

Overview 3
Commits 0
Pipelines 0
Changes 14

This makes the provenance template evolve to be able to declare a new grpc provenance server. It's still possible to deploy the rpc provenance server.

For now, it's focused on not touching the provenance rpc servers already deployed in staging and production (some environment variables got moved so there is a slight configuration checksum change but no behavioral change).

And this adds 2 grpc server instances with the smallest dataset possible:

one in the local-cluster [1] (connection tested through toolbox [2])
another in the next-version [3]

In another mr, we'll deploy the provenance server in staging.

TODO:

local-cluster
- Check response of grpc service with grpcurl (determine queries first) [2]
- Check api provenance response with webapp connected to grpc [3]
- Adapt webapp configuration to use the new provenance [4]
next-version: Deploy provenance-grpc & webapp accordingly ^
provenance: Add downloading witness file (to avoid concurrent downloads)
graph: same ^

[1] local-cluster: Provenance Grpc running ok

2025-03-20T15:04:06.823292798Z Starting the swh-provenance GRPC server
2025-03-20T15:04:06.828395082Z 2025-03-20T15:04:06.828326Z  INFO swh_provenance_grpc_serve: Loading graph properties and database
2025-03-20T15:04:06.843855546Z 2025-03-20T15:04:06.843767Z  INFO swh_provenance::utils: Graph loaded
2025-03-20T15:04:06.904579636Z 2025-03-20T15:04:06.904494Z  INFO swh_provenance::utils: Database loaded
2025-03-20T15:04:06.904596958Z 2025-03-20T15:04:06.904532Z  INFO swh_provenance_grpc_serve: Starting server
2025-03-21T09:15:11.549147527Z graph-grpc-popular-20240823 2025-03-21T09:15:11.549070Z  INFO swh_graph_grpc_serve: Loading graph
2025-03-21T09:15:11.556021379Z graph-grpc-popular-20240823 2025-03-21T09:15:11.555943Z  INFO swh_graph_grpc_serve: Starting server
2025-03-21T09:15:31.921485587Z graph-grpc-popular-20240823 2025-03-21T09:15:31.921332Z  INFO request{id=0}:stats: swh_graph_grpc_server: StatsRequest
2025-03-21T09:15:31.930900836Z graph-grpc-popular-20240823 2025-03-21T09:15:31.930779Z ERROR request{id=0}:stats: swh_graph_grpc_server: Missing compratio in /
srv/graph/2024-08-23_popular-4-shell/compressed/graph.properties
2025-03-21T09:15:31.930938605Z graph-grpc-popular-20240823 2025-03-21T09:15:31.930796Z ERROR request{id=0}:stats: swh_graph_grpc_server: Missing bitspernode in
 /srv/graph/2024-08-23_popular-4-shell/compressed/graph.properties
2025-03-21T09:15:31.930951990Z graph-grpc-popular-20240823 2025-03-21T09:15:31.930802Z ERROR request{id=0}:stats: swh_graph_grpc_server: Missing bitsperlink in
 /srv/graph/2024-08-23_popular-4-shell/compressed/graph.properties
2025-03-21T09:15:31.930963150Z graph-grpc-popular-20240823 2025-03-21T09:15:31.930808Z ERROR request{id=0}:stats: swh_graph_grpc_server: Missing avglocality in
 /srv/graph/2024-08-23_popular-4-shell/compressed/graph.stats
2025-03-21T09:15:31.930990010Z graph-grpc-popular-20240823 2025-03-21T09:15:31.930897Z  INFO request{id=0}: swh_graph_grpc_server::metrics: 200 OK - /swh.graph
.TraversalService/Stats - response: 9.64976ms - streaming: 44.992µs
2025-03-21T09:17:55.699423515Z graph-grpc-popular-20240823 2025-03-21T09:17:55.699342Z  INFO request{id=1}:traverse: swh_graph_grpc_server: TraversalRequest {
src: ["swh:1:cnt:dcb2d732994e615aab0777bfe625bd1f07e486ac"], direction: Backward, edges: None, max_edges: None, min_depth: None, max_depth: None, return_nodes:
 None, mask: None, max_matching_nodes: None }
2025-03-21T09:17:55.704288980Z graph-grpc-popular-20240823 2025-03-21T09:17:55.704221Z  INFO request{id=1}:traverse: swh_graph_grpc_server: error=status: NotFo
und, message: "Unknown SWHID: swh:1:cnt:dcb2d732994e615aab0777bfe625bd1f07e486ac", details: [], metadata: MetadataMap { headers: {} }
2025-03-21T09:17:55.704312002Z graph-grpc-popular-20240823 2025-03-21T09:17:55.704277Z  INFO request{id=1}: swh_graph_grpc_server::metrics: 200 OK - /swh.graph
.TraversalService/Traverse - response: 4.972231ms - streaming: 431ns
2025-03-21T09:27:40.574344623Z graph-grpc-popular-20240823 2025-03-21T09:27:40.574270Z  INFO request{id=2}:traverse: swh_graph_grpc_server: TraversalRequest {
src: ["swh:1:cnt:dcb2d732994e615aab0777bfe625bd1f07e486ac"], direction: Backward, edges: None, max_edges: None, min_depth: None, max_depth: None, return_nodes:
 None, mask: None, max_matching_nodes: None }
2025-03-21T09:27:40.574390096Z graph-grpc-popular-20240823 2025-03-21T09:27:40.574319Z  INFO request{id=2}:traverse: swh_graph_grpc_server: error=status: NotFo
und, message: "Unknown SWHID: swh:1:cnt:dcb2d732994e615aab0777bfe625bd1f07e486ac", details: [], metadata: MetadataMap { headers: {} }
2025-03-21T09:27:40.574395456Z graph-grpc-popular-20240823 2025-03-21T09:27:40.574339Z  INFO request{id=2}: swh_graph_grpc_server::metrics: 200 OK - /swh.graph
.TraversalService/Traverse - response: 100.985µs - streaming: 230ns

[2] local-cluster: Connection to grpc provenance ok

swh@swh-toolbox-5548445f74-x8vmb:~$ server=provenance-grpc-popular-ingress:80
grpcurl --plaintext $server list swh.provenance.ProvenanceService
swh.provenance.ProvenanceService.WhereAreOne
swh.provenance.ProvenanceService.WhereIsOne
swh@swh-toolbox-5548445f74-x8vmb:~$ unknown_swhid="swh:1:cnt:27766b99cdcab4e9b68501c3b50f1712e016c945"
grpcurl -d "{\"swhid\": \"${unknown_swhid}\"}"   --plaintext $server   swh.provenance.ProvenanceService.WhereIsOne
ERROR:
  Code: Internal
  Message: status: NotFound, message: "Unknown SWHID: swh:1:cnt:27766b99cdcab4e9b68501c3b50f1712e016c945", details: [], metadata: MetadataMap { headers: {} }
swh@swh-toolbox-5548445f74-x8vmb:~$ swhid="swh:1:cnt:07d9e8c75f4f7e7dba04b5b4e8589a158c8a6892"
grpcurl -d "{\"swhid\": \"${swhid}\"}"   --plaintext $server   swh.provenance.ProvenanceService.WhereIsOne
{
  "swhid": "swh:1:cnt:07d9e8c75f4f7e7dba04b5b4e8589a158c8a6892",
  "anchor": "swh:1:rev:aef9e137acd823aa0097f195b613f96aae619923"
}

[4] local-cluster: Connection through webapp to the new grpc provenance

$ curl -s http://web-local-archive-ingress/api/1/provenance/whereis/swh:1:cnt:07d9e8c75f4f7e7dba04b5b4e8589a158c8a6892/ | jq .

"swh:1:cnt:07d9e8c75f4f7e7dba04b5b4e8589a158c8a6892;anchor=swh:1:rev:aef9e137acd823aa0097f195b613f96aae619923"

# Missing contents
$ curl -s http://web-local-archive-ingress/api/1/provenance/whereis/swh:1:cnt:dcb2d732994e615aab0777bfe625bd1f07e486ac/ | jq .

{
  "exception": "_InactiveRpcError",
  "reason": "<_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.INTERNAL\n\tdetails = \"status: NotFound, message: \"Unknown SWHID: swh:1:cnt:dcb2d732994e615aab0777bfe625bd1f07e486ac\", details: [], metadata: MetadataMap { headers: {} }\"\n\tdebug_error_string = \"UNKNOWN:Error received from peer  {grpc_message:\"status: NotFound, message: \\\"Unknown SWHID: swh:1:cnt:dcb2d732994e615aab0777bfe625bd1f07e486ac\\\", details: [], metadata: MetadataMap { headers: {} }\", grpc_status:13, created_time:\"2025-03-21T15:42:58.066805842+00:00\"}\"\n>"
}
$ curl -s http://web-local-archive-ingress/api/1/provenance/whereis/swh:1:cnt:dcb2d732994e615aab0777bfe625bd1f07e486ac/ | jq .
{
  "exception": "_InactiveRpcError",
  "reason": "<_InactiveRpcError of RPC that terminated with:\n\tstatus = StatusCode.INTERNAL\n\tdetails = \"status: NotFound, message: \"Unknown SWHID: swh:1:cnt:dcb2d732994e615aab0777bfe625bd1f07e486ac\", details: [], metadata: MetadataMap { headers: {} }\"\n\tdebug_error_string = \"UNKNOWN:Error received from peer  {grpc_message:\"status: NotFound, message: \\\"Unknown SWHID: swh:1:cnt:dcb2d732994e615aab0777bfe625bd1f07e486ac\\\", details: [], metadata: MetadataMap { headers: {} }\", grpc_status:13, created_time:\"2025-03-21T15:43:39.432082899+00:00\"}\"\n>"
}

[3] Deploy next-version provenance grpc instance

[swh] Comparing changes between branches production and mr/adapt-provenance-deployment (per environment)...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment staging, namespace swh...
[swh] Generate config in production branch for environment staging, namespace swh-cassandra...
[swh] Generate config in production branch for environment staging, namespace next-version...
[swh] Generate config in mr/adapt-provenance-deployment branch for environment staging...
[swh] Generate config in mr/adapt-provenance-deployment branch for environment staging...
[swh] Generate config in mr/adapt-provenance-deployment branch for environment staging...
Your branch is up to date with 'origin/production'.
[swh] Generate config in production branch for environment production, namespace swh...
[swh] Generate config in production branch for environment production, namespace swh-cassandra...
[swh] Generate config in production branch for environment production, namespace next-version...
[swh] Generate config in mr/adapt-provenance-deployment branch for environment production...
[swh] Generate config in mr/adapt-provenance-deployment branch for environment production...
[swh] Generate config in mr/adapt-provenance-deployment branch for environment production...


------------- diff for environment staging namespace swh -------------

--- /tmp/swh-chart.swh.lQqjoZXu/staging-swh.before	2025-03-21 17:04:33.345117955 +0100
+++ /tmp/swh-chart.swh.lQqjoZXu/staging-swh.after	2025-03-21 17:04:33.997092876 +0100
@@ -1914,21 +1914,21 @@
     fi
 
     graph_transposed_name=${GRAPH_NAME}-transposed.graph
     if [ -L ${DATASET_LOCATION}/${graph_transposed_name} ] || ! [ -f ${DATASET_LOCATION}/${graph_transposed_name} ]; then
       cp -v --remove-destination ${DATASET_SOURCE}/${graph_transposed_name} ${DATASET_LOCATION}/;
     fi
 
     # Finally, we make explicit the graph is ready
     touch ${WITNESS_FILE}
 
-  graph-wait-for-dataset.sh: |
+  wait-for-dataset.sh: |
     #!/usr/bin/env bash
     # Uses env variables WITNESS_FILE
     [ -z "${WITNESS_FILE}" ] && \
       echo "<WITNESS_FILE> env variable must be set" && exit 1
 
     while [ ! -f ${WITNESS_FILE} ]; do
         echo "${WITNESS_FILE} not present, wait for it to start the graph..."
         sleep $PERIOD
     done
 
@@ -2010,20 +2010,125 @@
         echo "${WITNESS_SOURCE_FILE} missing, waiting graph dataset installation..."
         sleep $PERIOD
     done
 
     # For old datasets missing a .ef or in the wrong format, this fails with
     # `Cannot map Elias-Fano pointer list .../graph.ef`. The solution is to
     # reindex the dataset
     swh graph reindex --ef ${DATASET_LOCATION}/${GRAPH_NAME} && \
       touch $WITNESS_REINDEX_FILE
 
+  provenance-fetch-datasets.sh: |
+    #!/usr/bin/env bash
+    [ -z "${WITNESS_FETCH_FILE}" ] && \
+      echo "<WITNESS_FETCH_FILE> env variable must be set" && exit 1
+    [ -z "${DATASET_VERSION}" ] && \
+      echo "<DATASET_VERSION> env variable must be set" && exit 1
+    [ -z "${PROVENANCE_PATH}" ] && \
+      echo "<PROVENANCE_PATH> env variable must be set" && exit 1
+    [ -z "${GRAPH_PATH}" ] && \
+      echo "<GRAPH_PATH> env variable must be set" && exit 1
+
+    [ -f ${WITNESS_FETCH_FILE} ] && \
+        echo "Datasets graph & provenance <${DATASET_VERSION}> already present. Skip." && \
+        exit 0
+
+    set -e
+
+    # Create destination paths
+    mkdir -p ${PROVENANCE_PATH} ${GRAPH_PATH}
+
+    echo "Fetching datasets..."
+
+    if [ ${PROVENANCE_DATASET_FULL} = true ]; then
+        # Retrieve all the provenance dataset
+        REFS=all
+    else
+        # This excludes revisions not targetted by a snapshot
+        # Ok to use for test purposes
+        REFS=heads
+    fi
+
+    URL_PROVENANCE="s3://softwareheritage/derived_datasets/${DATASET_VERSION}/provenance/${REFS}/"
+
+    CMD_GET="aws s3 cp --no-progress --no-sign-request"
+
+    echo "1. Fetching provenance dataset (parquet files)..."
+    ${CMD_GET} --recursive "${URL_PROVENANCE}" "${PROVENANCE_PATH}"
+    echo "1. Provenance datasets installed!"
+
+    echo "2. Fetching extra graph files..."
+    URL_GRAPH="s3://softwareheritage/graph/${DATASET_VERSION}/compressed"
+
+    mkdir -p "${GRAPH_PATH}"
+    for filename in graph.pthash graph.pthash.order graph.nodes.count.txt \
+                    graph.property.message.bin.zst \
+                    graph.property.message.offset.bin.zst \
+                    graph.property.tag_name.bin.zst \
+                    graph.property.tag_name.offset.bin.zst \
+                    graph.node2swhid.bin.zst graph.node2type.bin.zst; do
+        ${CMD_GET} "${URL_GRAPH}/${filename}" "${GRAPH_PATH}"
+    done
+    echo "2. Extra graph files installed!"
+
+    echo "3. Uncompressing graph files..."
+    set -x
+    # Uncompress the compressed graph *.zst files
+    for filepath in $(ls ${GRAPH_PATH}/*.zst); do
+        # Uncompress and delete the .zst file
+        [ -f "${filepath}" ] && unzstd --force --rm "${filepath}"
+    done
+    set +x
+    echo "3. Graph files uncompressed!"
+
+    # Make explicit the provenance datasets are fetched
+    touch ${WITNESS_FETCH_FILE}
+
+    echo "Provenance datasets installed!"
+
+  provenance-index-dataset.sh: |
+    #!/usr/bin/env bash
+    [ -z "${WITNESS_DATASETS_FILE}" ] && \
+      echo "<WITNESS_DATASETS_FILE> env variable must be set" && exit 1
+    [ -z "${WITNESS_INDEX_FILE}" ] && \
+      echo "<WITNESS_INDEX_FILE> env variable must be set" && exit 1
+    [ -z "${PERIOD}" ] && \
+      echo "<PERIOD> env variable must be set" && exit 1
+    [ -z "${PROVENANCE_PATH}" ] && \
+      echo "<PROVENANCE_PATH> env variable must be set" && exit 1
+
+    [ -f ${WITNESS_INDEX_FILE} ] && echo "Provenance already indexed, do nothing." && \
+      exit 0
+
+    set -eu
+
+    # Let's wait for the dataset installation
+    while [ ! -f "${WITNESS_DATASETS_FILE}" ]; do
+        echo "${WITNESS_DATASETS_FILE} missing, waiting provenance dataset installation..."
+        sleep $PERIOD
+    done
+
+    echo "Datasets file installed, build provenance dataset indexes..."
+
+    echo "provenance path: $PROVENANCE_PATH"
+    set -x
+
+    # To make the query faster, the provenance needs to build index out of the
+    # current dataset files. We store the output indexes in the same path as
+    # the dataset.
+    swh-provenance-index \
+      --database file://${PROVENANCE_PATH} && \
+      touch "${WITNESS_INDEX_FILE}" && \
+      echo "Provenance indexes built!" || \
+
+    echo "Provenance indexes failed!"
+
   initialize-search-backend.sh: |
     #!/usr/bin/env bash
 
     set -eux
 
     # Uses internally the environment variable SWH_CONFIG_FILENAME
     swh search initialize
   register-task-types.sh: |
     #!/usr/bin/env bash
 
@@ -3974,21 +4079,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-content
       annotations:
         checksum/config: 6dab6ee0c1a4a8b88f25c2ae5ae03e8f9123247ab730bc299f03ad5e552fdd2e
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -4114,21 +4219,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-directory
       annotations:
         checksum/config: 6e201e0b3d31c59906f2f6bb40eed69c44f72f5fc46aa224a9115980194888b5
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -4254,21 +4359,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-extid
       annotations:
         checksum/config: fb5ef271dff488758f430d36ab153d3d71c2bcd5466401f8b4b8f56eecdbab09
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -4394,21 +4499,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-metadata
       annotations:
         checksum/config: 59c200997e2837af6c79d357abf2b9d3887ddd9fa1da218e6867372137a6c12c
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -4534,21 +4639,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-origin
       annotations:
         checksum/config: 8a444fbf8b876ca0413f40eced16d8f67f37e21f9d9af30dc0e9d230f99353f0
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -4674,21 +4779,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-origin-visit
       annotations:
         checksum/config: 0232863ea6af9728905e510a2c7cda793b30e51282bb61e15951295b1b4ae5be
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -4814,21 +4919,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-origin-visit-status
       annotations:
         checksum/config: 50ccfa4fef14a26d4e62b4a4a7e9548e96ecd33bff9ed38f4caee11ad4872f50
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -4954,21 +5059,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-raw-extrinsic-metadata
       annotations:
         checksum/config: f7c02005d918fc96845fbe96ca5b80e02cbbe8db33c52304977fdf5775a7b39b
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -5094,21 +5199,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-release
       annotations:
         checksum/config: 2080d9cc684b645df5f3408a379de43116e1cd92dda2b81e90d60dbeee8d4b7a
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -5234,21 +5339,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-revision
       annotations:
         checksum/config: 576ef144c0f7a064b95e0584e23ed8e1bb49d8d54ca2314992484b79b549b116
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -5374,21 +5479,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-skipped-content
       annotations:
         checksum/config: 7bb2dae23e587e50bb4d18ddbd484ea63585e1d728b9dfdcdd27fc62435cee7f
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -5514,21 +5619,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-snapshot
       annotations:
         checksum/config: cddbe0bf9d91c24560eb3af9f843a00986a2d524dba0fdc5c111a6502caeed38
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -5655,21 +5760,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-postgresql-read-only
       annotations:
         checksum/config: 557a29778d601193c743a35a7075315560447fc8740ba7922d5d52ce3f0c621e
         checksum/config-logging: cca6d0318bd776cd9bee0901e67e4db9fa401456f6f03f569c15468c5e62bea7
-        checksum/backend-utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/backend-utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
         checksum/config-utils: d75ca13b805bce6a8ab59c8e24c938f2283108f6a79134f6e71db86308651dc6
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/storage
                 operator: In
                 values:
@@ -5820,21 +5925,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-postgresql-read-write
       annotations:
         checksum/config: 7673b003223acd4a3a0d130516efd4f89740163d29c4800d53c9afe16b8d21a2
         checksum/config-logging: c9f05b677492d0f7443fc8193c82673ce3c550f351b82dcf616a247f7477fae0
-        checksum/backend-utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/backend-utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
         checksum/config-utils: d75ca13b805bce6a8ab59c8e24c938f2283108f6a79134f6e71db86308651dc6
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/storage
                 operator: In
                 values:


------------- diff for environment staging namespace swh-cassandra -------------

--- /tmp/swh-chart.swh.lQqjoZXu/staging-swh-cassandra.before	2025-03-21 17:04:33.705104109 +0100
+++ /tmp/swh-chart.swh.lQqjoZXu/staging-swh-cassandra.after	2025-03-21 17:04:34.321080414 +0100
@@ -8875,21 +8875,21 @@
     fi
 
     graph_transposed_name=${GRAPH_NAME}-transposed.graph
     if [ -L ${DATASET_LOCATION}/${graph_transposed_name} ] || ! [ -f ${DATASET_LOCATION}/${graph_transposed_name} ]; then
       cp -v --remove-destination ${DATASET_SOURCE}/${graph_transposed_name} ${DATASET_LOCATION}/;
     fi
 
     # Finally, we make explicit the graph is ready
     touch ${WITNESS_FILE}
 
-  graph-wait-for-dataset.sh: |
+  wait-for-dataset.sh: |
     #!/usr/bin/env bash
     # Uses env variables WITNESS_FILE
     [ -z "${WITNESS_FILE}" ] && \
       echo "<WITNESS_FILE> env variable must be set" && exit 1
 
     while [ ! -f ${WITNESS_FILE} ]; do
         echo "${WITNESS_FILE} not present, wait for it to start the graph..."
         sleep $PERIOD
     done
 
@@ -8971,20 +8971,125 @@
         echo "${WITNESS_SOURCE_FILE} missing, waiting graph dataset installation..."
         sleep $PERIOD
     done
 
     # For old datasets missing a .ef or in the wrong format, this fails with
     # `Cannot map Elias-Fano pointer list .../graph.ef`. The solution is to
     # reindex the dataset
     swh graph reindex --ef ${DATASET_LOCATION}/${GRAPH_NAME} && \
       touch $WITNESS_REINDEX_FILE
 
+  provenance-fetch-datasets.sh: |
+    #!/usr/bin/env bash
+    [ -z "${WITNESS_FETCH_FILE}" ] && \
+      echo "<WITNESS_FETCH_FILE> env variable must be set" && exit 1
+    [ -z "${DATASET_VERSION}" ] && \
+      echo "<DATASET_VERSION> env variable must be set" && exit 1
+    [ -z "${PROVENANCE_PATH}" ] && \
+      echo "<PROVENANCE_PATH> env variable must be set" && exit 1
+    [ -z "${GRAPH_PATH}" ] && \
+      echo "<GRAPH_PATH> env variable must be set" && exit 1
+
+    [ -f ${WITNESS_FETCH_FILE} ] && \
+        echo "Datasets graph & provenance <${DATASET_VERSION}> already present. Skip." && \
+        exit 0
+
+    set -e
+
+    # Create destination paths
+    mkdir -p ${PROVENANCE_PATH} ${GRAPH_PATH}
+
+    echo "Fetching datasets..."
+
+    if [ ${PROVENANCE_DATASET_FULL} = true ]; then
+        # Retrieve all the provenance dataset
+        REFS=all
+    else
+        # This excludes revisions not targetted by a snapshot
+        # Ok to use for test purposes
+        REFS=heads
+    fi
+
+    URL_PROVENANCE="s3://softwareheritage/derived_datasets/${DATASET_VERSION}/provenance/${REFS}/"
+
+    CMD_GET="aws s3 cp --no-progress --no-sign-request"
+
+    echo "1. Fetching provenance dataset (parquet files)..."
+    ${CMD_GET} --recursive "${URL_PROVENANCE}" "${PROVENANCE_PATH}"
+    echo "1. Provenance datasets installed!"
+
+    echo "2. Fetching extra graph files..."
+    URL_GRAPH="s3://softwareheritage/graph/${DATASET_VERSION}/compressed"
+
+    mkdir -p "${GRAPH_PATH}"
+    for filename in graph.pthash graph.pthash.order graph.nodes.count.txt \
+                    graph.property.message.bin.zst \
+                    graph.property.message.offset.bin.zst \
+                    graph.property.tag_name.bin.zst \
+                    graph.property.tag_name.offset.bin.zst \
+                    graph.node2swhid.bin.zst graph.node2type.bin.zst; do
+        ${CMD_GET} "${URL_GRAPH}/${filename}" "${GRAPH_PATH}"
+    done
+    echo "2. Extra graph files installed!"
+
+    echo "3. Uncompressing graph files..."
+    set -x
+    # Uncompress the compressed graph *.zst files
+    for filepath in $(ls ${GRAPH_PATH}/*.zst); do
+        # Uncompress and delete the .zst file
+        [ -f "${filepath}" ] && unzstd --force --rm "${filepath}"
+    done
+    set +x
+    echo "3. Graph files uncompressed!"
+
+    # Make explicit the provenance datasets are fetched
+    touch ${WITNESS_FETCH_FILE}
+
+    echo "Provenance datasets installed!"
+
+  provenance-index-dataset.sh: |
+    #!/usr/bin/env bash
+    [ -z "${WITNESS_DATASETS_FILE}" ] && \
+      echo "<WITNESS_DATASETS_FILE> env variable must be set" && exit 1
+    [ -z "${WITNESS_INDEX_FILE}" ] && \
+      echo "<WITNESS_INDEX_FILE> env variable must be set" && exit 1
+    [ -z "${PERIOD}" ] && \
+      echo "<PERIOD> env variable must be set" && exit 1
+    [ -z "${PROVENANCE_PATH}" ] && \
+      echo "<PROVENANCE_PATH> env variable must be set" && exit 1
+
+    [ -f ${WITNESS_INDEX_FILE} ] && echo "Provenance already indexed, do nothing." && \
+      exit 0
+
+    set -eu
+
+    # Let's wait for the dataset installation
+    while [ ! -f "${WITNESS_DATASETS_FILE}" ]; do
+        echo "${WITNESS_DATASETS_FILE} missing, waiting provenance dataset installation..."
+        sleep $PERIOD
+    done
+
+    echo "Datasets file installed, build provenance dataset indexes..."
+
+    echo "provenance path: $PROVENANCE_PATH"
+    set -x
+
+    # To make the query faster, the provenance needs to build index out of the
+    # current dataset files. We store the output indexes in the same path as
+    # the dataset.
+    swh-provenance-index \
+      --database file://${PROVENANCE_PATH} && \
+      touch "${WITNESS_INDEX_FILE}" && \
+      echo "Provenance indexes built!" || \
+
+    echo "Provenance indexes failed!"
+
   initialize-search-backend.sh: |
     #!/usr/bin/env bash
 
     set -eux
 
     # Uses internally the environment variable SWH_CONFIG_FILENAME
     swh search initialize
   register-task-types.sh: |
     #!/usr/bin/env bash
 
@@ -11668,21 +11773,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: graph-grpc-python3k
       annotations:
         checksum/config: b73b013412ed4679009823fcd1967f46c6c74ce0cac466277d96a7303b1ee5e9
         checksum/config-utils: 13a26f6add17e96ce01550153c77dcd48de60241a3f4db3c93d5467234be2a7f
-        checksum/backend-utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/backend-utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       nodeSelector:
         kubernetes.io/hostname: rancher-node-staging-rke2-metal01
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/graph
                 operator: In
@@ -11737,21 +11842,21 @@
         
           - name: graph-python3k-persistent
             mountPath: /srv/dataset
             readOnly: false
         
         
         - name: wait-for-dataset
           image: container-registry.softwareheritage.org/swh/infra/swh-apps/utils:20250211.1
           imagePullPolicy: IfNotPresent
           command:
-          - /entrypoints/graph-wait-for-dataset.sh
+          - /entrypoints/wait-for-dataset.sh
           env:
             - name: WITNESS_FILE
               value: /srv/graph/2021-03-23-popular-3k-python/compressed/.graph-is-initialized
             - name: DATASET_LOCATION
               value: /srv/graph/2021-03-23-popular-3k-python/compressed
             - name: PERIOD
               value: "3"
           volumeMounts:
           - name: backend-utils
             mountPath: /entrypoints
@@ -11882,21 +11987,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: graph-rpc-python3k
       annotations:
         checksum/config: cd2257ef14a7e6adb8b613ab147d995cfc4b4f250daba77c5b5cdbd63dcb1a35
         checksum/config-utils: 13a26f6add17e96ce01550153c77dcd48de60241a3f4db3c93d5467234be2a7f
-        checksum/backend-utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/backend-utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/graph
                 operator: In
                 values:
                 - "true"
@@ -12129,21 +12234,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: indexer-storage-rpc
       annotations:
         checksum/config: aba89c9cffc506b56207b7cc0377f0183f9e75493e42ee5a8e199dbc7d573ade
         checksum/config-logging: 7d9616b680a77c6ec7ba4c1a1c0f3fbf6343c4fc132847f8f4e313d965014749
-        checksum/backend-utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/backend-utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/rpc
                 operator: In
                 values:
                 - "true"
@@ -19984,20 +20089,21 @@
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: provenance-graph-granet
       annotations:
         checksum/config: e466b9f13c5124eedcaa557de44e64caf7c7ff4aa9ab5dab35b7ceede6a09568
         checksum/config-logging: ddcd27d991938c46f4fc0ad7ee028cb3005f186b3db022596c9ae94363881e4f
         checksum/config-utils: 13a26f6add17e96ce01550153c77dcd48de60241a3f4db3c93d5467234be2a7f
+        checksum/backend-utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/rpc
                 operator: In
                 values:
                 - "true"
@@ -20017,21 +20123,21 @@
             mountPath: /etc/swh/configuration-template
           - name: config-utils
             mountPath: /entrypoints
             readOnly: true
       containers:
         - name: provenance-graph-granet
           resources:
             requests:
               memory: 512Mi
               cpu: 500m
-          image: container-registry.softwareheritage.org/swh/infra/swh-apps/provenance:20250319.1
+          image: container-registry.softwareheritage.org/swh/infra/swh-apps/provenance:20250321.1
           imagePullPolicy: IfNotPresent
           ports:
             - containerPort: 5014
               name: rpc
           readinessProbe:
             httpGet:
               path: /
               port: rpc
             initialDelaySeconds: 15
             failureThreshold: 30
@@ -20040,76 +20146,88 @@
             tcpSocket:
               port: rpc
             initialDelaySeconds: 10
             periodSeconds: 5
           command:
           - /bin/bash
           args:
           - -c
           - /opt/swh/entrypoint.sh
           env:
+            - name: PROVENANCE_TYPE
+              value: rpc
+            - name: PORT
+              value: "5014"
             - name: WORKERS
               value: "4"
             - name: THREADS
               value: "1"
             - name: TIMEOUT
               value: "60"
+            - name: SWH_CONFIG_FILENAME
+              value: /etc/swh/config.yml
+            - name: SWH_LOG_CONFIG_JSON
+              value: /etc/swh/logging/logging-gunicorn.json
+            - name: STATSD_SERVICE_TYPE
+              value: provenance-graph-granet
             - name: STATSD_HOST
               value: prometheus-statsd-exporter
             - name: STATSD_PORT
               value: "9125"
             - name: STATSD_TAGS
               value: deployment:provenance-graph-granet
-            - name: STATSD_SERVICE_TYPE
-              value: provenance-graph-granet
             - name: SWH_LOG_LEVEL
-              value: "INFO"
-            - name: SWH_LOG_CONFIG_JSON
-              value: /etc/swh/logging/logging-gunicorn.json
+              value: INFO
             - name: SWH_SENTRY_ENVIRONMENT
               value: staging
             - name: SWH_MAIN_PACKAGE
               value: swh.provenance
             - name: SWH_SENTRY_DSN
               valueFrom:
                 secretKeyRef:
                   name: common-secrets
                   key: provenance-sentry-dsn
                   # 'name' secret should exist & include key
                   # if the setting doesn't exist, sentry pushes will be disabled
                   optional: true
             - name: SWH_SENTRY_DISABLE_LOGGING_EVENTS
               value: "true"
           volumeMounts:
           - name: configuration
             mountPath: /etc/swh
           - name: configuration-logging
             mountPath: /etc/swh/logging
+          
       volumes:
       - name: configuration
         emptyDir: {}
       - name: configuration-template
         configMap:
           name: provenance-graph-granet-configuration-template
           items:
           - key: "config.yml.template"
             path: "config.yml.template"
       - name: configuration-logging
         configMap:
           name: provenance-graph-granet-configuration-logging
           items:
           - key: "logging-gunicorn.json"
             path: "logging-gunicorn.json"
+      
       - name: config-utils
         configMap:
           name: config-utils
           defaultMode: 0555
+      - name: backend-utils
+        configMap:
+          name: backend-utils
+          defaultMode: 0555
 ---
 # Source: swh/templates/scheduler/extra-services-deployment.yaml
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   namespace: swh-cassandra
   name: scheduler-listener
   labels:
     app: scheduler-listener
 spec:
@@ -21523,21 +21641,21 @@
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: search-rpc
       annotations:
         checksum/config: cb3458d5372d0f78a475da3ac1b4f474cebe95188ad4f0931eba0a7c9657122e
         checksum/config-logging: 7bffbc6ce2cb11d88208ef0c5f1d8e6822659c361717afb51dcf0f4da02fe1f7
         checksum/config-utils: 13a26f6add17e96ce01550153c77dcd48de60241a3f4db3c93d5467234be2a7f
-        checksum/backend-utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/backend-utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/rpc
                 operator: In
                 values:
                 - "true"
@@ -21679,21 +21797,21 @@
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: search-static-rpc
       annotations:
         checksum/config: 7a5b75d976f3579a1c831aa35c012f5f15ff2e2b488fe1ae7f1da9ee4bb8ca3e
         checksum/config-logging: bc2025f41b3eb8aa28b66033b96fdc1cb963f5c01fe33b2417c2378f715dbc32
         checksum/config-utils: 13a26f6add17e96ce01550153c77dcd48de60241a3f4db3c93d5467234be2a7f
-        checksum/backend-utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/backend-utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/rpc
                 operator: In
                 values:
                 - "true"
@@ -21879,21 +21997,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-content
       annotations:
         checksum/config: 524d18d676bdcddf14d63ac8397a0e82dfc86601e69a4684fef07cfaa6953fd8
-        checksum/config_utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/config_utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -22016,21 +22134,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-directory
       annotations:
         checksum/config: 0cacef7c155ca3e3df82cb77ae2b6bb8bb9ac49b19209f2e065c96bc4ba76ef8
-        checksum/config_utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/config_utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -22153,21 +22271,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-extid
       annotations:
         checksum/config: a05be94c147c42ea8e51b234f50f2049496d777e0a00887817305ca00e31b924
-        checksum/config_utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/config_utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -22290,21 +22408,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-metadata
       annotations:
         checksum/config: d237c40e4c2887e8c5268cd357ec129e5ef9d1e9d0fc6ef9603878a0a4f18acf
-        checksum/config_utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/config_utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -22427,21 +22545,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-origin
       annotations:
         checksum/config: 6fb97eb4a6d4f4fa34cb922c7c7a2d54079ade1818d0146e5c7bb9d299b4fd34
-        checksum/config_utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/config_utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -22564,21 +22682,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-origin-visit
       annotations:
         checksum/config: 8feb5282dc24309e969ad447d420801a0cd826820817bcef0862df9805aea973
-        checksum/config_utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/config_utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -22701,21 +22819,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-origin-visit-status
       annotations:
         checksum/config: a345553096d29938201239f63e9d271d80fb1e516a63745da5c165060b2de2b8
-        checksum/config_utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/config_utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -22838,21 +22956,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-raw-extrinsic-metadata
       annotations:
         checksum/config: 43306ba69c4c8557fbd4bfca185bd5edfb8f70fc84bee6667019bd333f602054
-        checksum/config_utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/config_utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -22975,21 +23093,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-release
       annotations:
         checksum/config: eb1a796fdffbeba1776f6709ddd4855d41e82fd600284d8095e95fe1419c651d
-        checksum/config_utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/config_utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -23112,21 +23230,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-revision
       annotations:
         checksum/config: cf1bf19391e5330e49b63bd2a1baf795b6fc08a8229fa4e774ca960fb05eaaf9
-        checksum/config_utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/config_utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -23249,21 +23367,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-skipped-content
       annotations:
         checksum/config: 4213e5e6ff3ae51225dc1ec6f9f48a8ee3f206e6b8e69fd61dfe6335437d9bf9
-        checksum/config_utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/config_utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -23386,21 +23504,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-snapshot
       annotations:
         checksum/config: d10ea86ff29eebbdd9fbf159d297fed599fe1833a8140149f0229df7eaba1b34
-        checksum/config_utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/config_utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -23524,21 +23642,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-cassandra
       annotations:
         checksum/config: b840eaf8faafacbf7f7f08c78e69c3b7028b992f5681acb5de76b177f0a2b3a9
         checksum/config-logging: 2f7a56936b194188f70175c52dc180320fcc071e5c110562a9f031116fadefd2
-        checksum/backend-utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/backend-utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
         checksum/config-utils: 13a26f6add17e96ce01550153c77dcd48de60241a3f4db3c93d5467234be2a7f
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/storage
                 operator: In
                 values:
@@ -23695,21 +23813,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-cassandra-read-only
       annotations:
         checksum/config: 5fe1511b81d6079cf97c2918094816926a283f83778bec40e117a7100441a4c9
         checksum/config-logging: 7403d71b4a2e4da28cc9c1af0b9d022e85bf0b5ffcb738dc9f2b6dcfa3e14456
-        checksum/backend-utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/backend-utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
         checksum/config-utils: 13a26f6add17e96ce01550153c77dcd48de60241a3f4db3c93d5467234be2a7f
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/storage
                 operator: In
                 values:


------------- diff for environment staging namespace next-version -------------

--- /tmp/swh-chart.swh.lQqjoZXu/staging-next-version.before	2025-03-21 17:04:33.857098261 +0100
+++ /tmp/swh-chart.swh.lQqjoZXu/staging-next-version.after	2025-03-21 17:04:34.469074721 +0100
@@ -4005,21 +4005,21 @@
     fi
 
     graph_transposed_name=${GRAPH_NAME}-transposed.graph
     if [ -L ${DATASET_LOCATION}/${graph_transposed_name} ] || ! [ -f ${DATASET_LOCATION}/${graph_transposed_name} ]; then
       cp -v --remove-destination ${DATASET_SOURCE}/${graph_transposed_name} ${DATASET_LOCATION}/;
     fi
 
     # Finally, we make explicit the graph is ready
     touch ${WITNESS_FILE}
 
-  graph-wait-for-dataset.sh: |
+  wait-for-dataset.sh: |
     #!/usr/bin/env bash
     # Uses env variables WITNESS_FILE
     [ -z "${WITNESS_FILE}" ] && \
       echo "<WITNESS_FILE> env variable must be set" && exit 1
 
     while [ ! -f ${WITNESS_FILE} ]; do
         echo "${WITNESS_FILE} not present, wait for it to start the graph..."
         sleep $PERIOD
     done
 
@@ -4101,20 +4101,125 @@
         echo "${WITNESS_SOURCE_FILE} missing, waiting graph dataset installation..."
         sleep $PERIOD
     done
 
     # For old datasets missing a .ef or in the wrong format, this fails with
     # `Cannot map Elias-Fano pointer list .../graph.ef`. The solution is to
     # reindex the dataset
     swh graph reindex --ef ${DATASET_LOCATION}/${GRAPH_NAME} && \
       touch $WITNESS_REINDEX_FILE
 
+  provenance-fetch-datasets.sh: |
+    #!/usr/bin/env bash
+    [ -z "${WITNESS_FETCH_FILE}" ] && \
+      echo "<WITNESS_FETCH_FILE> env variable must be set" && exit 1
+    [ -z "${DATASET_VERSION}" ] && \
+      echo "<DATASET_VERSION> env variable must be set" && exit 1
+    [ -z "${PROVENANCE_PATH}" ] && \
+      echo "<PROVENANCE_PATH> env variable must be set" && exit 1
+    [ -z "${GRAPH_PATH}" ] && \
+      echo "<GRAPH_PATH> env variable must be set" && exit 1
+
+    [ -f ${WITNESS_FETCH_FILE} ] && \
+        echo "Datasets graph & provenance <${DATASET_VERSION}> already present. Skip." && \
+        exit 0
+
+    set -e
+
+    # Create destination paths
+    mkdir -p ${PROVENANCE_PATH} ${GRAPH_PATH}
+
+    echo "Fetching datasets..."
+
+    if [ ${PROVENANCE_DATASET_FULL} = true ]; then
+        # Retrieve all the provenance dataset
+        REFS=all
+    else
+        # This excludes revisions not targetted by a snapshot
+        # Ok to use for test purposes
+        REFS=heads
+    fi
+
+    URL_PROVENANCE="s3://softwareheritage/derived_datasets/${DATASET_VERSION}/provenance/${REFS}/"
+
+    CMD_GET="aws s3 cp --no-progress --no-sign-request"
+
+    echo "1. Fetching provenance dataset (parquet files)..."
+    ${CMD_GET} --recursive "${URL_PROVENANCE}" "${PROVENANCE_PATH}"
+    echo "1. Provenance datasets installed!"
+
+    echo "2. Fetching extra graph files..."
+    URL_GRAPH="s3://softwareheritage/graph/${DATASET_VERSION}/compressed"
+
+    mkdir -p "${GRAPH_PATH}"
+    for filename in graph.pthash graph.pthash.order graph.nodes.count.txt \
+                    graph.property.message.bin.zst \
+                    graph.property.message.offset.bin.zst \
+                    graph.property.tag_name.bin.zst \
+                    graph.property.tag_name.offset.bin.zst \
+                    graph.node2swhid.bin.zst graph.node2type.bin.zst; do
+        ${CMD_GET} "${URL_GRAPH}/${filename}" "${GRAPH_PATH}"
+    done
+    echo "2. Extra graph files installed!"
+
+    echo "3. Uncompressing graph files..."
+    set -x
+    # Uncompress the compressed graph *.zst files
+    for filepath in $(ls ${GRAPH_PATH}/*.zst); do
+        # Uncompress and delete the .zst file
+        [ -f "${filepath}" ] && unzstd --force --rm "${filepath}"
+    done
+    set +x
+    echo "3. Graph files uncompressed!"
+
+    # Make explicit the provenance datasets are fetched
+    touch ${WITNESS_FETCH_FILE}
+
+    echo "Provenance datasets installed!"
+
+  provenance-index-dataset.sh: |
+    #!/usr/bin/env bash
+    [ -z "${WITNESS_DATASETS_FILE}" ] && \
+      echo "<WITNESS_DATASETS_FILE> env variable must be set" && exit 1
+    [ -z "${WITNESS_INDEX_FILE}" ] && \
+      echo "<WITNESS_INDEX_FILE> env variable must be set" && exit 1
+    [ -z "${PERIOD}" ] && \
+      echo "<PERIOD> env variable must be set" && exit 1
+    [ -z "${PROVENANCE_PATH}" ] && \
+      echo "<PROVENANCE_PATH> env variable must be set" && exit 1
+
+    [ -f ${WITNESS_INDEX_FILE} ] && echo "Provenance already indexed, do nothing." && \
+      exit 0
+
+    set -eu
+
+    # Let's wait for the dataset installation
+    while [ ! -f "${WITNESS_DATASETS_FILE}" ]; do
+        echo "${WITNESS_DATASETS_FILE} missing, waiting provenance dataset installation..."
+        sleep $PERIOD
+    done
+
+    echo "Datasets file installed, build provenance dataset indexes..."
+
+    echo "provenance path: $PROVENANCE_PATH"
+    set -x
+
+    # To make the query faster, the provenance needs to build index out of the
+    # current dataset files. We store the output indexes in the same path as
+    # the dataset.
+    swh-provenance-index \
+      --database file://${PROVENANCE_PATH} && \
+      touch "${WITNESS_INDEX_FILE}" && \
+      echo "Provenance indexes built!" || \
+
+    echo "Provenance indexes failed!"
+
   initialize-search-backend.sh: |
     #!/usr/bin/env bash
 
     set -eux
 
     # Uses internally the environment variable SWH_CONFIG_FILENAME
     swh search initialize
   register-task-types.sh: |
     #!/usr/bin/env bash
 
@@ -4669,23 +4774,22 @@
       enable_requests_retry: true
       url: http://storage-ro-cassandra:5002
   
     corner_ribbon_text: StagingNextVersion
     show_corner_ribbon: "true"
     search:
       cls: remote
       enable_requests_retry: true
       url: http://search-rpc:5010
     provenance:
-      cls: remote
-      enable_requests_retry: true
-      url: http://webapp-provenance-ingress-next-version
+      cls: grpc
+      url: provenance-grpc-next-version-ingress:80
     scheduler:
       cls: remote
       url: http://scheduler-rpc:5008
     vault:
       cls: remote
       enable_requests_retry: true
       url: http://vault-rpc:5005
     graph:
       max_edges:
         anonymous: 1000
@@ -4803,20 +4907,37 @@
         - kafka-cluster-kafka-brokers:9092
       auto_offset_reset: latest
       group_id: staging-next-version-archive-webhooks
       object_types:
       - origin_visit_status
 ---
 # Source: swh/templates/volumes/persistent-volume-claims.yaml
 apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
+  name: provenance-popular-persistent-pvc
+  namespace: swh-cassandra-next-version
+  labels:
+    app: provenance-grpc
+spec:
+  accessModes:
+  - ReadWriteOnce
+  resources:
+    requests:
+      storage: 1Gi
+  storageClassName: local-persistent
+  volumeMode: Filesystem
+---
+# Source: swh/templates/volumes/persistent-volume-claims.yaml
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata:
   name: swh-graph-grpc-dataset-example-pvc
   namespace: swh-cassandra-next-version
   labels:
     app: graph-grpc-example
 spec:
   accessModes:
   - ReadWriteOnce
   resources:
     requests:
       storage: 1Gi
@@ -5015,20 +5136,30 @@
   name: webapp-provenance-ingress-next-version
   namespace: swh-cassandra-next-version
 spec:
   type: ExternalName
   externalName: archive-staging-rke2-ingress-nginx-controller.ingress-nginx.svc.cluster.local
 ---
 # Source: swh/templates/external-services/cname.yaml
 apiVersion: v1
 kind: Service
 metadata:
+  name: provenance-grpc-next-version-ingress
+  namespace: swh-cassandra-next-version
+spec:
+  type: ExternalName
+  externalName: archive-staging-rke2-ingress-nginx-controller.ingress-nginx.svc.cluster.local
+---
+# Source: swh/templates/external-services/cname.yaml
+apiVersion: v1
+kind: Service
+metadata:
   name: graph-rpc-ingress
   namespace: swh-cassandra-next-version
 spec:
   type: ExternalName
   externalName: archive-staging-rke2-ingress-nginx-controller.ingress-nginx.svc.cluster.local
 ---
 # Source: swh/templates/external-services/cname.yaml
 apiVersion: v1
 kind: Service
 metadata:
@@ -5231,20 +5362,37 @@
     app: provenance-graph-granet
 spec:
   type: ClusterIP
   selector:
     app: provenance-graph-granet
   ports:
     - port: 5014
       targetPort: 5014
       name: rpc
 ---
+# Source: swh/templates/provenance/service.yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: provenance-grpc
+  namespace: swh-cassandra-next-version
+  labels:
+    app: provenance-grpc
+spec:
+  type: ClusterIP
+  selector:
+    app: provenance-grpc
+  ports:
+    - port: 50141
+      targetPort: 50141
+      name: grpc
+---
 # Source: swh/templates/scheduler/rpc-service.yaml
 apiVersion: v1
 kind: Service
 metadata:
   name: scheduler-rpc
   namespace: swh-cassandra-next-version
   labels:
     app: scheduler-rpc
 spec:
   type: ClusterIP
@@ -6052,21 +6200,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: graph-grpc-example
       annotations:
         checksum/config: e29fc7dba6ab2f3d519be71fcac3361e63dfcbb4b3655a0a78439168961022bb
         checksum/config-utils: 94d255131467f84bef964a4c72b2b792c5ebaf711bb1c77829d7cd1007a8ac22
-        checksum/backend-utils: 3d2301f0fc8b4715e380acef12da66bdd16981f91351f1269a057ac022babc5a
+        checksum/backend-utils: 979dcd12a9ecdb43f1c4c9191012acbc31c14afa348c1b91e8dfb2aa7d105fae
     spec:
       nodeSelector:
         kubernetes.io/hostname: rancher-node-staging-rke2-metal01
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/graph
                 operator: In
@@ -6121,21 +6269,21 @@
         
           - name: swh-graph-grpc-inmemory
             mountPath: /srv/graph
             readOnly: false
         
         
         - name: wait-for-dataset
           image: container-registry.softwareheritage.org/swh/infra/swh-apps/utils:20250211.1
           imagePullPolicy: IfNotPresent
           command:
-          - /entrypoints/graph-wait-for-dataset.sh
+          - /entrypoints/wait-for-dataset.sh
           env:
             - name: WITNESS_FILE
               value: /srv/graph/test/compressed/.graph-is-initialized
             - name: DATASET_LOCATION
               value: /srv/graph/test/compressed
             - name: PERIOD
               value: "3"
           volumeMounts:
           - name: backend-utils
             mountPath: /entrypoints
@@ -6266,21 +6414,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: graph-rpc-example
       annotations:
         checksum/config: 670b17f81e650742736762ab973d6485c0a8573c2052b1357a1fc94534856b7d
         checksum/config-utils: 94d255131467f84bef964a4c72b2b792c5ebaf711bb1c77829d7cd1007a8ac22
-        checksum/backend-utils: 3d2301f0fc8b4715e380acef12da66bdd16981f91351f1269a057ac022babc5a
+        checksum/backend-utils: 979dcd12a9ecdb43f1c4c9191012acbc31c14afa348c1b91e8dfb2aa7d105fae
     spec:
       nodeSelector:
         kubernetes.io/hostname: rancher-node-staging-rke2-metal01
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/graph
                 operator: In
@@ -6516,21 +6664,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: indexer-storage-rw
       annotations:
         checksum/config: 5ab3a947db0a5acead8abc2f73699f859d84fcb4aa54856a72245fa1d471f963
         checksum/config-logging: 514b813a6cc082d2a14192b1b6946c52c586735490a886eb2a498eaf2da4e731
-        checksum/backend-utils: 3d2301f0fc8b4715e380acef12da66bdd16981f91351f1269a057ac022babc5a
+        checksum/backend-utils: 979dcd12a9ecdb43f1c4c9191012acbc31c14afa348c1b91e8dfb2aa7d105fae
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/rpc
                 operator: In
                 values:
                 - "true"
@@ -9015,20 +9163,21 @@
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: provenance-graph-granet
       annotations:
         checksum/config: 3a920cab49ad7bb0f2c6e36ef83ad7764740050a70f204675c9b36eb544a59b1
         checksum/config-logging: 3ec68ca129865387885cf527bf08f90bda9e6d3ae5e50d948534cbe73306d6fb
         checksum/config-utils: 94d255131467f84bef964a4c72b2b792c5ebaf711bb1c77829d7cd1007a8ac22
+        checksum/backend-utils: 979dcd12a9ecdb43f1c4c9191012acbc31c14afa348c1b91e8dfb2aa7d105fae
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/rpc
                 operator: In
                 values:
                 - "true"
@@ -9048,21 +9197,21 @@
             mountPath: /etc/swh/configuration-template
           - name: config-utils
             mountPath: /entrypoints
             readOnly: true
       containers:
         - name: provenance-graph-granet
           resources:
             requests:
               memory: 512Mi
               cpu: 500m
-          image: container-registry.softwareheritage.org/swh/infra/swh-apps/provenance:20250319.1
+          image: container-registry.softwareheritage.org/swh/infra/swh-apps/provenance:20250321.1
           imagePullPolicy: IfNotPresent
           ports:
             - containerPort: 5014
               name: rpc
           readinessProbe:
             httpGet:
               path: /
               port: rpc
             initialDelaySeconds: 15
             failureThreshold: 30
@@ -9071,76 +9220,270 @@
             tcpSocket:
               port: rpc
             initialDelaySeconds: 10
             periodSeconds: 5
           command:
           - /bin/bash
           args:
           - -c
           - /opt/swh/entrypoint.sh
           env:
+            - name: PROVENANCE_TYPE
+              value: rpc
+            - name: PORT
+              value: "5014"
             - name: WORKERS
               value: "2"
             - name: THREADS
               value: "2"
             - name: TIMEOUT
               value: "60"
+            - name: SWH_CONFIG_FILENAME
+              value: /etc/swh/config.yml
+            - name: SWH_LOG_CONFIG_JSON
+              value: /etc/swh/logging/logging-gunicorn.json
+            - name: STATSD_SERVICE_TYPE
+              value: provenance-graph-granet
             - name: STATSD_HOST
               value: prometheus-statsd-exporter
             - name: STATSD_PORT
               value: "9125"
             - name: STATSD_TAGS
               value: deployment:provenance-graph-granet
-            - name: STATSD_SERVICE_TYPE
-              value: provenance-graph-granet
             - name: SWH_LOG_LEVEL
-              value: "INFO"
-            - name: SWH_LOG_CONFIG_JSON
-              value: /etc/swh/logging/logging-gunicorn.json
+              value: INFO
             - name: SWH_SENTRY_ENVIRONMENT
               value: staging
             - name: SWH_MAIN_PACKAGE
               value: swh.provenance
             - name: SWH_SENTRY_DSN
               valueFrom:
                 secretKeyRef:
                   name: common-secrets
                   key: provenance-sentry-dsn
                   # 'name' secret should exist & include key
                   # if the setting doesn't exist, sentry pushes will be disabled
                   optional: true
             - name: SWH_SENTRY_DISABLE_LOGGING_EVENTS
               value: "true"
           volumeMounts:
           - name: configuration
             mountPath: /etc/swh
           - name: configuration-logging
             mountPath: /etc/swh/logging
+          
       volumes:
       - name: configuration
         emptyDir: {}
       - name: configuration-template
         configMap:
           name: provenance-graph-granet-configuration-template
           items:
           - key: "config.yml.template"
             path: "config.yml.template"
       - name: configuration-logging
         configMap:
           name: provenance-graph-granet-configuration-logging
           items:
           - key: "logging-gunicorn.json"
             path: "logging-gunicorn.json"
+      
       - name: config-utils
         configMap:
           name: config-utils
           defaultMode: 0555
+      - name: backend-utils
+        configMap:
+          name: backend-utils
+          defaultMode: 0555
+---
+# Source: swh/templates/provenance/deployment.yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  namespace: swh-cassandra-next-version
+  name: provenance-grpc
+  labels:
+    app: provenance-grpc
+spec:
+  revisionHistoryLimit: 2
+  selector:
+    matchLabels:
+      app: provenance-grpc
+  strategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxSurge: 1
+  template:
+    metadata:
+      labels:
+        app: provenance-grpc
+      annotations:
+        checksum/config: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
+        checksum/config-logging: 9fa299d379f661eab9d312fce16ef38fb94e197e908b92ac100aff85b4c36bb4
+        checksum/config-utils: 94d255131467f84bef964a4c72b2b792c5ebaf711bb1c77829d7cd1007a8ac22
+        checksum/backend-utils: 979dcd12a9ecdb43f1c4c9191012acbc31c14afa348c1b91e8dfb2aa7d105fae
+    spec:
+      affinity:
+        nodeAffinity:
+          requiredDuringSchedulingIgnoredDuringExecution:
+            nodeSelectorTerms:
+            - matchExpressions:
+              - key: swh/rpc
+                operator: In
+                values:
+                - "true"
+      priorityClassName: swh-cassandra-next-version-frontend-rpc
+      initContainers:
+        - name: fetch-provenance-dataset
+          image: container-registry.softwareheritage.org/swh/infra/swh-apps/provenance:20250321.1
+          command:
+          - /entrypoints/provenance-fetch-datasets.sh
+          env:
+          - name: WITNESS_FETCH_FILE
+            value: /srv/dataset/provenance/.provenance-is-initialized
+          - name: SWH_CONFIG_FILENAME
+            value: /etc/swh/config.yml
+          - name: PROVENANCE_PATH
+            value: /srv/dataset/provenance
+          - name: PROVENANCE_DATASET_FULL
+            value: "false"
+          - name: GRAPH_PATH
+            value: /srv/dataset/graph
+          - name: DATASET_VERSION
+            value: 2024-08-23-popular-500-python
+          volumeMounts:
+          - name: configuration
+            mountPath: /etc/swh
+          - name: backend-utils
+            mountPath: /entrypoints
+          - name: dataset-persistent
+            mountPath: /srv/dataset
+            readOnly: false
+          
+        - name: index-provenance-dataset
+          image: container-registry.softwareheritage.org/swh/infra/swh-apps/provenance:20250321.1
+          imagePullPolicy: IfNotPresent
+          command:
+          - /entrypoints/provenance-index-dataset.sh
+          env:
+            - name: WITNESS_DATASETS_FILE
+              value: /srv/dataset/provenance/.provenance-is-initialized
+            - name: WITNESS_INDEX_FILE
+              value: /srv/dataset/provenance/.provenance-is-indexed
+            - name: PROVENANCE_PATH
+              value: /srv/dataset/provenance
+            - name: PERIOD
+              value: "3"
+          volumeMounts:
+          - name: backend-utils
+            mountPath: /entrypoints
+            readOnly: true
+          - name: dataset-persistent
+            mountPath: /srv/dataset
+            readOnly: false
+          
+        - name: wait-for-dataset
+          image: container-registry.softwareheritage.org/swh/infra/swh-apps/utils:20250211.1
+          imagePullPolicy: IfNotPresent
+          command:
+          - /entrypoints/wait-for-dataset.sh
+          env:
+            - name: WITNESS_FILE
+              value: /srv/dataset/provenance/.provenance-is-initialized
+            - name: PERIOD
+              value: "3"
+          volumeMounts:
+          - name: backend-utils
+            mountPath: /entrypoints
+            readOnly: true
+          - name: dataset-persistent
+            mountPath: /srv/dataset
+            readOnly: false
+          
+      containers:
+        - name: provenance-grpc
+          resources:
+            requests:
+              memory: 512Mi
+              cpu: 500m
+          image: container-registry.softwareheritage.org/swh/infra/swh-apps/provenance:20250321.1
+          imagePullPolicy: IfNotPresent
+          ports:
+            - containerPort: 50141
+              name: grpc
+          readinessProbe:
+            tcpSocket:
+              port: grpc
+            initialDelaySeconds: 15
+            failureThreshold: 30
+            periodSeconds: 5
+          livenessProbe:
+            tcpSocket:
+              port: grpc
+            initialDelaySeconds: 10
+            periodSeconds: 5
+          command:
+          - /bin/bash
+          args:
+          - -c
+          - /opt/swh/entrypoint.sh
+          env:
+            - name: PROVENANCE_TYPE
+              value: grpc
+            - name: PORT
+              value: "50141"
+            - name: PROVENANCE_PATH
+              value: /srv/dataset/provenance
+            - name: GRAPH_PATH
+              value: /srv/dataset/graph/graph
+            - name: STATSD_HOST
+              value: prometheus-statsd-exporter
+            - name: STATSD_PORT
+              value: "9125"
+            - name: STATSD_TAGS
+              value: deployment:provenance-grpc
+            - name: SWH_LOG_LEVEL
+              value: INFO
+            - name: SWH_SENTRY_ENVIRONMENT
+              value: staging
+            - name: SWH_MAIN_PACKAGE
+              value: swh.provenance
+            - name: SWH_SENTRY_DSN
+              valueFrom:
+                secretKeyRef:
+                  name: common-secrets
+                  key: provenance-sentry-dsn
+                  # 'name' secret should exist & include key
+                  # if the setting doesn't exist, sentry pushes will be disabled
+                  optional: true
+            - name: SWH_SENTRY_DISABLE_LOGGING_EVENTS
+              value: "true"
+          volumeMounts:
+          - name: dataset-persistent
+            mountPath: /srv/dataset
+            readOnly: false
+          
+      volumes:
+      - name: configuration
+        emptyDir: {}
+      - name: config-utils
+        configMap:
+          name: config-utils
+          defaultMode: 0555
+      - name: backend-utils
+        configMap:
+          name: backend-utils
+          defaultMode: 0555
+      - name: dataset-persistent
+        persistentVolumeClaim:
+          claimName: provenance-popular-persistent-pvc
 ---
 # Source: swh/templates/scheduler/extra-services-deployment.yaml
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   namespace: swh-cassandra-next-version
   name: scheduler-listener
   labels:
     app: scheduler-listener
 spec:
@@ -10090,21 +10433,21 @@
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: search-rpc
       annotations:
         checksum/config: 4526bc9eea3c5b071fc8d14d9a9993ae1819e8c7de57d97340b7f3d4a10b8b4f
         checksum/config-logging: 0bc72d1f0a5e779cba1b812d82f00ae1973bb8c5140ff975d94cb4e7f4181000
         checksum/config-utils: 94d255131467f84bef964a4c72b2b792c5ebaf711bb1c77829d7cd1007a8ac22
-        checksum/backend-utils: 3d2301f0fc8b4715e380acef12da66bdd16981f91351f1269a057ac022babc5a
+        checksum/backend-utils: 979dcd12a9ecdb43f1c4c9191012acbc31c14afa348c1b91e8dfb2aa7d105fae
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/rpc
                 operator: In
                 values:
                 - "true"
@@ -10290,21 +10633,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-origin
       annotations:
         checksum/config: 2d12ac61e189624151390a01fbdc28c53b735945f71f4b4fbe01a83a1065bd34
-        checksum/config_utils: 3d2301f0fc8b4715e380acef12da66bdd16981f91351f1269a057ac022babc5a
+        checksum/config_utils: 979dcd12a9ecdb43f1c4c9191012acbc31c14afa348c1b91e8dfb2aa7d105fae
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: node-role.kubernetes.io/etcd
                 operator: NotIn
                 values:
                 - "true"
@@ -10408,21 +10751,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-ro-cassandra
       annotations:
         checksum/config: 82ddd066ef3b39ba30a9377b2e3d5f34e9c6771d244cb9966170d8821659043e
         checksum/config-logging: 539b1f63c51f751ac609a212d1d29b53ccee1fe8861d4b40cf156cabdbbcc9af
-        checksum/backend-utils: 3d2301f0fc8b4715e380acef12da66bdd16981f91351f1269a057ac022babc5a
+        checksum/backend-utils: 979dcd12a9ecdb43f1c4c9191012acbc31c14afa348c1b91e8dfb2aa7d105fae
         checksum/config-utils: 94d255131467f84bef964a4c72b2b792c5ebaf711bb1c77829d7cd1007a8ac22
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/storage
                 operator: In
                 values:
@@ -10565,21 +10908,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-ro-postgresql
       annotations:
         checksum/config: ee389a0c378431ad964e527086008e8983bd8181007a5cef4bdc6b2f3719b891
         checksum/config-logging: 04d1f9a399d1326e46f5b78ca589b7dc5a6187afba6e61a46a7b333d07cb16aa
-        checksum/backend-utils: 3d2301f0fc8b4715e380acef12da66bdd16981f91351f1269a057ac022babc5a
+        checksum/backend-utils: 979dcd12a9ecdb43f1c4c9191012acbc31c14afa348c1b91e8dfb2aa7d105fae
         checksum/config-utils: 94d255131467f84bef964a4c72b2b792c5ebaf711bb1c77829d7cd1007a8ac22
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/storage
                 operator: In
                 values:
@@ -10722,21 +11065,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-rw-cassandra
       annotations:
         checksum/config: 899b95e866ca9a7e5b053415e3df669b133875d96fbf5d91db52e0b105986297
         checksum/config-logging: c4f758383b8e60062735e5b03cfe1724b60a3a961b95632b75c511025ddd5d28
-        checksum/backend-utils: 3d2301f0fc8b4715e380acef12da66bdd16981f91351f1269a057ac022babc5a
+        checksum/backend-utils: 979dcd12a9ecdb43f1c4c9191012acbc31c14afa348c1b91e8dfb2aa7d105fae
         checksum/config-utils: 94d255131467f84bef964a4c72b2b792c5ebaf711bb1c77829d7cd1007a8ac22
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/storage
                 operator: In
                 values:
@@ -10893,21 +11236,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-rw-postgresql
       annotations:
         checksum/config: 053e84875d8867dbb844f84faa7b55a7d5647ab98845e4c0fd8e83ed06aaaf04
         checksum/config-logging: b6c995f5c5944a279018efc4d112fe1395da696289b71ebf4ea382011400cd03
-        checksum/backend-utils: 3d2301f0fc8b4715e380acef12da66bdd16981f91351f1269a057ac022babc5a
+        checksum/backend-utils: 979dcd12a9ecdb43f1c4c9191012acbc31c14afa348c1b91e8dfb2aa7d105fae
         checksum/config-utils: 94d255131467f84bef964a4c72b2b792c5ebaf711bb1c77829d7cd1007a8ac22
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/storage
                 operator: In
                 values:
@@ -11597,21 +11940,21 @@
       app: web-cassandra
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: web-cassandra
       annotations:
-        checksum/config: 412e790069b96930e00d38eea4a3ac5fcaac2ea46e0f75f429be7690318080bf
+        checksum/config: 4c81b069c0173732d5e8c5acb7f38671a4daba9145b4f75fc6dee8d19d42fc1c
         checksum/config-logging: f266f784128ac9c57c6d0f154a646e15f06d0ad7557f191487df0d1b385acb48
         checksum/config-utils: 94d255131467f84bef964a4c72b2b792c5ebaf711bb1c77829d7cd1007a8ac22
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/web
                 operator: In
@@ -12281,20 +12624,51 @@
     http:
       paths:
       - path: /
         pathType: Prefix
         backend:
           service:
             name: provenance-graph-granet
             port:
               number: 5014
 ---
+# Source: swh/templates/provenance/ingress.yaml
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  namespace: swh-cassandra-next-version
+  name: provenance-grpc-ingress-default
+  labels:
+    app: provenance-grpc
+    endpoint-definition: default
+  annotations: 
+    nginx.ingress.kubernetes.io/backend-protocol: GRPC
+    nginx.ingress.kubernetes.io/client-body-buffer-size: 128K
+    nginx.ingress.kubernetes.io/proxy-body-size: 4G
+    nginx.ingress.kubernetes.io/proxy-buffering: "on"
+    nginx.ingress.kubernetes.io/service-upstream: "true"
+    nginx.ingress.kubernetes.io/ssl-redirect: "true"
+    nginx.ingress.kubernetes.io/whitelist-source-range: 10.42.0.0/16,10.43.0.0/16
+spec:
+  ingressClassName: nginx
+  rules:
+  - host: provenance-grpc-next-version-ingress
+    http:
+      paths:
+      - path: /
+        pathType: Prefix
+        backend:
+          service:
+            name: provenance-grpc
+            port:
+              number: 50141
+---
 # Source: swh/templates/web/ingress.yaml
 apiVersion: networking.k8s.io/v1
 kind: Ingress
 metadata:
   namespace: swh-cassandra-next-version
   name: web-cassandra-ingress-authenticated
   labels:
     app: web-cassandra
     endpoint-definition: authenticated
   annotations: 


------------- diff for environment production namespace swh -------------

--- /tmp/swh-chart.swh.lQqjoZXu/production-swh.before	2025-03-21 17:04:34.629068567 +0100
+++ /tmp/swh-chart.swh.lQqjoZXu/production-swh.after	2025-03-21 17:04:35.153048411 +0100
@@ -2585,21 +2585,21 @@
     fi
 
     graph_transposed_name=${GRAPH_NAME}-transposed.graph
     if [ -L ${DATASET_LOCATION}/${graph_transposed_name} ] || ! [ -f ${DATASET_LOCATION}/${graph_transposed_name} ]; then
       cp -v --remove-destination ${DATASET_SOURCE}/${graph_transposed_name} ${DATASET_LOCATION}/;
     fi
 
     # Finally, we make explicit the graph is ready
     touch ${WITNESS_FILE}
 
-  graph-wait-for-dataset.sh: |
+  wait-for-dataset.sh: |
     #!/usr/bin/env bash
     # Uses env variables WITNESS_FILE
     [ -z "${WITNESS_FILE}" ] && \
       echo "<WITNESS_FILE> env variable must be set" && exit 1
 
     while [ ! -f ${WITNESS_FILE} ]; do
         echo "${WITNESS_FILE} not present, wait for it to start the graph..."
         sleep $PERIOD
     done
 
@@ -2681,20 +2681,125 @@
         echo "${WITNESS_SOURCE_FILE} missing, waiting graph dataset installation..."
         sleep $PERIOD
     done
 
     # For old datasets missing a .ef or in the wrong format, this fails with
     # `Cannot map Elias-Fano pointer list .../graph.ef`. The solution is to
     # reindex the dataset
     swh graph reindex --ef ${DATASET_LOCATION}/${GRAPH_NAME} && \
       touch $WITNESS_REINDEX_FILE
 
+  provenance-fetch-datasets.sh: |
+    #!/usr/bin/env bash
+    [ -z "${WITNESS_FETCH_FILE}" ] && \
+      echo "<WITNESS_FETCH_FILE> env variable must be set" && exit 1
+    [ -z "${DATASET_VERSION}" ] && \
+      echo "<DATASET_VERSION> env variable must be set" && exit 1
+    [ -z "${PROVENANCE_PATH}" ] && \
+      echo "<PROVENANCE_PATH> env variable must be set" && exit 1
+    [ -z "${GRAPH_PATH}" ] && \
+      echo "<GRAPH_PATH> env variable must be set" && exit 1
+
+    [ -f ${WITNESS_FETCH_FILE} ] && \
+        echo "Datasets graph & provenance <${DATASET_VERSION}> already present. Skip." && \
+        exit 0
+
+    set -e
+
+    # Create destination paths
+    mkdir -p ${PROVENANCE_PATH} ${GRAPH_PATH}
+
+    echo "Fetching datasets..."
+
+    if [ ${PROVENANCE_DATASET_FULL} = true ]; then
+        # Retrieve all the provenance dataset
+        REFS=all
+    else
+        # This excludes revisions not targetted by a snapshot
+        # Ok to use for test purposes
+        REFS=heads
+    fi
+
+    URL_PROVENANCE="s3://softwareheritage/derived_datasets/${DATASET_VERSION}/provenance/${REFS}/"
+
+    CMD_GET="aws s3 cp --no-progress --no-sign-request"
+
+    echo "1. Fetching provenance dataset (parquet files)..."
+    ${CMD_GET} --recursive "${URL_PROVENANCE}" "${PROVENANCE_PATH}"
+    echo "1. Provenance datasets installed!"
+
+    echo "2. Fetching extra graph files..."
+    URL_GRAPH="s3://softwareheritage/graph/${DATASET_VERSION}/compressed"
+
+    mkdir -p "${GRAPH_PATH}"
+    for filename in graph.pthash graph.pthash.order graph.nodes.count.txt \
+                    graph.property.message.bin.zst \
+                    graph.property.message.offset.bin.zst \
+                    graph.property.tag_name.bin.zst \
+                    graph.property.tag_name.offset.bin.zst \
+                    graph.node2swhid.bin.zst graph.node2type.bin.zst; do
+        ${CMD_GET} "${URL_GRAPH}/${filename}" "${GRAPH_PATH}"
+    done
+    echo "2. Extra graph files installed!"
+
+    echo "3. Uncompressing graph files..."
+    set -x
+    # Uncompress the compressed graph *.zst files
+    for filepath in $(ls ${GRAPH_PATH}/*.zst); do
+        # Uncompress and delete the .zst file
+        [ -f "${filepath}" ] && unzstd --force --rm "${filepath}"
+    done
+    set +x
+    echo "3. Graph files uncompressed!"
+
+    # Make explicit the provenance datasets are fetched
+    touch ${WITNESS_FETCH_FILE}
+
+    echo "Provenance datasets installed!"
+
+  provenance-index-dataset.sh: |
+    #!/usr/bin/env bash
+    [ -z "${WITNESS_DATASETS_FILE}" ] && \
+      echo "<WITNESS_DATASETS_FILE> env variable must be set" && exit 1
+    [ -z "${WITNESS_INDEX_FILE}" ] && \
+      echo "<WITNESS_INDEX_FILE> env variable must be set" && exit 1
+    [ -z "${PERIOD}" ] && \
+      echo "<PERIOD> env variable must be set" && exit 1
+    [ -z "${PROVENANCE_PATH}" ] && \
+      echo "<PROVENANCE_PATH> env variable must be set" && exit 1
+
+    [ -f ${WITNESS_INDEX_FILE} ] && echo "Provenance already indexed, do nothing." && \
+      exit 0
+
+    set -eu
+
+    # Let's wait for the dataset installation
+    while [ ! -f "${WITNESS_DATASETS_FILE}" ]; do
+        echo "${WITNESS_DATASETS_FILE} missing, waiting provenance dataset installation..."
+        sleep $PERIOD
+    done
+
+    echo "Datasets file installed, build provenance dataset indexes..."
+
+    echo "provenance path: $PROVENANCE_PATH"
+    set -x
+
+    # To make the query faster, the provenance needs to build index out of the
+    # current dataset files. We store the output indexes in the same path as
+    # the dataset.
+    swh-provenance-index \
+      --database file://${PROVENANCE_PATH} && \
+      touch "${WITNESS_INDEX_FILE}" && \
+      echo "Provenance indexes built!" || \
+
+    echo "Provenance indexes failed!"
+
   initialize-search-backend.sh: |
     #!/usr/bin/env bash
 
     set -eux
 
     # Uses internally the environment variable SWH_CONFIG_FILENAME
     swh search initialize
   register-task-types.sh: |
     #!/usr/bin/env bash
 
@@ -4386,21 +4491,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-content
       annotations:
         checksum/config: ad9969915c9d4f098e176250342e634c3f9950c21b4bfce3c59a756eebd29d5a
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -4509,21 +4614,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-directory
       annotations:
         checksum/config: d10ee8d64b973f71f54ac97d9b23a984ddcaf85a14e4e7d0c1ffbe6606745a9f
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -4632,21 +4737,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-extid
       annotations:
         checksum/config: d6e51b2acf85824083b41c3fc454e0bde5cda180b13fd0ea0f7a90de3b13dd10
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -4755,21 +4860,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-metadata
       annotations:
         checksum/config: 568c27f168a777aa1cd02d52a482105f91ea07a5ca96c490faecb6f0f126d510
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -4878,21 +4983,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-origin
       annotations:
         checksum/config: 5a0c927d61eea568c15a84881cdcc36061b4c530db7c11f565ede65c5b3936c3
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -5001,21 +5106,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-origin-visit
       annotations:
         checksum/config: 0e471b3f26d83e3d374296d1a6ce7077b31cb935705c692929ee6332dacb03fa
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -5124,21 +5229,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-origin-visit-status
       annotations:
         checksum/config: 87f95fa2d03d52fdec4dcbb83ba5e790821fe8d739e8922a97046f0d2a10abae
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -5247,21 +5352,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-raw-extrinsic-metadata
       annotations:
         checksum/config: daffe1093b0bb5c08485e25baeb2f10f5f39f0fd826c3a40283693a9d43fae37
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -5370,21 +5475,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-release
       annotations:
         checksum/config: 41773bc062731699b038ae98bec197c7290766735e9ae57977da8d0d1b0a82d5
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -5493,21 +5598,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-revision
       annotations:
         checksum/config: 24e62d01eeed14fdd9eab5bcbbdfc2165b37c246e93270c6a3593a86760b296f
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -5616,21 +5721,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-skipped-content
       annotations:
         checksum/config: 61ed93650af06dedc4ee939164381d2e9ac65098d43af26405e34c2d6fd3cae8
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -5739,21 +5844,21 @@
   strategy:
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-replayer-snapshot
       annotations:
         checksum/config: 32cb30621d4c74bac7750080475ba1bece7f561dbc99a798bdbda13b29c5e9c0
-        checksum/config_utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/config_utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/replayer
                 operator: In
                 values:
                 - "true"
@@ -5863,21 +5968,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-postgresql-azure-readonly
       annotations:
         checksum/config: f82d8c731f4709a9e756911b666f9a28a25429375b97c614a4ef5c7bc231e3c8
         checksum/config-logging: 0ecdc326a2b3e525e21e5743d89eb3c4bfbadc12aee4fbe1a32ba77ab7bde899
-        checksum/backend-utils: 82ab9d2291625dd17a30267f551b45870420eefd4a90bb40f14412553c45556a
+        checksum/backend-utils: 233f1b432787895386fcdfff598b35a77ca1f18d4a8f7f0136af55928674c9a9
         checksum/config-utils: d75ca13b805bce6a8ab59c8e24c938f2283108f6a79134f6e71db86308651dc6
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/storage
                 operator: In
                 values:


------------- diff for environment production namespace swh-cassandra -------------

--- /tmp/swh-chart.swh.lQqjoZXu/production-swh-cassandra.before	2025-03-21 17:04:35.009053950 +0100
+++ /tmp/swh-chart.swh.lQqjoZXu/production-swh-cassandra.after	2025-03-21 17:04:35.521034257 +0100
@@ -10509,21 +10509,21 @@
     fi
 
     graph_transposed_name=${GRAPH_NAME}-transposed.graph
     if [ -L ${DATASET_LOCATION}/${graph_transposed_name} ] || ! [ -f ${DATASET_LOCATION}/${graph_transposed_name} ]; then
       cp -v --remove-destination ${DATASET_SOURCE}/${graph_transposed_name} ${DATASET_LOCATION}/;
     fi
 
     # Finally, we make explicit the graph is ready
     touch ${WITNESS_FILE}
 
-  graph-wait-for-dataset.sh: |
+  wait-for-dataset.sh: |
     #!/usr/bin/env bash
     # Uses env variables WITNESS_FILE
     [ -z "${WITNESS_FILE}" ] && \
       echo "<WITNESS_FILE> env variable must be set" && exit 1
 
     while [ ! -f ${WITNESS_FILE} ]; do
         echo "${WITNESS_FILE} not present, wait for it to start the graph..."
         sleep $PERIOD
     done
 
@@ -10605,20 +10605,125 @@
         echo "${WITNESS_SOURCE_FILE} missing, waiting graph dataset installation..."
         sleep $PERIOD
     done
 
     # For old datasets missing a .ef or in the wrong format, this fails with
     # `Cannot map Elias-Fano pointer list .../graph.ef`. The solution is to
     # reindex the dataset
     swh graph reindex --ef ${DATASET_LOCATION}/${GRAPH_NAME} && \
       touch $WITNESS_REINDEX_FILE
 
+  provenance-fetch-datasets.sh: |
+    #!/usr/bin/env bash
+    [ -z "${WITNESS_FETCH_FILE}" ] && \
+      echo "<WITNESS_FETCH_FILE> env variable must be set" && exit 1
+    [ -z "${DATASET_VERSION}" ] && \
+      echo "<DATASET_VERSION> env variable must be set" && exit 1
+    [ -z "${PROVENANCE_PATH}" ] && \
+      echo "<PROVENANCE_PATH> env variable must be set" && exit 1
+    [ -z "${GRAPH_PATH}" ] && \
+      echo "<GRAPH_PATH> env variable must be set" && exit 1
+
+    [ -f ${WITNESS_FETCH_FILE} ] && \
+        echo "Datasets graph & provenance <${DATASET_VERSION}> already present. Skip." && \
+        exit 0
+
+    set -e
+
+    # Create destination paths
+    mkdir -p ${PROVENANCE_PATH} ${GRAPH_PATH}
+
+    echo "Fetching datasets..."
+
+    if [ ${PROVENANCE_DATASET_FULL} = true ]; then
+        # Retrieve all the provenance dataset
+        REFS=all
+    else
+        # This excludes revisions not targetted by a snapshot
+        # Ok to use for test purposes
+        REFS=heads
+    fi
+
+    URL_PROVENANCE="s3://softwareheritage/derived_datasets/${DATASET_VERSION}/provenance/${REFS}/"
+
+    CMD_GET="aws s3 cp --no-progress --no-sign-request"
+
+    echo "1. Fetching provenance dataset (parquet files)..."
+    ${CMD_GET} --recursive "${URL_PROVENANCE}" "${PROVENANCE_PATH}"
+    echo "1. Provenance datasets installed!"
+
+    echo "2. Fetching extra graph files..."
+    URL_GRAPH="s3://softwareheritage/graph/${DATASET_VERSION}/compressed"
+
+    mkdir -p "${GRAPH_PATH}"
+    for filename in graph.pthash graph.pthash.order graph.nodes.count.txt \
+                    graph.property.message.bin.zst \
+                    graph.property.message.offset.bin.zst \
+                    graph.property.tag_name.bin.zst \
+                    graph.property.tag_name.offset.bin.zst \
+                    graph.node2swhid.bin.zst graph.node2type.bin.zst; do
+        ${CMD_GET} "${URL_GRAPH}/${filename}" "${GRAPH_PATH}"
+    done
+    echo "2. Extra graph files installed!"
+
+    echo "3. Uncompressing graph files..."
+    set -x
+    # Uncompress the compressed graph *.zst files
+    for filepath in $(ls ${GRAPH_PATH}/*.zst); do
+        # Uncompress and delete the .zst file
+        [ -f "${filepath}" ] && unzstd --force --rm "${filepath}"
+    done
+    set +x
+    echo "3. Graph files uncompressed!"
+
+    # Make explicit the provenance datasets are fetched
+    touch ${WITNESS_FETCH_FILE}
+
+    echo "Provenance datasets installed!"
+
+  provenance-index-dataset.sh: |
+    #!/usr/bin/env bash
+    [ -z "${WITNESS_DATASETS_FILE}" ] && \
+      echo "<WITNESS_DATASETS_FILE> env variable must be set" && exit 1
+    [ -z "${WITNESS_INDEX_FILE}" ] && \
+      echo "<WITNESS_INDEX_FILE> env variable must be set" && exit 1
+    [ -z "${PERIOD}" ] && \
+      echo "<PERIOD> env variable must be set" && exit 1
+    [ -z "${PROVENANCE_PATH}" ] && \
+      echo "<PROVENANCE_PATH> env variable must be set" && exit 1
+
+    [ -f ${WITNESS_INDEX_FILE} ] && echo "Provenance already indexed, do nothing." && \
+      exit 0
+
+    set -eu
+
+    # Let's wait for the dataset installation
+    while [ ! -f "${WITNESS_DATASETS_FILE}" ]; do
+        echo "${WITNESS_DATASETS_FILE} missing, waiting provenance dataset installation..."
+        sleep $PERIOD
+    done
+
+    echo "Datasets file installed, build provenance dataset indexes..."
+
+    echo "provenance path: $PROVENANCE_PATH"
+    set -x
+
+    # To make the query faster, the provenance needs to build index out of the
+    # current dataset files. We store the output indexes in the same path as
+    # the dataset.
+    swh-provenance-index \
+      --database file://${PROVENANCE_PATH} && \
+      touch "${WITNESS_INDEX_FILE}" && \
+      echo "Provenance indexes built!" || \
+
+    echo "Provenance indexes failed!"
+
   initialize-search-backend.sh: |
     #!/usr/bin/env bash
 
     set -eux
 
     # Uses internally the environment variable SWH_CONFIG_FILENAME
     swh search initialize
   register-task-types.sh: |
     #!/usr/bin/env bash
 
@@ -13978,21 +14083,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: graph-grpc-20241206
       annotations:
         checksum/config: b4edb88c0bcb74769dc2f39025a598580a6d6a39cece80ba52904365cd7380eb
         checksum/config-utils: 13a26f6add17e96ce01550153c77dcd48de60241a3f4db3c93d5467234be2a7f
-        checksum/backend-utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/backend-utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/graph
                 operator: In
                 values:
                 - "true"
@@ -14164,21 +14269,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: graph-rpc-20241206
       annotations:
         checksum/config: 095d223956d75728c8f8a26368053a8882cb3026736517767d8aacfc9895e159
         checksum/config-utils: 13a26f6add17e96ce01550153c77dcd48de60241a3f4db3c93d5467234be2a7f
-        checksum/backend-utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/backend-utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/graph
                 operator: In
                 values:
                 - "true"
@@ -14523,21 +14628,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: indexer-storage-read-only
       annotations:
         checksum/config: e233fce2b3a7a714653810d4a8084763fa3d456d691d5964f00c546ebbaaa49d
         checksum/config-logging: 3c46e3e49b8224015ed0a6ef21fec2ba66c4af22a8718cd0ad4f61483cd5e8be
-        checksum/backend-utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/backend-utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/rpc
                 operator: In
                 values:
                 - "true"
@@ -14670,21 +14775,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: indexer-storage-read-write
       annotations:
         checksum/config: 3b882fc68a6b1c70f8ce6b82965db4903c5f13557d4a0a43fd6d858745c72e90
         checksum/config-logging: 2b18f7d6ed7689e52685ba77e412dffc3ee95be9bddd0aa15d728ee8ef45591d
-        checksum/backend-utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/backend-utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/rpc
                 operator: In
                 values:
                 - "true"
@@ -25360,20 +25465,21 @@
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: provenance-graph-granet
       annotations:
         checksum/config: fcc422ce13f035bd4de309693c6044e4eee6a37fdc487ec2f9fef5437dfd954e
         checksum/config-logging: ddcd27d991938c46f4fc0ad7ee028cb3005f186b3db022596c9ae94363881e4f
         checksum/config-utils: 13a26f6add17e96ce01550153c77dcd48de60241a3f4db3c93d5467234be2a7f
+        checksum/backend-utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/rpc
                 operator: In
                 values:
                 - "true"
@@ -25393,21 +25499,21 @@
             mountPath: /etc/swh/configuration-template
           - name: config-utils
             mountPath: /entrypoints
             readOnly: true
       containers:
         - name: provenance-graph-granet
           resources:
             requests:
               memory: 512Mi
               cpu: 500m
-          image: container-registry.softwareheritage.org/swh/infra/swh-apps/provenance:20250319.1
+          image: container-registry.softwareheritage.org/swh/infra/swh-apps/provenance:20250321.1
           imagePullPolicy: IfNotPresent
           ports:
             - containerPort: 5014
               name: rpc
           readinessProbe:
             httpGet:
               path: /
               port: rpc
             initialDelaySeconds: 15
             failureThreshold: 30
@@ -25416,76 +25522,88 @@
             tcpSocket:
               port: rpc
             initialDelaySeconds: 10
             periodSeconds: 5
           command:
           - /bin/bash
           args:
           - -c
           - /opt/swh/entrypoint.sh
           env:
+            - name: PROVENANCE_TYPE
+              value: rpc
+            - name: PORT
+              value: "5014"
             - name: WORKERS
               value: "4"
             - name: THREADS
               value: "1"
             - name: TIMEOUT
               value: "60"
+            - name: SWH_CONFIG_FILENAME
+              value: /etc/swh/config.yml
+            - name: SWH_LOG_CONFIG_JSON
+              value: /etc/swh/logging/logging-gunicorn.json
+            - name: STATSD_SERVICE_TYPE
+              value: provenance-graph-granet
             - name: STATSD_HOST
               value: prometheus-statsd-exporter
             - name: STATSD_PORT
               value: "9125"
             - name: STATSD_TAGS
               value: deployment:provenance-graph-granet
-            - name: STATSD_SERVICE_TYPE
-              value: provenance-graph-granet
             - name: SWH_LOG_LEVEL
-              value: "INFO"
-            - name: SWH_LOG_CONFIG_JSON
-              value: /etc/swh/logging/logging-gunicorn.json
+              value: INFO
             - name: SWH_SENTRY_ENVIRONMENT
               value: production
             - name: SWH_MAIN_PACKAGE
               value: swh.provenance
             - name: SWH_SENTRY_DSN
               valueFrom:
                 secretKeyRef:
                   name: common-secrets
                   key: provenance-sentry-dsn
                   # 'name' secret should exist & include key
                   # if the setting doesn't exist, sentry pushes will be disabled
                   optional: true
             - name: SWH_SENTRY_DISABLE_LOGGING_EVENTS
               value: "true"
           volumeMounts:
           - name: configuration
             mountPath: /etc/swh
           - name: configuration-logging
             mountPath: /etc/swh/logging
+          
       volumes:
       - name: configuration
         emptyDir: {}
       - name: configuration-template
         configMap:
           name: provenance-graph-granet-configuration-template
           items:
           - key: "config.yml.template"
             path: "config.yml.template"
       - name: configuration-logging
         configMap:
           name: provenance-graph-granet-configuration-logging
           items:
           - key: "logging-gunicorn.json"
             path: "logging-gunicorn.json"
+      
       - name: config-utils
         configMap:
           name: config-utils
           defaultMode: 0555
+      - name: backend-utils
+        configMap:
+          name: backend-utils
+          defaultMode: 0555
 ---
 # Source: swh/templates/scheduler/extra-services-deployment.yaml
 apiVersion: apps/v1
 kind: Deployment
 metadata:
   namespace: swh-cassandra
   name: scheduler-listener
   labels:
     app: scheduler-listener
 spec:
@@ -27252,21 +27370,21 @@
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: search-rpc
       annotations:
         checksum/config: a76abbb3247f00560f78f1f1aaafc30e0c3958dc059e75911400596ddb51b4e2
         checksum/config-logging: 7bffbc6ce2cb11d88208ef0c5f1d8e6822659c361717afb51dcf0f4da02fe1f7
         checksum/config-utils: 13a26f6add17e96ce01550153c77dcd48de60241a3f4db3c93d5467234be2a7f
-        checksum/backend-utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/backend-utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/rpc
                 operator: In
                 values:
                 - "true"
@@ -27440,21 +27558,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-cassandra-azure-readonly
       annotations:
         checksum/config: 63fbb2e5c758f9faab28192d2a0458eea22410b824e63d0b35de085b50fc3e6e
         checksum/config-logging: 6d3a84a071464bdb72aea996f9c90be8ff89eedb1f09a8cc71c7699e652c8a47
-        checksum/backend-utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/backend-utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
         checksum/config-utils: 13a26f6add17e96ce01550153c77dcd48de60241a3f4db3c93d5467234be2a7f
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/storage
                 operator: In
                 values:
@@ -27693,21 +27811,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-cassandra-readonly
       annotations:
         checksum/config: 7a7e39b703ff92c12c87d9b4d8f0ea91c4d79e96c78f05476f37ab783c4687ff
         checksum/config-logging: 800fc3f5bdfec12955f3689d3c319b74f52a02d09b08fb710cea854d815dfad6
-        checksum/backend-utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/backend-utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
         checksum/config-utils: 13a26f6add17e96ce01550153c77dcd48de60241a3f4db3c93d5467234be2a7f
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/storage
                 operator: In
                 values:
@@ -27946,21 +28064,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-cassandra-readonly-internal
       annotations:
         checksum/config: 0094e08338f11eeeda7a0958ec8402ab9d37544aecc81f62772fddad37c38dfe
         checksum/config-logging: 2ecb5a0cb1eaeb9246bc272bee5df3292a73f3c87134840a682f8c3fb03ac008
-        checksum/backend-utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/backend-utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
         checksum/config-utils: 13a26f6add17e96ce01550153c77dcd48de60241a3f4db3c93d5467234be2a7f
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/storage
                 operator: In
                 values:
@@ -28199,21 +28317,21 @@
     type: RollingUpdate
     rollingUpdate:
       maxSurge: 1
   template:
     metadata:
       labels:
         app: storage-cassandra-winery
       annotations:
         checksum/config: 17b0f2faa5762626ef5c966dc8a2aa810c208c8c86643305a8921e538ea21583
         checksum/config-logging: 21f19120491561669a337c91f8a5b62fb9b081d0c0ca55a9e69fdc26e1a5350a
-        checksum/backend-utils: 498bb7b35f4e2d6996251c8615bff7661ffa9981f2f69734acb80824fc37d2b1
+        checksum/backend-utils: bfb79c03c5f47eacbf1cedde17860a255223193cdeb60f9e3455084af3bab88c
         checksum/config-utils: 13a26f6add17e96ce01550153c77dcd48de60241a3f4db3c93d5467234be2a7f
     spec:
       affinity:
         nodeAffinity:
           requiredDuringSchedulingIgnoredDuringExecution:
             nodeSelectorTerms:
             - matchExpressions:
               - key: swh/storage
                 operator: In
                 values:

Refs. swh/infra/sysadm-environment#5608 (closed)

Edited 2 weeks ago by Antoine R. Dumont

Merge request reports

0 Assignees

0 Reviewers

Request review from

Labels

None

Select labels

Manage project labels

Milestone

None

Time tracking

No estimate or time spent

0 Participants

Compare