Deploy the storage scrubbers in elastic staging
In order to upgrade and reactivate the storage scubbers in the elastic infra
- Create the helm charts to deploy them
- Upgrade the toolbox configuration to be able to register the checkers
- Deploy in staging
- Add the scrubber configurations [1]
- Decommission scrubber0.internal.staging.swh.network
[1] Declare the scrubber configurations:
- for
swh
# in 2 loops to group the ids
for type in directory snapshot release revision
do
swh scrubber check init storage --object-type=$type --name storage-references-$type --no-check-hashes
done
for type in directory snapshot release revision
do
swh scrubber check init storage --object-type=$type --name storage-hashes-$type --no-check-references
done- for
swh-cassandra
# in 2 loops to group the ids
for type in directory snapshot release revision
do
swh scrubber check init storage --object-type=$type --name storage-cassandra-references-$type --no-check-hashes
done
for type in directory snapshot release revision
do
swh scrubber check init storage --object-type=$type --name storage-cassandra-hashes-$type --no-check-references
doneActivity
changed milestone to %Regularly scrub journal, storage, and objstorage [Roadmap - Preserve]
added activity::Deployment label
assigned to @vsellier
mentioned in commit swh/infra/ci-cd/swh-charts@b01c3044
mentioned in merge request swh/infra/ci-cd/swh-charts!211 (merged)
mentioned in commit swh/infra/ci-cd/swh-charts@a5dcb7a9
marked this issue as related to swh/devel/swh-scrubber#4699
marked this issue as related to swh/devel/swh-scrubber#4700
mentioned in commit swh/infra/ci-cd/swh-charts@6f316a6c
mentioned in commit swh/infra/ci-cd/swh-charts@b04abd21
mentioned in commit swh/infra/ci-cd/swh-charts@469383b6
mentioned in commit swh/infra/ci-cd/swh-charts@804149a5
mentioned in merge request swh/infra/ci-cd/swh-charts!214 (merged)
mentioned in commit swh/infra/ci-cd/swh-charts@5d17301c
mentioned in commit swh/infra/ci-cd/swh-charts@30767b05
Storage scrubbers deployed and configured:
- postgresql
swh@swh-toolbox-56c7bb65bd-cpgkh:~$ export SWH_CONFIG_FILENAME=/etc/swh/config-scrubber-storage.yml swh@swh-toolbox-56c7bb65bd-cpgkh:~$ for type in directory snapshot release revision do swh scrubber check init storage --object-type=$type --name storage-references-$type --no-check-hashes done Created configuration storage-references-directory [5] for checking directory in postgresql storage Created configuration storage-references-snapshot [6] for checking snapshot in postgresql storage Created configuration storage-references-release [7] for checking release in postgresql storage Created configuration storage-references-revision [8] for checking revision in postgresql storage swh@swh-toolbox-56c7bb65bd-cpgkh:~$ for type in directory snapshot release revision do swh scrubber check init storage --object-type=$type --name storage-hashes-$type --no-check-references done Created configuration storage-hashes-directory [9] for checking directory in postgresql storage Created configuration storage-hashes-snapshot [10] for checking snapshot in postgresql storage Created configuration storage-hashes-release [11] for checking release in postgresql storage Created configuration storage-hashes-revision [12] for checking revision in postgresql storage- cassandra
swh@swh-toolbox-54c8bbc7f9-rjkhw:~$ export SWH_CONFIG_FILENAME=/etc/swh/config-scrubber-storage.yml swh@swh-toolbox-54c8bbc7f9-rjkhw:~$ for type in directory snapshot release revision do swh scrubber check init storage --object-type=$type --name storage-cassandra-references-$type --no-check-hashes done WARNING:cassandra.cluster:Downgrading core protocol version from 66 to 65 for 192.168.130.183:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version WARNING:cassandra.cluster:Downgrading core protocol version from 65 to 5 for 192.168.130.183:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version INFO:cassandra.policies:Using datacenter 'sesi_rocquencourt_staging' for DCAwareRoundRobinPolicy (via host '192.168.130.183:9042'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes Created configuration storage-cassandra-references-directory [13] for checking directory in cassandra storage WARNING:cassandra.cluster:Downgrading core protocol version from 66 to 65 for 192.168.130.181:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version WARNING:cassandra.cluster:Downgrading core protocol version from 65 to 5 for 192.168.130.181:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version INFO:cassandra.policies:Using datacenter 'sesi_rocquencourt_staging' for DCAwareRoundRobinPolicy (via host '192.168.130.181:9042'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes Created configuration storage-cassandra-references-snapshot [14] for checking snapshot in cassandra storage WARNING:cassandra.cluster:Downgrading core protocol version from 66 to 65 for 192.168.130.182:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version WARNING:cassandra.cluster:Downgrading core protocol version from 65 to 5 for 192.168.130.182:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version INFO:cassandra.policies:Using datacenter 'sesi_rocquencourt_staging' for DCAwareRoundRobinPolicy (via host '192.168.130.182:9042'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes Created configuration storage-cassandra-references-release [15] for checking release in cassandra storage WARNING:cassandra.cluster:Downgrading core protocol version from 66 to 65 for 192.168.130.182:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version WARNING:cassandra.cluster:Downgrading core protocol version from 65 to 5 for 192.168.130.182:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version INFO:cassandra.policies:Using datacenter 'sesi_rocquencourt_staging' for DCAwareRoundRobinPolicy (via host '192.168.130.182:9042'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes Created configuration storage-cassandra-references-revision [16] for checking revision in cassandra storage swh@swh-toolbox-54c8bbc7f9-rjkhw:~$ for type in directory snapshot release revision do swh scrubber check init storage --object-type=$type --name storage-cassandra-hashes-$type --no-check-references done WARNING:cassandra.cluster:Downgrading core protocol version from 66 to 65 for 192.168.130.181:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version WARNING:cassandra.cluster:Downgrading core protocol version from 65 to 5 for 192.168.130.181:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version INFO:cassandra.policies:Using datacenter 'sesi_rocquencourt_staging' for DCAwareRoundRobinPolicy (via host '192.168.130.181:9042'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes Created configuration storage-cassandra-hashes-directory [17] for checking directory in cassandra storage WARNING:cassandra.cluster:Downgrading core protocol version from 66 to 65 for 192.168.130.181:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.g ithub.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version WARNING:cassandra.cluster:Downgrading core protocol version from 65 to 5 for 192.168.130.181:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version INFO:cassandra.policies:Using datacenter 'sesi_rocquencourt_staging' for DCAwareRoundRobinPolicy (via host '192.168.130.181:9042'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes Created configuration storage-cassandra-hashes-snapshot [18] for checking snapshot in cassandra storage WARNING:cassandra.cluster:Downgrading core protocol version from 66 to 65 for 192.168.130.183:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.g ithub.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version WARNING:cassandra.cluster:Downgrading core protocol version from 65 to 5 for 192.168.130.183:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version INFO:cassandra.policies:Using datacenter 'sesi_rocquencourt_staging' for DCAwareRoundRobinPolicy (via host '192.168.130.183:9042'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes Created configuration storage-cassandra-hashes-release [19] for checking release in cassandra storage WARNING:cassandra.cluster:Downgrading core protocol version from 66 to 65 for 192.168.130.182:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version WARNING:cassandra.cluster:Downgrading core protocol version from 65 to 5 for 192.168.130.182:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version INFO:cassandra.policies:Using datacenter 'sesi_rocquencourt_staging' for DCAwareRoundRobinPolicy (via host '192.168.130.182:9042'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes Created configuration storage-cassandra-hashes-revision [20] for checking revision in cassandra storageswh-scrubber=> select * from check_config; id | datastore | object_type | nb_partitions | check_hashes | check_references | name | comment ----+-----------+-------------+---------------+--------------+------------------+----------------------------------------+--------- 1 | 1 | directory | 4096 | t | f | journal-checker-directory | 2 | 1 | snapshot | 4096 | t | f | journal-checker-snapshot | 3 | 1 | release | 4096 | t | f | journal-checker-release | 4 | 1 | revision | 4096 | t | f | journal-checker-revision | 5 | 13 | directory | 4096 | f | t | storage-references-directory | 6 | 13 | snapshot | 4096 | f | t | storage-references-snapshot | 7 | 13 | release | 4096 | f | t | storage-references-release | 8 | 13 | revision | 4096 | f | t | storage-references-revision | 9 | 13 | directory | 4096 | t | f | storage-hashes-directory | 10 | 13 | snapshot | 4096 | t | f | storage-hashes-snapshot | 11 | 13 | release | 4096 | t | f | storage-hashes-release | 12 | 13 | revision | 4096 | t | f | storage-hashes-revision | 13 | 36 | directory | 4096 | f | t | storage-cassandra-references-directory | 14 | 36 | snapshot | 4096 | f | t | storage-cassandra-references-snapshot | 15 | 36 | release | 4096 | f | t | storage-cassandra-references-release | 16 | 36 | revision | 4096 | f | t | storage-cassandra-references-revision | 17 | 36 | directory | 4096 | t | f | storage-cassandra-hashes-directory | 18 | 36 | snapshot | 4096 | t | f | storage-cassandra-hashes-snapshot | 19 | 36 | release | 4096 | t | f | storage-cassandra-hashes-release | 20 | 36 | revision | 4096 | t | f | storage-cassandra-hashes-revision | (20 rows)
Please register or sign in to reply