Deploy swh-alter 1.3.0 with kafka notification pipeline
Since swh-alter version 1.3.0, the tool can send notifications of alterations made to the archive to a dedicated kafka topic that can then be used by mirrors to take action when these events occur.
This deployment probably means:
- create the proper kafka topic with dedicated permissions (should be read-accessible only by mirror accounts and writable by the account linked to the swh alter commands).
- ensure the environment in which swh alter commands are executed is properly configured.
The docker integration test for the mirror stack can be used as a working example of such a deployment (minus ACLs of the topic).
In this docker env, the topic prefix dedicated to these notifications is swh.journal.mirror-notifications (can be something else if it makes more sense or help operational purpose) and the topic itself is removal_notification (currently not configurable).
Thank you
Activity
assigned to @guillaume
mentioned in commit swh/infra/puppet/puppet-swh-site@ad7cff60
mentioned in merge request swh/infra/puppet/puppet-swh-site!731 (merged)
mentioned in commit swh/infra/ci-cd/swh-charts@1de7ece2
mentioned in commit swh/infra/ci-cd/swh-charts@7d2340b7
mentioned in merge request swh/infra/ci-cd/swh-charts!558 (merged)
mentioned in commit swh/infra/puppet/puppet-swh-site@6d4176da
mentioned in commit swh/infra/puppet/puppet-swh-site@3e69fc45
Creating the
swh.journal.mirror-notifications.removal_notificationKafka topic in staging:root@kafka1:~# /opt/kafka/bin/kafka-topics.sh --bootstrap-server "$SERVER" --create \ --config cleanup.policy=compact \ --partitions 1 \ --replication-factor 2 \ --topic swh.journal.mirror-notifications.removal_notification WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both. Created topic swh.journal.mirror-notifications.removal_notification.Updating the mirror existing user permissions:
root@getty:/usr/local/sbin# ./create_kafka_users_rocquencourt_staging.sh --mirror enea-stg-mirror-01 Creating user enea-stg-mirror-01, with privileged access to consumer group prefix enea-stg-mirror-01- Password for user enea-stg-mirror-01: Setting user credentials Warning: --zookeeper is deprecated and will be removed in a future version of Kafka. Use --bootstrap-server instead to specify a broker to connect to. Completed updating config for entity: user-principal 'enea-stg-mirror-01'. Granting access to topics swh.journal.objects. to enea-stg-mirror-01 [...] Granting access to topics swh.journal.mirror-notifications. to enea-stg-mirror-01 Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:enea-stg-mirror-01, host=*, operation=READ, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:enea-stg-mirror-01, host=*, operation=READ, permissionType=ALLOW) Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:enea-stg-mirror-01, host=*, operation=DESCRIBE, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:enea-stg-mirror-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:enea-stg-mirror-01, host=*, operation=DESCRIBE, permissionType=ALLOW) [...]root@getty:/usr/local/sbin# ./create_kafka_users_rocquencourt_staging.sh --mirror snyk-stg-01 Creating user snyk-stg-01, with privileged access to consumer group prefix snyk-stg-01- Password for user snyk-stg-01: Setting user credentials Warning: --zookeeper is deprecated and will be removed in a future version of Kafka. Use --bootstrap-server instead to specify a broker to connect to. Completed updating config for entity: user-principal 'snyk-stg-01'. Granting access to topics swh.journal.objects. to snyk-stg-01 [...] Granting access to topics swh.journal.mirror-notifications. to snyk-stg-01 Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:snyk-stg-01, host=*, operation=READ, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:snyk-stg-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:enea-stg-mirror-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:enea-stg-mirror-01, host=*, operation=DESCRIBE, permissionType=ALLOW) Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:snyk-stg-01, host=*, operation=DESCRIBE, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:enea-stg-mirror-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:snyk-stg-01, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:snyk-stg-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:enea-stg-mirror-01, host=*, operation=DESCRIBE, permissionType=ALLOW) [...]mentioned in commit swh/infra/ci-cd/swh-charts@43fba37a
mentioned in commit swh/infra/ci-cd/swh-charts@dc5d1f09
Deployed in staging:
ᐅ kbs get po -n swh-cassandra -l app=alter NAME READY STATUS RESTARTS AGE alter-85c5bc87f-tlmmk 1/1 Running 0 5m32sᐅ kbs exec -ti -n swh-cassandra -c alter \ $(kbs get po -n swh-cassandra -l app=alter -o name) -- \ cat /etc/swh/config.yml | yq .journal_writer anonymize: true brokers: - journal2.internal.staging.swh.network:9092 - kafka3.internal.staging.swh.network:9092 client_id: swh.alter.removals.alter-85c5bc87f-tlmmk cls: kafka prefix: swh.journal.mirror-notifications producer_config: message.max.bytes: 1000000000Test in staging:
- ingest an origin with SCN:
ᐅ stern --context archive-staging-rke2 -n swh-cassandra \ -l app=loader-save-code-now --since 10m --only-log-lines --no-follow \ -e "kube-probe/1.28" | awk 'END {$1=$2="";print}' | jq { "asctime": "2025-03-24 12:35:51,888", "threadName": "MainThread", "pathname": "/opt/swh/venv/lib/python3.11/site-packages/celery/app/trace.py", "lineno": 128, "funcName": "info", "task_name": null, "task_id": null, "name": "celery.app.trace", "levelname": "INFO", "message": "Task swh.loader.git.tasks.UpdateGitRepository[d6b41d24-d39f-4170-be5a-eab021a81b8b] succeeded in 4.682983160018921s: {'status': 'eventful'}", "data": { "id": "d6b41d24-d39f-4170-be5a-eab021a81b8b", "name": "swh.loader.git.tasks.UpdateGitRepository", "return_value": "{'status': 'eventful'}", "runtime": 4.682983160018921, "args": "()", "kwargs": "{'url': 'https://git.communityrack.org/infra/knotci-docker.git'}" }- remove ingested origin with alter:
swh@alter-85c5bc87f-tlmmk:~$ cd /srv/recovery-bundles/ swh@alter-85c5bc87f-tlmmk:/srv/recovery-bundles$ IDENTIFIER=$(date +%F|sed 's/-//g')-communityrack swh@alter-85c5bc87f-tlmmk:/srv/recovery-bundles$ cat > $IDENTIFIER.origins https://git.communityrack.org/infra/knotci-docker.git ^C swh@alter-85c5bc87f-tlmmk:/srv/recovery-bundles$ cat $IDENTIFIER.origins https://git.communityrack.org/infra/knotci-docker.gitswh@alter-85c5bc87f-tlmmk:/srv/recovery-bundles$ swh alter remove --identifier $IDENTIFIER \ --recovery-bundle /srv/recovery-bundles/$IDENTIFIER.zip \ --reason 'Test kafka mirrors notification topic' $(cat $IDENTIFIER.origins) WARNING:cassandra.cluster:Downgrading core protocol version from 66 to 65 for 192.168.130.181:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version WARNING:cassandra.cluster:Downgrading core protocol version from 65 to 5 for 192.168.130.181:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version INFO:cassandra.policies:Using datacenter 'sesi_rocquencourt_staging' for DCAwareRoundRobinPolicy (via host '192.168.130.181:9042'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes WARNING:cassandra.cluster:Downgrading core protocol version from 66 to 65 for 192.168.130.181:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version WARNING:cassandra.cluster:Downgrading core protocol version from 65 to 5 for 192.168.130.181:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version INFO:cassandra.policies:Using datacenter 'sesi_rocquencourt_staging' for DCAwareRoundRobinPolicy (via host '192.168.130.181:9042'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes Removing the following origins: - swh:1:ori:6fe8619df111539cbe0a5a1760b758a1542e87ca Finding removable objects… Inventorying all reachable objects… [------------------------------------] done (93 objects found / 0 left to look up) Determining which objects can be safely removed… [####################################] 100% Finding RawExtrinsicMetadata objects… [####################################] 100% Removal plan: - Origin: 1 - Snapshot: 1 - Release: 5 - Revision: 29 - Directory: 25 - Content: 32 - … and more objects that are not addresseable by a SWHID (OriginVisit, OriginVisitStatus, ExtID). Proceed? [y/N]: y Creating recovery bundle… Backing up Content objects… [####################################] 100% Backing up Directory objects… [####################################] 100% Backing up Revision objects… [####################################] 100% Backing up Release objects… [####################################] 100% Backing up Snapshot objects… [####################################] 100% Backing up Origin objects… [####################################] 100% Recovery bundle created. Recovery bundle decryption key: xxxxxx Removing origins from search “search”… [####################################] 100% 1 origins removed from search “search”. Removing objects from storage “cassandra”… [####################################] 100% 96 objects removed from storage “cassandra”. Removing objects from storage “postgresql”… [####################################] 100% 96 objects removed from storage “postgresql”. Removing objects from journal “journal”… [####################################] 100% Objects removed from journal “journal”. Removing objects from objstorage “db1”… [####################################] 100% 32 objects removed from objstorage “db1”. Total time: 264.61 milliseconds, average: 8.27 milliseconds per object, standard deviation: 10.46 milliseconds Removing objects from objstorage “storage1”… [####################################] 100% 0 objects removed from objstorage “storage1”. Looking for newly added references… [####################################] 100% Sending removal notification… Removal notification sent.swh@alter-85c5bc87f-tlmmk:/srv/recovery-bundles$ cat > $IDENTIFIER.key swh@alter-85c5bc87f-tlmmk:/srv/recovery-bundles$ ls -trl $IDENTIFIER* -rw-r--r-- 1 swh swh 178561 Mar 24 13:05 20250324-communityrack.zip -rw-r--r-- 1 swh swh 75 Mar 24 13:06 20250324-communityrack.key- check
swh.journal.mirror-notifications.removal_notificationKafka topic:
root@kafka1:~# /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server $SERVER --topic swh.journal.mirror-notifications.removal_notification --from-beginning removal_identifier20250324-communityrackreason%Test kafka mirrors notification topicrequested2swh:1:ori:6fe8619df111539cbe0a5a1760b758a1542e87caremoved_objects]2swh:1:cnt:0b21e041c34a66c637d3e0ba1d786c85a3f365a02swh:1:cnt:0d69efae1dbab3118143bcb830944e02393a8ac32swh:1:cnt:12833f14966d101004c0181dd74e79cd50220fea2swh:1:cnt:22166396aef538511135c74532d228eaa2f270242swh:1:cnt:2c9927b9d36b68b7dad43ea8221d36cd1527f2952swh:1:cnt:3052ddb1dee640f7f1aedf87daa5ab6ab9a026bf2swh:1:cnt:38fb1c1038c1f85fcb1ac5d16ce18b06616e58e42swh:1:cnt:3ce424c2718300d3730fedb6b31e0a0ac8162cdb2swh:1:cnt:47f4ceb40e381d93afb2194b3fa97ddb329ac2282swh:1:cnt:4cfc3ce3fdcd96956fc29cc4b8977eb2c79bf6a92swh:1:cnt:4f5ee387e5d84c4869863a913ac84625b52869502swh:1:cnt:559f4cfeacc8f75dcd17ccfb64e96ee7b5141b592swh:1:cnt:5ace870f5b5bf3e4473bdeb7d1fb0b903faed9772swh:1:cnt:5ba60a1c1b9ab2f7d00de69958d566e41ff70e802swh:1:cnt:5d1c4b96929a5946dbe4025a057814e94693e7432swh:1:cnt:62b6a06c9eb610ef8a942dd647f13f2d42815ba12swh:1:cnt:6e92e66ead4bca12c4f9e225abc9cdc881f41f042swh:1:cnt:7771aa11f9a01aa24051ec87ea8dba2f05e1da5c2swh:1:cnt:851d9891d555c1f3ecc18eb3a2e04491e8ef496b2swh:1:cnt:8e41e3a25cd6c6a092e0cf63f0c6095b50187e162swh:1:cnt:924d5863002a7b3b12411a0cea2fb510157725282swh:1:cnt:a1b874068bd2b2397253330dbbf605eeb31e03042swh:1:cnt:a734bdc7182db5b0300037769976f629fcceb7682swh:1:cnt:af69dd0ff4e12b3eabaefa74724a1effbd2800ff2swh:1:cnt:b0119d6111faa3acf7ca51edcb6ada7bfd7f6b8f2swh:1:cnt:bb95992207fe9b76974153371f6431c70929e3f62swh:1:cnt:ce8f542cf3d02fd808b003997b1005b26feeecec2swh:1:cnt:d7e1ded9ef0a6df68b07b07dc4ce7ec1c89e2e0d2swh:1:cnt:db89ba1af6b469ed8f50bd2effb912a7f8004dee2swh:1:cnt:deab63e5d95f5dfcdfaa291293ee6eb9511f79cf2swh:1:cnt:eeb0a86400933b7291aecdfaa22a73ce1f79c4802swh:1:cnt:f5247681efbe1bd8f1ac1e03ad80a3ea799d3fbf2swh:1:dir:05c949dfcbb6b843617b5e65fecb2d78b0983e5d2swh:1:dir:093bf7b805ad1e84f9b83a346ebf987dd59461a32swh:1:dir:126edcfdc24e704110479a6785f905a00df924962swh:1:dir:17d8aafc0281203386ee332317d5b7db483063422swh:1:dir:31b974ec76088a90dcd9eb7afa3f873de0cd31fa2swh:1:dir:37c156c6b618ed017dcf63a324171fd6fcead5832swh:1:dir:432929d682b227eba10b94cf7750d317a0a224832swh:1:dir:4b9f7506442b5d39602b535b9df93bc0a4d126af2swh:1:dir:586f01d7b234a4139b7643c97ca5241bb6c871b42swh:1:dir:661af8e2284d19e890fdc7b4ae027a4d663e49c32swh:1:dir:6a752c1271643ba91a4a44f38698094123f52ec62swh:1:dir:767e3b13db6fad8ed0a0762bb20d726b2156cc6e2swh:1:dir:769d2f59d0f7328f221f919c5c04756c5050016a2swh:1:dir:7eb7bea57f9a59fb708859dfc1398e5fd4dca4192swh:1:dir:94bf9eae837086c136964b83933856f031ab9fcb2swh:1:dir:9a3b9243b35290e11b74d568115c81103bb62faa2swh:1:dir:ab17ac3c93b4bb41cf582bb5ed691edb39dc0f4d2swh:1:dir:ac73a39afae9ef61ef36cacbb8dbf03ca8a06dc62swh:1:dir:bc8ee02a534c0ad460b48791d129e6c4027e23032swh:1:dir:c84b385f26d2442eae6ac9bb94a2455ed6ea8c642swh:1:dir:cbcc94019116486d9798a3953b8a5bf71d1746d82swh:1:dir:ccd5f4ee4f42ddc52fc837f0c1b439d73c0a056f2swh:1:dir:d03d6bc9cd6d6e67f839edd3d4be5045b14f28332swh:1:dir:d9efac2cf7deef0d301452e19ac39f4d8665bf2c2swh:1:dir:fc738998fbd2e90b388065d09236c9c2a4fb53ec2swh:1:rev:17057789940b9e8d13186b22eb7386e89c4faaf42swh:1:rev:1940a6c27489d8b226fa67a7877908537d984ae62swh:1:rev:1e3f87d8397f1ed3cbf5f32f0e71cefa01a1c5142swh:1:rev:24c0e65c055ff9c769c646d50917e817cecc49532swh:1:rev:378a148df5f9f0d70ed7c1f5ceee68bbc9f33b942swh:1:rev:39f0f2770048e4b6d80ccf831250e8cb54f1649d2swh:1:rev:41d5582f445de79ffeb9a7ccba055e263d5b6adc2swh:1:rev:43c3a23bebfe81a1f57501e11ac3e198ede736e32swh:1:rev:43ff594de10ae5486a0034e597cd0fc23c7331472swh:1:rev:45ecb8aa91643915aa6522f2747097838ab68d3c2swh:1:rev:580f2fe0d072bc58c3d145acf6417e5a7ef2a6512swh:1:rev:60730c8e66d013feb91b8f65874c74c2868867132swh:1:rev:6546ebd4bbdc5d4c82c877b4ece8039d855e216f2swh:1:rev:8f72bbb5485619b57ab3715e5069077d134e1d432swh:1:rev:9d777616c4e1d0bb0b480ac9a2aadc79bbe432702swh:1:rev:a2ae78de737db45444d20e10a828450cc07c7dc32swh:1:rev:b88cf2405e24705c07a4c2a99d0dad05cadfa5e72swh:1:rev:c083c79ba5c51fd5891d97dfc22a3afcaedb3fb42swh:1:rev:c2c85d1ef3de788143b1eac5ad8afccdecdba57c2swh:1:rev:c4df6ddc12893dd0f1475897b3570b59fd2a47482swh:1:rev:cadac890d8003f31ac6cb8fe31c3ba8f814d75382swh:1:rev:cbeef735eee360012dd42858b7ef034d6e7c0e192swh:1:rev:e1c1b06f9126b6d5ec8bcfba7c44ca6129a629f52swh:1:rev:e8fff33f86e75c58b4e2638e7df75d703a7e7a052swh:1:rev:e9a80082ed2ccb13a38a0ac2782f1671279a79442swh:1:rev:ede49f166beb68d594223f3a58a6a1da0f02d7c12swh:1:rev:eef4deaf681000d47e3ff6c7250effac0f0b9a042swh:1:rev:fa4b4b18eabe71033b6f32ef9fc4676d2fdf878b2swh:1:rev:fa7bbbc94122a38763110df5ad0246a9634a7a9b2swh:1:rel:110b522eea1a7ff9747a6c62f9e88e22d90455202swh:1:rel:6e286527a011e81606580b9cfdb9766f7a0a16812swh:1:rel:898af8f1519d46e1aab10831ed12b463b919667b2swh:1:rel:92a3f533ae307d5f4fea80e7fca71148a274d0f62swh:1:rel:a7b4b650bb3849d731d2d3d6dc236cd1064ae3572swh:1:snp:052bbdc7d9daf4587b55c8915fd6e0e2310f7eea2swh:1:ori:6fe8619df111539cbe0a5a1760b758a1542e87caEverything seems fine, I will prepare the deployment in production.
mentioned in commit swh/infra/ci-cd/swh-charts@eb1c7cb3
mentioned in merge request swh/infra/ci-cd/swh-charts!562 (merged)
Creating the
swh.journal.mirror-notifications.removal_notificationKafka topic in production:root@kafka1:~# /opt/kafka/bin/kafka-topics.sh --bootstrap-server "$SERVER" --create \ --config cleanup.policy=compact \ --partitions 1 \ --replication-factor 2 \ --topic swh.journal.mirror-notifications.removal_notification WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both. Created topic swh.journal.mirror-notifications.removal_notification.Updating the mirror existing users permissions (
enea-prod-01,grnet-prod-01,snyk-prod-01,unidue-prod-01):root@getty:/usr/local/sbin# ./create_kafka_users_rocquencourt.sh --mirror enea-prod-01 Creating user enea-prod-01, with privileged access to consumer group prefix enea-prod-01- Password for user enea-prod-01: Setting user credentials [...] Granting access to topics swh.journal.mirror-notifications. to enea-prod-01 Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:enea-prod-01, host=*, operation=READ, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:enea-prod-01, host=*, operation=READ, permissionType=ALLOW) Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:enea-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:enea-prod-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:enea-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) Granting access to consumer group prefix enea-prod-01- to enea-prod-01 Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=enea-prod-01-, patternType=PREFIXED)`: (principal=User:enea-prod-01, host=*, operation=READ, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=enea-prod-01-, patternType=PREFIXED)`: (principal=User:enea-prod-01, host=*, operation=READ, permissionType=ALLOW) Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=enea-prod-01-, patternType=PREFIXED)`: (principal=User:enea-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=enea-prod-01-, patternType=PREFIXED)`: (principal=User:enea-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:enea-prod-01, host=*, operation=READ, permissionType=ALLOW) Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=enea-prod-01-, patternType=PREFIXED)`: (principal=User:enea-prod-01, host=*, operation=DELETE, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=enea-prod-01-, patternType=PREFIXED)`: (principal=User:enea-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:enea-prod-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:enea-prod-01, host=*, operation=DELETE, permissionType=ALLOW)root@getty:/usr/local/sbin# ./create_kafka_users_rocquencourt.sh --mirror grnet-prod-01 Creating user grnet-prod-01, with privileged access to consumer group prefix grnet-prod-01- Password for user grnet-prod-01: Setting user credentials [...] Granting access to topics swh.journal.mirror-notifications. to grnet-prod-01 Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:grnet-prod-01, host=*, operation=READ, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:enea-prod-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:enea-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:grnet-prod-01, host=*, operation=READ, permissionType=ALLOW) Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:grnet-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:grnet-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:enea-prod-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:enea-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:grnet-prod-01, host=*, operation=READ, permissionType=ALLOW) Granting access to consumer group prefix grnet-prod-01- to grnet-prod-01 Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=grnet-prod-01-, patternType=PREFIXED)`: (principal=User:grnet-prod-01, host=*, operation=READ, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=grnet-prod-01-, patternType=PREFIXED)`: (principal=User:grnet-prod-01, host=*, operation=DELETE, permissionType=ALLOW) (principal=User:grnet-prod-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:grnet-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=grnet-prod-01-, patternType=PREFIXED)`: (principal=User:grnet-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=grnet-prod-01-, patternType=PREFIXED)`: (principal=User:grnet-prod-01, host=*, operation=DELETE, permissionType=ALLOW) (principal=User:grnet-prod-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:grnet-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=grnet-prod-01-, patternType=PREFIXED)`: (principal=User:grnet-prod-01, host=*, operation=DELETE, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=grnet-prod-01-, patternType=PREFIXED)`: (principal=User:grnet-prod-01, host=*, operation=DELETE, permissionType=ALLOW) (principal=User:grnet-prod-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:grnet-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW)root@getty:/usr/local/sbin# ./create_kafka_users_rocquencourt.sh --mirror snyk-prod-01 Creating user snyk-prod-01, with privileged access to consumer group prefix snyk-prod-01- Password for user snyk-prod-01: Setting user credentials [...] Granting access to topics swh.journal.mirror-notifications. to snyk-prod-01 Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:snyk-prod-01, host=*, operation=READ, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:enea-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:enea-prod-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:grnet-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:grnet-prod-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:snyk-prod-01, host=*, operation=READ, permissionType=ALLOW) Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:snyk-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:enea-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:snyk-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:enea-prod-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:grnet-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:grnet-prod-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:snyk-prod-01, host=*, operation=READ, permissionType=ALLOW) Granting access to consumer group prefix snyk-prod-01- to snyk-prod-01 Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=snyk-prod-01-, patternType=PREFIXED)`: (principal=User:snyk-prod-01, host=*, operation=READ, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=snyk-prod-01-, patternType=PREFIXED)`: (principal=User:snyk-prod-01, host=*, operation=READ, permissionType=ALLOW) Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=snyk-prod-01-, patternType=PREFIXED)`: (principal=User:snyk-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=snyk-prod-01-, patternType=PREFIXED)`: (principal=User:snyk-prod-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:snyk-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=snyk-prod-01-, patternType=PREFIXED)`: (principal=User:snyk-prod-01, host=*, operation=DELETE, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=snyk-prod-01-, patternType=PREFIXED)`: (principal=User:snyk-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:snyk-prod-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:snyk-prod-01, host=*, operation=DELETE, permissionType=ALLOW)root@getty:/usr/local/sbin# ./create_kafka_users_rocquencourt.sh --mirror unidue-prod-01 Creating user unidue-prod-01, with privileged access to consumer group prefix unidue-prod-01- Password for user unidue-prod-01: Setting user credentials [...] Granting access to topics swh.journal.mirror-notifications. to unidue-prod-01 Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:unidue-prod-01, host=*, operation=READ, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:unidue-prod-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:enea-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:snyk-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:enea-prod-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:grnet-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:grnet-prod-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:snyk-prod-01, host=*, operation=READ, permissionType=ALLOW) Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:unidue-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.mirror-notifications., patternType=PREFIXED)`: (principal=User:unidue-prod-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:enea-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:snyk-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:enea-prod-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:grnet-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:unidue-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:grnet-prod-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:snyk-prod-01, host=*, operation=READ, permissionType=ALLOW) Granting access to consumer group prefix unidue-prod-01- to unidue-prod-01 Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=unidue-prod-01-, patternType=PREFIXED)`: (principal=User:unidue-prod-01, host=*, operation=READ, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=unidue-prod-01-, patternType=PREFIXED)`: (principal=User:unidue-prod-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:unidue-prod-01, host=*, operation=DELETE, permissionType=ALLOW) (principal=User:unidue-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=unidue-prod-01-, patternType=PREFIXED)`: (principal=User:unidue-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=unidue-prod-01-, patternType=PREFIXED)`: (principal=User:unidue-prod-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:unidue-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:unidue-prod-01, host=*, operation=DELETE, permissionType=ALLOW) Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=unidue-prod-01-, patternType=PREFIXED)`: (principal=User:unidue-prod-01, host=*, operation=DELETE, permissionType=ALLOW) Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=unidue-prod-01-, patternType=PREFIXED)`: (principal=User:unidue-prod-01, host=*, operation=READ, permissionType=ALLOW) (principal=User:unidue-prod-01, host=*, operation=DESCRIBE, permissionType=ALLOW) (principal=User:unidue-prod-01, host=*, operation=DELETE, permissionType=ALLOW)Deployed in production:
~ ᐅ kbp get deploy,po -n swh-cassandra -l app=alter NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/alter 1/1 1 1 104d NAME READY STATUS RESTARTS AGE pod/alter-5c8d7bddd6-v2zgp 1/1 Running 0 115s~ ᐅ kbp exec -ti -n swh-cassandra -c alter \ $(kbp get po -n swh-cassandra -l app=alter -o name) -- \ cat /etc/swh/config.yml | yq .journal_writer anonymize: true brokers: - kafka1.internal.softwareheritage.org:9092 - kafka2.internal.softwareheritage.org:9092 - kafka3.internal.softwareheritage.org:9092 - kafka4.internal.softwareheritage.org:9092 client_id: swh.alter.removals.alter-5c8d7bddd6-v2zgp cls: kafka prefix: swh.journal.mirror-notifications producer_config: message.max.bytes: 1000000000mentioned in issue swh/devel/swh-alter#24
Please register or sign in to reply