Skip to content
Snippets Groups Projects
New look: Off

staging: Rename consumer group to swh-archive-stg.loader_metadata.journal_client...

    • Closed Issue created by Antoine R. Dumont @ardumont

    ... without losing the current topics offsets:

    • Stop journal clients
    • Export the offsets of the current consumer group
    • Import them into the new name
    • swh-charts: Rename consumer group loader_metadata.journal_client
    • Then restart the clients with the new name
    • Install tags in grafana
    • Finally clean up the old consumer group

    Current name is: swh-archive-stg.swh.loader_metadata.journal_client

    ✓ 7 of 7 checklist items completed · Edited

    Designs

      Child items ...

      2. Activity

      • Antoine R. Dumont changed milestone to %MRO 2023

        changed milestone to %MRO 2023

      • Antoine R. Dumont mentioned in commit swh/infra/ci-cd/swh-charts@133ecfb3

        mentioned in commit swh/infra/ci-cd/swh-charts@133ecfb3

      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Author Owner

        As a requisite, the journal client should be stopped ^:

        $ OFFSET_FILE=swh-archive-stg-swh.loader_metadata.journal_client.offsets-$(date +%Y%m%d-%H%M).csv
$ /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --reset-offsets --all-topics --to-current --dry-run --export --group $GROUP_ID 2>&1 | tee $OFFSET_FILE
 +%Y%m%d-%H%M).csv
$ cat $OFFSET_FILE
"swh.journal.objects.origin",54,101123
"swh.journal.objects.origin",43,101508
"swh.journal.objects.origin",35,101308
"swh.journal.objects.origin",4,100957
"swh.journal.objects.origin",30,101191
"swh.journal.objects.origin",10,100875
"swh.journal.objects.origin",48,100550
"swh.journal.objects.origin",39,101208
"swh.journal.objects.origin",21,101077
"swh.journal.objects.origin",38,100901
"swh.journal.objects.origin",51,101084
"swh.journal.objects.origin",59,101627
"swh.journal.objects.origin",14,101302
"swh.journal.objects.origin",9,101379
"swh.journal.objects.origin",40,101244
"swh.journal.objects.origin",56,101703
"swh.journal.objects.origin",17,100988
"swh.journal.objects.origin",33,101750
"swh.journal.objects.origin",29,101094
"swh.journal.objects.origin",37,101059
"swh.journal.objects.origin",25,101411
"swh.journal.objects.origin",47,101012
"swh.journal.objects.origin",32,100727
"swh.journal.objects.origin",24,100861
"swh.journal.objects.origin",36,100942
"swh.journal.objects.origin",16,101528
"swh.journal.objects.origin",8,101478
"swh.journal.objects.origin",49,101019
"swh.journal.objects.origin",26,101116
"swh.journal.objects.origin",44,100633
"swh.journal.objects.origin",18,101545
"swh.journal.objects.origin",52,101245
"swh.journal.objects.origin",60,100926
"swh.journal.objects.origin",13,101388
"swh.journal.objects.origin",11,101324
"swh.journal.objects.origin",6,101097
"swh.journal.objects.origin",46,101402
"swh.journal.objects.origin",3,100997
"swh.journal.objects.origin",62,101199
"swh.journal.objects.origin",23,101368
"swh.journal.objects.origin",19,100919
"swh.journal.objects.origin",31,101210
"swh.journal.objects.origin",27,101435
"swh.journal.objects.origin",2,101401
"swh.journal.objects.origin",15,101108
"swh.journal.objects.origin",7,101626
"swh.journal.objects.origin",12,101533
"swh.journal.objects.origin",63,101118
"swh.journal.objects.origin",61,100670
"swh.journal.objects.origin",22,101059
"swh.journal.objects.origin",1,100572
"swh.journal.objects.origin",53,101155
"swh.journal.objects.origin",42,100962
"swh.journal.objects.origin",55,100932
"swh.journal.objects.origin",41,101179
"swh.journal.objects.origin",45,101106
"swh.journal.objects.origin",50,101260
"swh.journal.objects.origin",58,101237
"swh.journal.objects.origin",20,101840
"swh.journal.objects.origin",28,100966
"swh.journal.objects.origin",0,100990
"swh.journal.objects.origin",57,100931
"swh.journal.objects.origin",34,101461
"swh.journal.objects.origin",5,101349
        Edited by Antoine R. Dumont
      • Antoine R. Dumont marked the checklist item Stop journal clients as completed

        marked the checklist item Stop journal clients as completed

      • Antoine R. Dumont marked the checklist item Export the offsets of the current consumer group as completed

        marked the checklist item Export the offsets of the current consumer group as completed

      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Author Owner

        Configure new consumer group with the proper offset:

        root@storage1:~# export GROUP_ID=swh-archive-stg-loader_metadata.journal_client
root@storage1:~# export OFFSET_FILE=$file
root@storage1:~# echo $OFFSET_FILE
swh-archive-stg-swh.loader_metadata.journal_client.offsets-20230405-1238.csv
root@storage1:~# $KAFKA_CONSUMER_GROUPS --bootstrap-server $SERVER \
>   --reset-offsets --all-topics --from-file $OFFSET_FILE \
>   --group $GROUP_ID --execute

GROUP                          TOPIC                          PARTITION  NEW-OFFSET
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     54         101123
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     43         101508
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     35         101308
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     4          100957
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     30         101191
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     10         100875
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     48         100550
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     39         101208
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     21         101077
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     38         100901
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     51         101084
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     59         101627
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     14         101302
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     9          101379
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     40         101244
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     56         101703
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     17         100988
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     33         101750
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     29         101094
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     37         101059
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     25         101411
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     47         101012
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     32         100727
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     24         100861
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     36         100942
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     16         101528
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     8          101478
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     49         101019
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     26         101116
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     44         100633
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     18         101545
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     52         101245
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     60         100926
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     13         101388
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     11         101324
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     6          101097
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     46         101402
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     3          100997
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     62         101199
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     23         101368
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     19         100919
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     31         101210
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     27         101435
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     2          101401
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     15         101108
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     7          101626
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     12         101533
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     63         101118
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     61         100670
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     22         101059
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     1          100572
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     53         101155
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     42         100962
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     55         100932
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     41         101179
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     45         101106
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     50         101260
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     58         101237
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     20         101840
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     28         100966
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     0          100990
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     57         100931
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     34         101461
swh-archive-stg-loader_metadata.journal_client swh.journal.objects.origin     5          101349
      • Antoine R. Dumont marked the checklist item Import them into the new name as completed

        marked the checklist item Import them into the new name as completed

      • Antoine R. Dumont marked the checklist item swh-charts: Rename consumer group loader_metadata.journal_client as completed

        marked the checklist item swh-charts: Rename consumer group loader_metadata.journal_client as completed

      • Antoine R. Dumont mentioned in commit swh/infra/ci-cd/swh-charts@42962e35

        mentioned in commit swh/infra/ci-cd/swh-charts@42962e35

      • Antoine R. Dumont marked the checklist item Then restart the clients with the new name as completed

        marked the checklist item Then restart the clients with the new name as completed

      • Antoine R. Dumont changed the description

        changed the description

      • Antoine R. Dumont marked the checklist item Finally clean up the old consumer group as completed

        marked the checklist item Finally clean up the old consumer group as completed

      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Author Owner

        It could have been left alone since it's done automatically by kafka:

        root@storage1:~# export OLD_GROUP_ID=swh-archive-stg-swh.loader_metadata.journal_client
root@storage1:~# $KAFKA_CONSUMER_GROUPS \
>   --bootstrap-server $SERVER \
>   --all-topics \
>   --delete \
>   --group $OLD_GROUP_ID
Deletion of requested consumer groups ('swh-archive-stg-swh.loader_metadata.journal_client') was successful.
        Edited by Antoine R. Dumont
      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Author Owner

        It's all fine and dandy with the new consumer group [1]

        [1] https://grafana.softwareheritage.org/goto/89z40KL4z?orgId=1

      • Antoine R. Dumont closed

        closed

      • Antoine R. Dumont added 30m of time spent

        added 30m of time spent

      Please register or sign in to reply