Activate loader-metadata in production

To be merged once the necessary checks on the staging dataset is done (next monday).

Refs. swh/infra/sysadm-environment#4794 (closed)

cc @vlorentz (jsyk)

mentioned in issue swh/infra/sysadm-environment#4794 (closed)

added 1 commit

• aa53b311 - Activate loader-metadata in production

Compare with previous version

resolved all threads

added 7 commits

• aa53b311...4ba124fa - 6 commits from branch production
• fd061022 - Activate loader-metadata in production

Compare with previous version

looks like the consumer group is not prefixed by the user name

I recall it's the chart that does the proper naming when using authentication (always now). I'll double check.

you are right, with: {{ $journalUser }}-{{ $consumerGroup }} so you'll have a consumer group swh-archive-prod.swh.loader_metadata.journal_client

WDYT to just use loader_metadata.journal_client

I recall it's the chart that does the proper naming when using authentication (always now). I'll double check.

^ yes, that's it!

      group_id: {{ $journalUser }}-{{ $consumerGroup }}

WDYT to just use loader_metadata.journal_client

I'm fine with the idea. What should happen for staging though?

changed this line in version 4 of the diff

Simplified the consumer group (drop swh.). done

I'm fine with the idea. What should happen for staging though?

I would say 2 options, let it with the long name, or change it and restart from scratch. ;)

i'm a bit underwhelmed at the idea of 1. so i'll do nothing (and keep the long name ;)

With the kafka cli, you can export the offsets of the current consumer group, import them into the new name, then restart the clients with the new name, and finally clean up the old consumer group

thanks, i'll add this to the bottom of my weekly todo list.

.

- [ ] staging: loader-metadata [0/6]
  - [ ] swh-charts: Rename consumer group loader_metadata.journal_client
  - [ ] Stop journal clients
  - [ ] Export the offsets of the current consumer group
  - [ ] Import them into the new name
  - [ ] Then restart the clients with the new name
  - [ ] Finally clean up the old consumer group

added 1 commit

• 2194f3da - Activate loader-metadata in production

Compare with previous version

approved this merge request

merged