staging origin intrinsic metadata indexer are stuck
There is no consumer for that indexer in staging. After a systemctl status on the dedicated journal client service, they are all showing the same warning [2].
root@pergamon:~# clush -b -w @staging-workers 'systemctl status swh-indexer-journal-client@origin_intrinsic_metadata_1'
---------------
worker0.internal.staging.swh.network
---------------
● swh-indexer-journal-client@origin_intrinsic_metadata_1.service - Software Heritage Indexer Journal Client origin_intrinsic_metadata_1
Loaded: loaded (/etc/systemd/system/swh-indexer-journal-client@.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/swh-indexer-journal-client@origin_intrinsic_metadata_1.service.d
└─parameters.conf
Active: active (running) since Wed 2022-08-31 14:05:35 UTC; 33min ago
Main PID: 2748153 (swh)
Tasks: 6 (limit: 28838)
Memory: 140.7M
CGroup: /system.slice/system-swh\x2dindexer\x2djournal\x2dclient.slice/swh-indexer-journal-client@origin_intrinsic_metadata_1.service
└─2748153 /usr/bin/python3 /usr/bin/swh --log-level INFO --log-level azure.core.pipeline.policies.http_logging_policy:WARNING indexer --config-file /etc/softwareheritage/indexer/origin_intrinsic_metadata.yml journal-client origin_intrinsic_metadata
Aug 31 14:05:35 worker0 systemd[1]: swh-indexer-journal-client@origin_intrinsic_metadata_1.service: Scheduled restart job, restart counter is at 1.
Aug 31 14:05:35 worker0 systemd[1]: Stopped Software Heritage Indexer Journal Client origin_intrinsic_metadata_1.
Aug 31 14:05:35 worker0 systemd[1]: Started Software Heritage Indexer Journal Client origin_intrinsic_metadata_1.
Aug 31 14:20:58 worker0 swh[2748153]: WARNING:swh.journal.client.rdkafka:MAXPOLL [rdkafka#consumer-1] [thrd:main]: Application maximum poll interval (300000ms) exceeded by 313ms (adjust max.poll.interval.ms for long-running message processing): leaving group
Aug 31 14:20:58 worker0 swh[2748153]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=UNKNOWN_MEMBER_ID,val=25,str="Broker: Unknown member"}
Aug 31 14:20:59 worker0 swh[2748153]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_MAX_POLL_EXCEEDED,val=-147,str="Application maximum poll interval (300000ms) exceeded by 313ms"}
Aug 31 14:32:15 worker0 swh[2748153]: WARNING:swh.journal.client.rdkafka:MAXPOLL [rdkafka#consumer-1] [thrd:main]: Application maximum poll interval (300000ms) exceeded by 396ms (adjust max.poll.interval.ms for long-running message processing): leaving group
Aug 31 14:32:15 worker0 swh[2748153]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=UNKNOWN_MEMBER_ID,val=25,str="Broker: Unknown member"}
Aug 31 14:32:16 worker0 swh[2748153]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_MAX_POLL_EXCEEDED,val=-147,str="Application maximum poll interval (300000ms) exceeded by 396ms"}
---------------
worker1.internal.staging.swh.network
---------------
● swh-indexer-journal-client@origin_intrinsic_metadata_1.service - Software Heritage Indexer Journal Client origin_intrinsic_metadata_1
Loaded: loaded (/etc/systemd/system/swh-indexer-journal-client@.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/swh-indexer-journal-client@origin_intrinsic_metadata_1.service.d
└─parameters.conf
Active: active (running) since Wed 2022-08-31 14:03:27 UTC; 35min ago
Main PID: 2743429 (swh)
Tasks: 6 (limit: 28838)
Memory: 134.7M
CGroup: /system.slice/system-swh\x2dindexer\x2djournal\x2dclient.slice/swh-indexer-journal-client@origin_intrinsic_metadata_1.service
└─2743429 /usr/bin/python3 /usr/bin/swh --log-level INFO --log-level azure.core.pipeline.policies.http_logging_policy:WARNING indexer --config-file /etc/softwareheritage/indexer/origin_intrinsic_metadata.yml journal-client origin_intrinsic_metadata
Aug 31 14:03:27 worker1 systemd[1]: Started Software Heritage Indexer Journal Client origin_intrinsic_metadata_1.
Aug 31 14:16:38 worker1 swh[2743429]: WARNING:swh.journal.client.rdkafka:MAXPOLL [rdkafka#consumer-1] [thrd:main]: Application maximum poll interval (300000ms) exceeded by 462ms (adjust max.poll.interval.ms for long-running message processing): leaving group
Aug 31 14:16:38 worker1 swh[2743429]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=UNKNOWN_MEMBER_ID,val=25,str="Broker: Unknown member"}
Aug 31 14:16:39 worker1 swh[2743429]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_MAX_POLL_EXCEEDED,val=-147,str="Application maximum poll interval (300000ms) exceeded by 462ms"}
Aug 31 14:32:50 worker1 swh[2743429]: WARNING:swh.journal.client.rdkafka:MAXPOLL [rdkafka#consumer-1] [thrd:main]: Application maximum poll interval (300000ms) exceeded by 418ms (adjust max.poll.interval.ms for long-running message processing): leaving group
Aug 31 14:32:50 worker1 swh[2743429]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=UNKNOWN_MEMBER_ID,val=25,str="Broker: Unknown member"}
Aug 31 14:32:51 worker1 swh[2743429]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_MAX_POLL_EXCEEDED,val=-147,str="Application maximum poll interval (300000ms) exceeded by 418ms"}
---------------
worker2.internal.staging.swh.network
---------------
● swh-indexer-journal-client@origin_intrinsic_metadata_1.service - Software Heritage Indexer Journal Client origin_intrinsic_metadata_1
Loaded: loaded (/etc/systemd/system/swh-indexer-journal-client@.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/swh-indexer-journal-client@origin_intrinsic_metadata_1.service.d
└─parameters.conf
Active: active (running) since Wed 2022-08-31 14:03:27 UTC; 35min ago
Main PID: 2758980 (swh)
Tasks: 6 (limit: 28840)
Memory: 62.2M
CGroup: /system.slice/system-swh\x2dindexer\x2djournal\x2dclient.slice/swh-indexer-journal-client@origin_intrinsic_metadata_1.service
└─2758980 /usr/bin/python3 /usr/bin/swh --log-level INFO --log-level azure.core.pipeline.policies.http_logging_policy:WARNING indexer --config-file /etc/softwareheritage/indexer/origin_intrinsic_metadata.yml journal-client origin_intrinsic_metadata
Aug 31 14:03:27 worker2 systemd[1]: Started Software Heritage Indexer Journal Client origin_intrinsic_metadata_1.
Aug 31 14:16:28 worker2 swh[2758980]: WARNING:swh.journal.client.rdkafka:MAXPOLL [rdkafka#consumer-1] [thrd:main]: Application maximum poll interval (300000ms) exceeded by 490ms (adjust max.poll.interval.ms for long-running message processing): leaving group
Aug 31 14:16:29 worker2 swh[2758980]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=UNKNOWN_MEMBER_ID,val=25,str="Broker: Unknown member"}
Aug 31 14:16:29 worker2 swh[2758980]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_MAX_POLL_EXCEEDED,val=-147,str="Application maximum poll interval (300000ms) exceeded by 490ms"}
Aug 31 14:29:24 worker2 swh[2758980]: WARNING:swh.journal.client.rdkafka:MAXPOLL [rdkafka#consumer-1] [thrd:main]: Application maximum poll interval (300000ms) exceeded by 337ms (adjust max.poll.interval.ms for long-running message processing): leaving group
Aug 31 14:29:24 worker2 swh[2758980]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=UNKNOWN_MEMBER_ID,val=25,str="Broker: Unknown member"}
Aug 31 14:29:25 worker2 swh[2758980]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_MAX_POLL_EXCEEDED,val=-147,str="Application maximum poll interval (300000ms) exceeded by 337ms"}
Aug 31 14:35:44 worker2 swh[2758980]: WARNING:swh.journal.client.rdkafka:MAXPOLL [rdkafka#consumer-1] [thrd:main]: Application maximum poll interval (300000ms) exceeded by 286ms (adjust max.poll.interval.ms for long-running message processing): leaving group
Aug 31 14:35:45 worker2 swh[2758980]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=UNKNOWN_MEMBER_ID,val=25,str="Broker: Unknown member"}
Aug 31 14:35:45 worker2 swh[2758980]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_MAX_POLL_EXCEEDED,val=-147,str="Application maximum poll interval (300000ms) exceeded by 286ms"}
---------------
worker3.internal.staging.swh.network
---------------
● swh-indexer-journal-client@origin_intrinsic_metadata_1.service - Software Heritage Indexer Journal Client origin_intrinsic_metadata_1
Loaded: loaded (/etc/systemd/system/swh-indexer-journal-client@.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/swh-indexer-journal-client@origin_intrinsic_metadata_1.service.d
└─parameters.conf
Active: active (running) since Wed 2022-08-31 14:03:27 UTC; 35min ago
Main PID: 2562808 (swh)
Tasks: 6 (limit: 28889)
Memory: 134.7M
CGroup: /system.slice/system-swh\x2dindexer\x2djournal\x2dclient.slice/swh-indexer-journal-client@origin_intrinsic_metadata_1.service
└─2562808 /usr/bin/python3 /usr/bin/swh --log-level INFO --log-level azure.core.pipeline.policies.http_logging_policy:WARNING indexer --config-file /etc/softwareheritage/indexer/origin_intrinsic_metadata.yml journal-client origin_intrinsic_metadata
Aug 31 14:03:27 worker3 systemd[1]: Started Software Heritage Indexer Journal Client origin_intrinsic_metadata_1.
Aug 31 14:03:28 worker3 swh[2562808]: WARNING:swh.core.cli:Could not load subcommand graph: DistributionNotFound(Requirement.parse('py4j'), None)
Aug 31 14:16:48 worker3 swh[2562808]: WARNING:swh.journal.client.rdkafka:MAXPOLL [rdkafka#consumer-1] [thrd:main]: Application maximum poll interval (300000ms) exceeded by 20ms (adjust max.poll.interval.ms for long-running message processing): leaving group
Aug 31 14:16:48 worker3 swh[2562808]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=UNKNOWN_MEMBER_ID,val=25,str="Broker: Unknown member"}
Aug 31 14:16:49 worker3 swh[2562808]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_MAX_POLL_EXCEEDED,val=-147,str="Application maximum poll interval (300000ms) exceeded by 20ms"}
Aug 31 14:32:44 worker3 swh[2562808]: WARNING:swh.journal.client.rdkafka:MAXPOLL [rdkafka#consumer-1] [thrd:main]: Application maximum poll interval (300000ms) exceeded by 461ms (adjust max.poll.interval.ms for long-running message processing): leaving group
Aug 31 14:32:44 worker3 swh[2562808]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=UNKNOWN_MEMBER_ID,val=25,str="Broker: Unknown member"}
Aug 31 14:32:45 worker3 swh[2562808]: INFO:swh.journal.client:Received non-fatal kafka error: KafkaError{code=_MAX_POLL_EXCEEDED,val=-147,str="Application maximum poll interval (300000ms) exceeded by 461ms"}Migrated from T4477 (view on Phabricator)