A couple of loaders are stuck after a celery/amqp timeout

loaders {"asctime": "2025-03-05 08:24:09,450", "threadName": "MainThread", "pathname": "/opt/swh/venv/lib/python3.11/site-packages/celery/worker/strategy.py", "lineno": 161, "funcName": "task_message_handler", "task_name": null, "task_id": null, "name": "celery.worker.strategy", "levelname": "INFO", "message": "Task swh.loader.svn.t
asks.DumpMountAndLoadSvnRepository[6e8361b5-822f-4a3c-a827-c55df1931e1c] received", "data": {"id": "6e8361b5-822f-4a3c-a827-c55df1931e1c", "name": "swh.loader.svn.tasks.DumpMountAndLoadSvnRepository", "args": "()", "kwargs": "{'url': 'https://svn.sdss.org/public/data/eboss/DR16cosmo/tags/v1_0_1'}", "eta": null}}
loaders {"asctime": "2025-03-05 08:24:09,623", "threadName": "MainThread", "pathname": "/opt/swh/venv/lib/python3.11/site-packages/swh/loader/core/loader.py", "lineno": 413, "funcName": "load", "task_name": null, "task_id": null, "name": "swh.loader.svn.loader.SvnLoaderFromRemoteDump", "levelname": "INFO", "message": "Load origin 'h
ttps://svn.sdss.org/public/data/eboss/DR16cosmo/tags/v1_0_1' with type 'svn'"}
loaders {"asctime": "2025-03-05 08:54:55,750", "threadName": "MainThread", "pathname": "/opt/swh/venv/lib/python3.11/site-packages/celery/worker/worker.py", "lineno": 206, "funcName": "start", "task_name": null, "task_id": null, "name": "celery.worker", "levelname": "CRITICAL", "message": "Unrecoverable error: PreconditionFailed(406
, 'PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more', (0, 0), '')", "exc_info": "Traceback (most recent call last):
  File \"/opt/swh/venv/lib/python3.11/site-packages/celery/worker/worker.py\", line 202
, in start
    self.blueprint.start(self)
  File \"/opt/swh/venv/lib/python3.11/site-packages/celery/bootsteps.py\", line 116, in start
    step.start(parent)
  File \"/opt/swh/venv/lib/python3.11/site-packages/celery/bootsteps.py\", line 365, in start
    return self.obj.start()
           ^^^^^^^^^^^^^^^^
  File \"/opt/swh/venv/lib/python3.11/site-packages/celery/worker/consumer/consumer.py\", line 340, in start blueprint.start(self)
  File \"/opt/swh/venv/lib/python3.11/site-packages/celery/bootsteps.py\", line 116, in start step.start(parent)
  File \"/opt/swh/venv/lib/python3.11/site-packages/celery/worker/consumer/consumer.py\", line 746, in start
    c.loop(*c.loop_args())
  File \"/opt/swh/venv/lib/python3.11/site-packages/celery/worker/loops.py\", line 97, in asynloop
    next(loop)
  File \"/opt/swh/venv/lib/python3.11/site-packages/kombu/asynchronous/hub.py\", line 373, in create_loop
    cb(*cbargs)
  File \"/opt/swh/venv/lib/python3.11/site-packages/kombu/transport/base.py\", line 248, in on_readable
    reader(loop)
  File \"/opt/swh/venv/lib/python3.11/site-packages/kombu/transport/base.py\", line 230, in _read
    drain_events(timeout=0)
  File \"/opt/swh/venv/lib/python3.11/site-packages/amqp/connection.py\", line 526, in drain_events
    while not self.blocking_read(timeout):
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/opt/swh/venv/lib/python3.11/site-packages/amqp/connection.py\", line 532, in blocking_read
    return self.on_inbound_frame(frame)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/opt/swh/venv/lib/python3.11/site-packages/amqp/method_framing.py\", line 53, in on_frame
    callback(channel, method_sig, buf, None)
  File \"/opt/swh/venv/lib/python3.11/site-packages/amqp/connection.py\", line 538, in on_inbound_method
    return self.channels[channel_id].dispatch_method(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/opt/swh/venv/lib/python3.11/site-packages/amqp/abstract_channel.py\", line 156, in dispatch_method
    listener(*args)
  File \"/opt/swh/venv/lib/python3.11/site-packages/amqp/channel.py\", line 293, in _on_close
    raise error_for_code(amqp.exceptions.PreconditionFailed: (0, 0): (406) PRECONDITION_FAILED - delivery acknowledgement on channel 1 timed out. Timeout value used: 1800000 ms. This timeout value can be configured, see consumers doc guide to learn more"}

It seems it starts after a reboot of saatchi and several loaders are impacted (svn scn, deposit, ...)

added environment: production label

The exception doesnt look to be sent to sentry.

It also add a couple of problems with the scheduler runner recurrent that could not create some queues.

the queues was present in rabbitmq but totally empty. After a manual removal, the runner could create the queue and sent messages on it

Sentry Issue: SWH-SCHEDULER-FF

saatchi was not only reboot but also upgraded from bullseye to bookworm.

the rabbitmq version was updated from 3.8 to 3.10.

It seems the upgrade to several minor version directly is not recommended but could work: https://www.rabbitmq.com/docs/3.13/upgrade#rabbitmq-version-upgradability

The docker environment and swh-next-version are running in 3.13 so it should not be an hard compatibility issue. Maybe a default configuration change.

upgrade:

rabbitmq-server:amd64 (3.8.9-3+deb11u1, 3.10.8-1.1+deb12u1)

Related to #5415 (comment 198571)

marked this issue as related to #5415 (closed)

It seems before 3.8.15 there were no default value for the consumer_timeout property

https://github.com/rabbitmq/rabbitmq-server/pull/2990

https://www.rabbitmq.com/docs/consumers#per-node-configuration

mentioned in commit swh/infra/puppet/puppet-swh-site@11595042

mentioned in merge request swh/infra/puppet/puppet-swh-site!727 (merged)

changed title from A couple of loader are stucks after a celery/amqp timeout to A couple of loaders are stucks after a celery/amqp timeout

mentioned in commit swh/infra/puppet/puppet-swh-site@5f562c67

mentioned in commit swh/infra/puppet/puppet-swh-site@b25fa1c0

mentioned in commit swh/infra/puppet/puppet-swh-site@9e11ea37

mentioned in commit swh/infra/puppet/puppet-swh-site@02aac500

mentioned in commit swh/infra/puppet/puppet-swh-site@db5f3f3b

mentioned in commit swh/infra/puppet/puppet-swh-site@2ed215ab

The configuration update is deployed in staging and production.

Everything looks good so far , no more exceptions in the loaders after 30mn.

Nothing is suspect in the ingestion dashboards or rabbitmq/scheduler dashboards.

I closed the bunch of sentry issues that was triggered by the restarts of the rabbitmq

closed

changed title from A couple of loaders are stucks after a celery/amqp timeout to A couple of loaders are stuck after a celery/amqp timeout