[docker-compose] Ensure RabbitMQ server is up before starting related services (!66) · Merge requests · Platform / Development / swh-docker-dev

Antoine Lambert requested to merge generated-differential-D1227-source into generated-differential-D1227-target Mar 07, 2019
To ensure all RabbitMQ related services get correctly initialized, simply wait for the RabbitMQ server being up (using the great wait-for-it script) before trying to connect to it.
For instance without that change, the scheduler-listener service fails to initialize while reporting the following errors:
swh-scheduler-listener_1      | Starting the swh-scheduler listener
swh-scheduler-listener_1      | [INFO] swh.core.config -- Loading config file /scheduler.yml
swh-scheduler-listener_1      | Traceback (most recent call last):
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/amqp/transport.py", line 141, in _connect
swh-scheduler-listener_1      |     host, port, family, socket.SOCK_STREAM, SOL_TCP)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/socket.py", line 745, in getaddrinfo
swh-scheduler-listener_1      |     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
swh-scheduler-listener_1      | socket.gaierror: [Errno -2] Name or service not known
swh-scheduler-listener_1      | 
swh-scheduler-listener_1      | During handling of the above exception, another exception occurred:
swh-scheduler-listener_1      | 
swh-scheduler-listener_1      | Traceback (most recent call last):
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
swh-scheduler-listener_1      |     yield
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 422, in ensure_connection
swh-scheduler-listener_1      |     callback, timeout=timeout)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/utils/functional.py", line 341, in retry_over_time
swh-scheduler-listener_1      |     return fun(*args, **kwargs)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 275, in connect
swh-scheduler-listener_1      |     return self.connection
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 823, in connection
swh-scheduler-listener_1      |     self._connection = self._establish_connection()
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 778, in _establish_connection
swh-scheduler-listener_1      |     conn = self.transport.establish_connection()
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
swh-scheduler-listener_1      |     conn.connect()
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/amqp/connection.py", line 307, in connect
swh-scheduler-listener_1      |     self.transport.connect()
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/amqp/transport.py", line 81, in connect
swh-scheduler-listener_1      |     self._connect(self.host, self.port, self.connect_timeout)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/amqp/transport.py", line 152, in _connect
swh-scheduler-listener_1      |     "failed to resolve broker hostname"))
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/amqp/transport.py", line 165, in _connect
swh-scheduler-listener_1      |     self.sock.connect(sa)
swh-scheduler-listener_1      | ConnectionRefusedError: [Errno 111] Connection refused
swh-scheduler-listener_1      | 
swh-scheduler-listener_1      | During handling of the above exception, another exception occurred:
swh-scheduler-listener_1      | 
swh-scheduler-listener_1      | Traceback (most recent call last):
swh-scheduler-listener_1      |   File "/usr/local/bin/swh-scheduler", line 11, in <module>
swh-scheduler-listener_1      |     load_entry_point('swh.scheduler', 'console_scripts', 'swh-scheduler')()
swh-scheduler-listener_1      |   File "/src/swh-scheduler/swh/scheduler/cli.py", line 745, in main
swh-scheduler-listener_1      |     return cli(auto_envvar_prefix='SWH_SCHEDULER')
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 764, in __call__
swh-scheduler-listener_1      |     return self.main(*args, **kwargs)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 717, in main
swh-scheduler-listener_1      |     rv = self.invoke(ctx)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
swh-scheduler-listener_1      |     return _process_result(sub_ctx.command.invoke(sub_ctx))
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 956, in invoke
swh-scheduler-listener_1      |     return ctx.invoke(self.callback, **ctx.params)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/click/core.py", line 555, in invoke
swh-scheduler-listener_1      |     return callback(*args, **kwargs)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 17, in new_func
swh-scheduler-listener_1      |     return f(get_current_context(), *args, **kwargs)
swh-scheduler-listener_1      |   File "/src/swh-scheduler/swh/scheduler/cli.py", line 613, in listener
swh-scheduler-listener_1      |     event_monitor(app, backend=scheduler)
swh-scheduler-listener_1      |   File "/src/swh-scheduler/swh/scheduler/celery_backend/listener.py", line 172, in event_monitor
swh-scheduler-listener_1      |     node_id='listener',
swh-scheduler-listener_1      |   File "/src/swh-scheduler/swh/scheduler/celery_backend/listener.py", line 25, in __init__
swh-scheduler-listener_1      |     channel, handlers, routing_key, node_id, app, queue_prefix, accept)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/celery/events/receiver.py", line 41, in __init__
swh-scheduler-listener_1      |     self.channel = maybe_channel(channel)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 1006, in maybe_channel
swh-scheduler-listener_1      |     return channel.default_channel
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 852, in default_channel
swh-scheduler-listener_1      |     self.ensure_connection(**conn_opts)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 422, in ensure_connection
swh-scheduler-listener_1      |     callback, timeout=timeout)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/contextlib.py", line 99, in __exit__
swh-scheduler-listener_1      |     self.gen.throw(type, value, traceback)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 436, in _reraise_as_library_errors
swh-scheduler-listener_1      |     sys.exc_info()[2])
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/vine/five.py", line 194, in reraise
swh-scheduler-listener_1      |     raise value.with_traceback(tb)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 431, in _reraise_as_library_errors
swh-scheduler-listener_1      |     yield
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 422, in ensure_connection
swh-scheduler-listener_1      |     callback, timeout=timeout)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/utils/functional.py", line 341, in retry_over_time
swh-scheduler-listener_1      |     return fun(*args, **kwargs)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 275, in connect
swh-scheduler-listener_1      |     return self.connection
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 823, in connection
swh-scheduler-listener_1      |     self._connection = self._establish_connection()
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/connection.py", line 778, in _establish_connection
swh-scheduler-listener_1      |     conn = self.transport.establish_connection()
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
swh-scheduler-listener_1      |     conn.connect()
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/amqp/connection.py", line 307, in connect
swh-scheduler-listener_1      |     self.transport.connect()
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/amqp/transport.py", line 81, in connect
swh-scheduler-listener_1      |     self._connect(self.host, self.port, self.connect_timeout)
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/amqp/transport.py", line 152, in _connect
swh-scheduler-listener_1      |     "failed to resolve broker hostname"))
swh-scheduler-listener_1      |   File "/usr/local/lib/python3.6/site-packages/amqp/transport.py", line 165, in _connect
swh-scheduler-listener_1      |     self.sock.connect(sa)
swh-scheduler-listener_1      | kombu.exceptions.OperationalError: [Errno 111] Connection refused
Migrated from D1227 (view on Phabricator)
[docker-compose] Ensure RabbitMQ server is up before starting related services

Merge request reports
