Date overflow error in scheduler journal client
In production and staging, the scheduler journal clients are failing with this error:
Aug 25 16:02:32 scheduler0 systemd[1]: Stopped Software Heritage Scheduler Journal Client.
Aug 25 16:02:32 scheduler0 systemd[1]: swh-scheduler-journal-client.service: Consumed 1.006s CPU time.
Aug 25 16:02:32 scheduler0 systemd[1]: Started Software Heritage Scheduler Journal Client.
Aug 25 16:02:36 scheduler0 swh[419915]: Traceback (most recent call last):
Aug 25 16:02:36 scheduler0 swh[419915]: File "/usr/bin/swh", line 11, in <module>
Aug 25 16:02:36 scheduler0 swh[419915]: load_entry_point('swh.core==0.14.4', 'console_scripts', 'swh')()
Aug 25 16:02:36 scheduler0 swh[419915]: File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 185, in main
Aug 25 16:02:36 scheduler0 swh[419915]: return swh(auto_envvar_prefix="SWH")
Aug 25 16:02:36 scheduler0 swh[419915]: File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
Aug 25 16:02:36 scheduler0 swh[419915]: return self.main(*args, **kwargs)
Aug 25 16:02:36 scheduler0 swh[419915]: File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
Aug 25 16:02:36 scheduler0 swh[419915]: rv = self.invoke(ctx)
Aug 25 16:02:36 scheduler0 swh[419915]: File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
Aug 25 16:02:36 scheduler0 swh[419915]: return _process_result(sub_ctx.command.invoke(sub_ctx))
Aug 25 16:02:36 scheduler0 swh[419915]: File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
Aug 25 16:02:36 scheduler0 swh[419915]: return _process_result(sub_ctx.command.invoke(sub_ctx))
Aug 25 16:02:36 scheduler0 swh[419915]: File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
Aug 25 16:02:36 scheduler0 swh[419915]: return ctx.invoke(self.callback, **ctx.params)
Aug 25 16:02:36 scheduler0 swh[419915]: File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
Aug 25 16:02:36 scheduler0 swh[419915]: return callback(*args, **kwargs)
Aug 25 16:02:36 scheduler0 swh[419915]: File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
Aug 25 16:02:36 scheduler0 swh[419915]: return f(get_current_context(), *args, **kwargs)
Aug 25 16:02:36 scheduler0 swh[419915]: File "/usr/lib/python3/dist-packages/swh/scheduler/cli/journal.py", line 52, in visit_stats_journal_client
Aug 25 16:02:36 scheduler0 swh[419915]: nb_messages = client.process(worker_fn)
Aug 25 16:02:36 scheduler0 swh[419915]: File "/usr/lib/python3/dist-packages/swh/journal/client.py", line 265, in process
Aug 25 16:02:36 scheduler0 swh[419915]: batch_processed, at_eof = self.handle_messages(messages, worker_fn)
Aug 25 16:02:36 scheduler0 swh[419915]: File "/usr/lib/python3/dist-packages/swh/journal/client.py", line 292, in handle_messages
Aug 25 16:02:36 scheduler0 swh[419915]: worker_fn(dict(objects))
Aug 25 16:02:36 scheduler0 swh[419915]: File "/usr/lib/python3/dist-packages/swh/scheduler/journal_client.py", line 263, in process_journal_objects
Aug 25 16:02:36 scheduler0 swh[419915]: queue_position_per_visit_type, visit_stats_d
Aug 25 16:02:36 scheduler0 swh[419915]: File "/usr/lib/python3/dist-packages/swh/scheduler/journal_client.py", line 95, in next_visit_queue_position
Aug 25 16:02:36 scheduler0 swh[419915]: return current_position + visit_interval
Aug 25 16:02:36 scheduler0 swh[419915]: OverflowError: date value out of range
Aug 25 16:02:36 scheduler0 systemd[1]: swh-scheduler-journal-client.service: Main process exited, code=exited, status=1/FAILUREthere is a lot of lag in the message consumption in both environments (11025670 messages in production, 888188 in staging)
Migrated from T3502 (view on Phabricator)