Skip to content
Snippets Groups Projects

lister.pattern: Restore flushing origin batch in the scheduler

Merged Antoine R. Dumont requested to merge ardumont/swh-lister:restore-batch-records into master

Prior to this commit, the newly introduced check on url validity was consuming the stream of origins. In effect, this would no longer write origin records regularly.

For all listers, that would translate to flush origins only at the end of the listing which could take a while for some (e.g. packagist lister has been running for more than 12h currently without writing anything in the scheduler).

Seen while working on deploying the packagist lister in production [1]. It got deployed yesterday afternoon and it's still not done. And nothing is currently written in the scheduler. If the process fails for some reason, then all will have to be listed again (and possibly with the same problem so infinite failure loop with a noop).

[1] swh/infra/sysadm-environment#4995 (closed)

Refs. swh/meta#5001 (closed)

Edited by Antoine R. Dumont

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
Please register or sign in to reply
Loading