Skip to content

pattern: Make lister flush regularly origins to scheduler

Antoine R. Dumont requested to merge generated-differential-D4965-source into generated-differential-D4965-target

As origins is a generator, the previous behavior would try to consume the overall generator to send the records.

This groups and sends batch of 100 origins to the scheduler for writing.

Related to #3003 (closed)

Test Plan

tox

Plus running the launchpad lister which now runs and writes data alongside the listing instead of failing without writing anything.

From 0 before to growing

swh-scheduler=> select now(), count(*) from listed_origins lo inner join listers l on lo.lister_id=l.id and l.name='launchpad' and l.instance_name='launchpad';
              now              | count
-------------------------------+-------
 2021-01-28 15:54:50.063613+00 | 18000
(1 row)

Migrated from D4965 (view on Phabricator)

Merge request reports