indexing-lister: Allow to define flush packet size
Prior to this commit, indexing lister instances were flushing every packet of 20. This can now be defined per sub classes.
For the bitbucket lister, as the number of repositories grew from 10 per page to 100 per page, that enlarged the time frame between flushes.
Depends on !378 (closed)
Test Plan
tox
Migrated from D1635 (view on Phabricator)
Merge request reports
Activity
Build has FAILED
Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tox/229/ See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tox/229/console
-
Build has FAILED
Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tox/231/ See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tox/231/console
Rebase on latest !377 (closed)'s commit content
-
Rebase on latest !377 (closed)
-
Build has FAILED
Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tox/233/ See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tox/233/console
-
Build has FAILED
Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tox/235/ See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tox/235/console
-
Build has FAILED
Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tox/246/ See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tox/246/console
-
Build has FAILED
Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tox/250/ See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tox/250/console
-
Build has FAILED
Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tox/258/ See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tox/258/console
-
Build has FAILED
Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tox/259/ See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tox/259/console
-
Build has FAILED
Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tox/263/ See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tox/263/console
-
Build is green See https://jenkins.softwareheritage.org/job/DLS/job/tox/267/ for more details.
30 30 if per_page != DEFAULT_BITBUCKET_PAGE: 31 31 self.PATH_TEMPLATE = '%s&pagelen=%s' % ( 32 32 self.PATH_TEMPLATE, per_page) 33 # to stay consistent with prior behavior (20 * 10 repositories then) -
Because i changed the packet size returned by the api from 10 repositories (too small) to 100 repositories (a tad better) for the bitbucket listing.
so 2 iterations of 100 repositories, i already have the 200 repositories to flush in db. If i kept the original indexing lister, i would have changed the behavior to flush every 2000 repositories.