indexing-lister: Allow to define flush packet size

Prior to this commit, indexing lister instances were flushing every packet of 20. This can now be defined per sub classes.

For the bitbucket lister, as the number of repositories grew from 10 per page to 100 per page, that enlarged the time frame between flushes.

Depends on !378 (closed)

Test Plan

tox

---

Migrated from D1635 (view on Phabricator)

swh/lister/bitbucket/lister.py

@@ -30,6 +30,9 @@ class BitBucketLister(IndexingHttpLister):
         if per_page != DEFAULT_BITBUCKET_PAGE:
             self.PATH_TEMPLATE = '%s&pagelen=%s' % (
                 self.PATH_TEMPLATE, per_page)
+        # to stay consistent with prior behavior (20 * 10 repositories then)
+        self.flush_packet_db = int(
+            (self.flush_packet_db * DEFAULT_BITBUCKET_PAGE) / per_page)
 
     def get_model_from_repo(self, repo):
         return {

[vlorentz]

I'm confused by this comment. Prior behavior of what? (I can deduce IndexingLister because it's in the same diff, but it won't make sense afterward.) And why does the Bitbucket lister need to override this behavior?

[Antoine R. Dumont]

Because i changed the packet size returned by the api from 10 repositories (too small) to 100 repositories (a tad better) for the bitbucket listing.

so 2 iterations of 100 repositories, i already have the 200 repositories to flush in db. If i kept the original indexing lister, i would have changed the behavior to flush every 2000 repositories.