Split models into smaller chunks to avoid oversized db transactions

When testing the incoming cgit lister on that instance: http://hdiff.luite.com/cgit/, the lister successfully retrieves all hosted repositories but fails to flush the results into the local lister database (python3 process ended up hanging).

The issue came from the fact the underlying db transaction was too large. The solution to resolve the issue is to split repo models into smaller chunks to avoid that behavior. Processing like this also flushes results to database more frequently.

Related #1659 (closed)

---

Migrated from D1660 (view on Phabricator)

swh/lister/core/simple_lister.py

@@ -45,7 +45,7 @@ class SimpleLister(ListerBase):
         models_list = self.transport_response_simplified(response)
         models_list = self.filter_before_inject(models_list)
         all_injected = []
-        for models in utils.grouper(models_list, n=10000):
+        for models in utils.grouper(models_list, n=1000):
             models = list(models)
             logging.debug('models: %s' % len(models))
             # inject into local db