Skip to content

Github lister erroneously delete some loading tasks

Like the Phabricator lister (#2000 (closed)), the Github lister erroneously disable some loading tasks each time a new page of repositories is returned by the Github API.

The cause of the issue is the same as for Phabricator, a wrong index is provided to the disable_deleted_repo_tasks method from the indexing lister.

After adding some debug logs, this is what we can currently observe when executing the lister:

swh-lister_1                  | [2019-09-19 10:27:58,360: DEBUG/ForkPoolWorker-1] <swh.lister.github.lister.GitHubLister object at 0x7f275fc3a550> CONFIG={'content_size_limit': 104857600, 'log_db': 'dbname=softwareheritage-log', 'storage': {'cls': 'remote', 'args': {'url': 'http://swh-storage:5002/'}}, 'scheduler': {'cls': 'remote', 'args': {'url': 'http://swh-scheduler-api:5008/'}}, 'lister': {'cls': 'local', 'args': {'db': 'postgresql://postgres@swh-listers-db/swh-listers'}}, 'celery': {'task_broker': 'amqp://guest:guest@amqp//', 'task_modules': ['swh.lister.bitbucket.tasks', 'swh.lister.cgit.tasks', 'swh.lister.cran.tasks', 'swh.lister.debian.tasks', 'swh.lister.github.tasks', 'swh.lister.gitlab.tasks', 'swh.lister.gnu.tasks', 'swh.lister.npm.tasks', 'swh.lister.packagist.tasks', 'swh.lister.phabricator.tasks', 'swh.lister.pypi.tasks'], 'task_queues': ['swh.lister.bitbucket.tasks.FullBitBucketRelister', 'swh.lister.bitbucket.tasks.IncrementalBitBucketLister', 'swh.lister.bitbucket.tasks.RangeBitBucketLister', 'swh.lister.bitbucket.tasks.ping', 'swh.lister.cgit.tasks.CGitListerTask', 'swh.lister.cgit.tasks.ping', 'swh.lister.cran.tasks.CRANListerTask', 'swh.lister.cran.tasks.ping', 'swh.lister.debian.tasks.DebianListerTask', 'swh.lister.debian.tasks.ping', 'swh.lister.github.tasks.FullGitHubRelister', 'swh.lister.github.tasks.IncrementalGitHubLister', 'swh.lister.github.tasks.RangeGitHubLister', 'swh.lister.github.tasks.ping', 'swh.lister.gitlab.tasks.FullGitLabRelister', 'swh.lister.gitlab.tasks.IncrementalGitLabLister', 'swh.lister.gitlab.tasks.RangeGitLabLister', 'swh.lister.gitlab.tasks.ping', 'swh.lister.gnu.tasks.GNUListerTask', 'swh.lister.gnu.tasks.ping', 'swh.lister.npm.tasks.NpmIncrementalListerTask', 'swh.lister.npm.tasks.NpmListerTask', 'swh.lister.npm.tasks.ping', 'swh.lister.packagist.tasks.PackagistListerTask', 'swh.lister.packagist.tasks.ping', 'swh.lister.phabricator.tasks.FullPhabricatorLister', 'swh.lister.phabricator.tasks.IncrementalPhabricatorLister', 'swh.lister.phabricator.tasks.ping', 'swh.lister.pypi.tasks.PyPIListerTask', 'swh.lister.pypi.tasks.ping']}, 'cache_responses': False, 'cache_dir': '/srv/softwareheritage/.cache/swh/lister/github'}
swh-lister_1                  | [2019-09-19 10:27:59,297: DEBUG/ForkPoolWorker-1] Index: 369
swh-lister_1                  | [2019-09-19 10:28:00,334: DEBUG/ForkPoolWorker-1] loading task for https://github.com/collectiveidea/css_naked_day disabled
swh-lister_1                  | [2019-09-19 10:28:00,334: DEBUG/ForkPoolWorker-1] Index: 876
swh-lister_1                  | [2019-09-19 10:28:01,537: DEBUG/ForkPoolWorker-1] loading task for https://github.com/jackdempsey/merb-core disabled
swh-lister_1                  | [2019-09-19 10:28:01,538: DEBUG/ForkPoolWorker-1] Index: 1531
swh-lister_1                  | [2019-09-19 10:28:02,260: DEBUG/ForkPoolWorker-1] loading task for https://github.com/bobbyrward/tonal disabled
swh-lister_1                  | [2019-09-19 10:28:02,260: DEBUG/ForkPoolWorker-1] Index: 1980
swh-lister_1                  | [2019-09-19 10:28:02,981: DEBUG/ForkPoolWorker-1] loading task for https://github.com/nex3/jazz disabled
swh-lister_1                  | [2019-09-19 10:28:02,981: DEBUG/ForkPoolWorker-1] Index: 2449
swh-lister_1                  | [2019-09-19 10:28:03,791: DEBUG/ForkPoolWorker-1] loading task for https://github.com/patmaddox/rspec-plugin-generator disabled
swh-lister_1                  | [2019-09-19 10:28:03,791: DEBUG/ForkPoolWorker-1] Index: 2905
swh-lister_1                  | [2019-09-19 10:28:04,394: DEBUG/ForkPoolWorker-1] loading task for https://github.com/swsnider/portfolio disabled
swh-lister_1                  | [2019-09-19 10:28:04,394: DEBUG/ForkPoolWorker-1] Index: 3385
swh-lister_1                  | [2019-09-19 10:28:05,437: DEBUG/ForkPoolWorker-1] loading task for https://github.com/lmarlow/godo disabled
swh-lister_1                  | [2019-09-19 10:28:05,437: DEBUG/ForkPoolWorker-1] Index: 3843
swh-lister_1                  | [2019-09-19 10:28:06,472: DEBUG/ForkPoolWorker-1] loading task for https://github.com/spicycode/spicy-config disabled
swh-lister_1                  | [2019-09-19 10:28:06,472: DEBUG/ForkPoolWorker-1] Index: 4396
swh-lister_1                  | [2019-09-19 10:28:07,267: DEBUG/ForkPoolWorker-1] loading task for https://github.com/herzi/source-browser disabled
swh-lister_1                  | [2019-09-19 10:28:07,267: DEBUG/ForkPoolWorker-1] Index: 4865
swh-lister_1                  | [2019-09-19 10:28:08,046: DEBUG/ForkPoolWorker-1] loading task for https://github.com/KirinDave/powerset_thrift disabled
swh-lister_1                  | [2019-09-19 10:28:08,046: DEBUG/ForkPoolWorker-1] Index: 5250
swh-lister_1                  | [2019-09-19 10:28:08,880: DEBUG/ForkPoolWorker-1] loading task for https://github.com/zimbatm/facts disabled
swh-lister_1                  | [2019-09-19 10:28:08,880: DEBUG/ForkPoolWorker-1] Index: 5792
swh-lister_1                  | [2019-09-19 10:28:09,631: DEBUG/ForkPoolWorker-1] loading task for https://github.com/jstewart/mhs_authentication_system disabled
swh-lister_1                  | [2019-09-19 10:28:09,632: DEBUG/ForkPoolWorker-1] Index: 6378
swh-lister_1                  | [2019-09-19 10:28:10,391: DEBUG/ForkPoolWorker-1] loading task for https://github.com/robey/configgy disabled
swh-lister_1                  | [2019-09-19 10:28:10,391: DEBUG/ForkPoolWorker-1] Index: 7030

Migrated from T2014 (view on Phabricator)