Skip to content

bitbucket lister: Deal with connection errors

Bitbucket lister did more work than before, currently listed 510k repositories (againt 212 IMSMW). It seems to be stuck there though.

It's either a rate limit, a blacklist or a bug happening somewhere.

From kibana's dashboard, we can see the following [3] (the date is the index).

[2019-06-28 08:46:30,942: WARNING/ForkPoolWorker-5] giving up on 2015-05-02 12:37:07.608575+00:00: max retries exceeded
[2019-06-28 08:46:20,931: WARNING/ForkPoolWorker-5] connection error on 2015-05-02 12:37:07.608575+00:00: sleep for 10 seconds
[2019-06-28 08:46:09,366: WARNING/ForkPoolWorker-5] connection error on 2015-05-02 12:37:07.608575+00:00: sleep for 10 seconds

...
Jul 04 07:57:20 worker09 python3[15600]: [2019-07-04 07:57:20,742: INFO/MainProcess] Received task: swh.lister.bitbucket.tasks.IncrementalBitBucketLister[5f4b1d51-c3e0-4615-9a55-59676ef13969]
Jul 04 07:57:22 worker09 python3[15609]: [2019-07-04 07:57:22,565: WARNING/ForkPoolWorker-2] connection error on 2015-05-02 12:37:07.608575+00:00: sleep for 10 seconds
Jul 04 07:57:33 worker09 python3[15609]: [2019-07-04 07:57:33,918: WARNING/ForkPoolWorker-2] connection error on 2015-05-02 12:37:07.608575+00:00: sleep for 10 seconds
Jul 04 07:57:45 worker09 python3[15609]: [2019-07-04 07:57:45,114: WARNING/ForkPoolWorker-2] connection error on 2015-05-02 12:37:07.608575+00:00: sleep for 10 seconds
Jul 04 07:57:56 worker09 python3[15609]: [2019-07-04 07:57:56,625: WARNING/ForkPoolWorker-2] connection error on 2015-05-02 12:37:07.608575+00:00: sleep for 10 seconds
Jul 04 07:58:08 worker09 python3[15609]: [2019-07-04 07:58:08,369: WARNING/ForkPoolWorker-2] connection error on 2015-05-02 12:37:07.608575+00:00: sleep for 10 seconds
Jul 04 07:58:19 worker09 python3[15609]: [2019-07-04 07:58:19,629: WARNING/ForkPoolWorker-2] connection error on 2015-05-02 12:37:07.608575+00:00: sleep for 10 seconds
Jul 04 07:58:30 worker09 python3[15609]: [2019-07-04 07:58:30,819: WARNING/ForkPoolWorker-2] connection error on 2015-05-02 12:37:07.608575+00:00: sleep for 10 seconds
Jul 04 07:58:40 worker09 python3[15609]: [2019-07-04 07:58:40,830: WARNING/ForkPoolWorker-2] giving up on 2015-05-02 12:37:07.608575+00:00: max retries exceeded
Jul 04 07:58:40 worker09 python3[15609]: [2019-07-04 07:58:40,896: INFO/ForkPoolWorker-2] Task swh.lister.bitbucket.tasks.IncrementalBitBucketLister[5f4b1d51-c3e0-4615-9a55-59676ef13969] succeeded in 80.05816953699104s: None

As there are rate limit policies in bitbucket [2], that could be the reason.

Migrated from T1859 (view on Phabricator)

Edited by Phabricator Migration user