Skip to content
Snippets Groups Projects

save code now: Requests are not getting updated from time to time

    • Open Issue created by Antoine R. Dumont @ardumont

    Some requests are systematically selected for an update without actually getting updated [1]. Even though the corresponding tasks may be done [2]

    • [1] swh-web db:
    Time: 91.013 ms
10:38:41 swh-web@belvedere:5432=> select * from save_origin_request where status='accepted' and (visit_date is null or visit_status is null or visit_status in ('ongoing', 'created')) and request_date >= now() - interval '30 days';
+-------+-------------------------------+------------+-------------------------------------------------------+----------+-----------------+-------------------------------+---------------------+--------------+----------+
|  id   |         request_date          | visit_type |                      origin_url                       |  status  | loading_task_id |          visit_date           | loading_task_status | visit_status | user_ids |
+-------+-------------------------------+------------+-------------------------------------------------------+----------+-----------------+-------------------------------+---------------------+--------------+----------+
| 86401 | 2021-07-17 22:43:29.870783+00 | git        | https://github.com/keybase/client                     | accepted |       396435047 | 2021-07-17 22:43:35.725966+00 | running             | created      | (null)   |
| 87538 | 2021-07-26 12:59:50.038714+00 | svn        | https://svn.r-project.org/R-dev-web/trunk/            | accepted |       396985198 | 2021-07-26 12:59:52.299569+00 | running             | created      | (null)   |
| 87536 | 2021-07-26 12:54:25.784339+00 | svn        | svn://svn.code.sf.net/p/codeblocks/code/trunk         | accepted |       396984987 | 2021-07-26 12:54:31.857131+00 | running             | created      | (null)   |
| 87544 | 2021-07-26 13:34:16.987444+00 | git        | https://android.googlesource.com/platform/system/core | accepted |       396985908 | 2021-07-26 13:34:25.182417+00 | running             | created      | (null)   |
+-------+-------------------------------+------------+-------------------------------------------------------+----------+-----------------+-------------------------------+---------------------+--------------+----------+
(4 rows)

Time: 44.824 ms
    • [2]
    10:40:00 softwareheritage-scheduler@belvedere:5432=> select * from task where id in  (396985198, 396435047, 396984987, 396985908);
+-----------+----------+------------------------------------------------------------------------------------------+-------------------------------+------------------+--------------------+---------+--------------+----------+
|    id     |   type   |                                        arguments                                         |           next_run            | current_interval |       status       | policy  | retries_left | priority |
+-----------+----------+------------------------------------------------------------------------------------------+-------------------------------+------------------+--------------------+---------+--------------+----------+
| 396435047 | load-git | {"args": [], "kwargs": {"url": "https://github.com/keybase/client"}}                     | 2021-08-03 17:22:21.784019+00 | 64 days          | next_run_scheduled | oneshot |            2 | high     |
| 396984987 | load-svn | {"args": [], "kwargs": {"url": "svn://svn.code.sf.net/p/codeblocks/code/trunk"}}         | 2021-08-03 19:05:09.045631+00 | 1 day            | disabled           | oneshot |            0 | high     |
| 396985198 | load-svn | {"args": [], "kwargs": {"url": "https://svn.r-project.org/R-dev-web/trunk/"}}            | 2021-08-03 16:15:35.306511+00 | 1 day            | completed          | oneshot |            2 | high     |
| 396985908 | load-git | {"args": [], "kwargs": {"url": "https://android.googlesource.com/platform/system/core"}} | 2021-08-03 16:15:35.306511+00 | 64 days          | completed          | oneshot |            2 | high     |
+-----------+----------+------------------------------------------------------------------------------------------+-------------------------------+------------------+--------------------+---------+--------------+----------+
(4 rows)

Time: 9.306 ms

    (don't mind keybase, it's another matter

    Migrated from T3458 (view on Phabricator)

    Edited
    To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information

    Child items 0

    • No child items are currently assigned. Use child items to break down this issue into smaller parts.

    Activity

    • Antoine R. Dumont added Save Code Now priority:Normal labels

      added Save Code Now priority:Normal labels

    • Antoine Lambert
      Antoine Lambert @anlambert ·
      Maintainer

      Indeed some save requests are still marked as running while they clearly are not.

      I slightly modified the refresh_savecodenow_statuses django command to update all requests marked as running regardless of their date. I also commented that piece of code temporarily.

      Below are the state of those save requests before attempting to update their statuses, for each of them are printed the SaveOriginRequest django model, the scheduler task and the scheduler task_run.

      <SaveOriginRequest: {'id': 90358, 'origin_url': 'https://github.com/mozilla/gecko-dev', 'visit_type': 'git', 'save_request_date': '2021-08-27T18:05:28.837000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': '2021-08-27T18:05:31.152000+00:00', 'loading_task_id': 398327718}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://github.com/mozilla/gecko-dev'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 398327718,
 'next_run': datetime.datetime(2021, 8, 27, 20, 48, 31, 653577, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 2,
 'status': 'completed',
 'type': 'load-git'}
{'backend_id': '7d17cf6d-1de5-43e7-af25-da5c96b19f3d',
 'ended': datetime.datetime(2021, 8, 27, 19, 48, 31, 653577, tzinfo=datetime.timezone.utc),
 'id': 1422313746,
 'metadata': {'worker': 'loader_high_priority@worker01.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 8, 27, 18, 5, 31, 125969, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 8, 27, 18, 5, 31, 151958, tzinfo=datetime.timezone.utc),
 'status': 'failed',
 'task': 398327718}

<SaveOriginRequest: {'id': 89569, 'origin_url': 'https://github.com/404-not-find/client', 'visit_type': 'git', 'save_request_date': '2021-08-21T08:06:47.263000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': '2021-08-21T08:06:48.337000+00:00', 'loading_task_id': 398036169}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://github.com/404-not-find/client'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 398036169,
 'next_run': datetime.datetime(2021, 8, 21, 8, 6, 47, 232838, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 3,
 'status': 'next_run_scheduled',
 'type': 'load-git'}
{'backend_id': 'd8c57ab6-01ed-43f2-aed0-644c7e40288b',
 'ended': None,
 'id': 1421317589,
 'metadata': None,
 'scheduled': datetime.datetime(2021, 8, 21, 8, 6, 48, 309219, tzinfo=datetime.timezone.utc),
 'started': None,
 'status': 'scheduled',
 'task': 398036169}

<SaveOriginRequest: {'id': 88780, 'origin_url': 'https://github.com/CocoaPods/Specs', 'visit_type': 'git', 'save_request_date': '2021-08-10T09:31:51.290000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': '2021-08-10T09:31:59.346000+00:00', 'loading_task_id': 397492293}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://github.com/CocoaPods/Specs'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 397492293,
 'next_run': datetime.datetime(2021, 8, 10, 15, 1, 10, 715052, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 0,
 'status': 'disabled',
 'type': 'load-git'}
{'backend_id': '1a3f1b06-21cd-428b-acd9-45cf9f8c7e36',
 'ended': datetime.datetime(2021, 8, 10, 10, 13, 9, 354302, tzinfo=datetime.timezone.utc),
 'id': 1420171853,
 'metadata': {'worker': 'loader_high_priority@worker10.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 8, 10, 9, 31, 59, 315942, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 8, 10, 9, 31, 59, 340566, tzinfo=datetime.timezone.utc),
 'status': 'failed',
 'task': 397492293}

<SaveOriginRequest: {'id': 88681, 'origin_url': 'https://github.com/y20k/transistor', 'visit_type': 'git', 'save_request_date': '2021-08-09T08:11:51.846000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': '2021-08-09T08:11:52.233000+00:00', 'loading_task_id': 397470397}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://github.com/y20k/transistor'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 397470397,
 'next_run': datetime.datetime(2021, 8, 9, 8, 11, 51, 823350, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 3,
 'status': 'next_run_scheduled',
 'type': 'load-git'}
{'backend_id': '0e0d4b91-a5ae-41d7-a2cc-b61db12ff335',
 'ended': None,
 'id': 1420145244,
 'metadata': None,
 'scheduled': datetime.datetime(2021, 8, 9, 8, 11, 52, 167431, tzinfo=datetime.timezone.utc),
 'started': None,
 'status': 'scheduled',
 'task': 397470397}

<SaveOriginRequest: {'id': 88680, 'origin_url': 'https://github.com/TheLastProject/Catima', 'visit_type': 'git', 'save_request_date': '2021-08-09T08:11:43.631000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': '2021-08-09T08:11:52.192000+00:00', 'loading_task_id': 397470396}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://github.com/TheLastProject/Catima'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 397470396,
 'next_run': datetime.datetime(2021, 8, 9, 8, 11, 43, 585929, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 3,
 'status': 'next_run_scheduled',
 'type': 'load-git'}
{'backend_id': 'df168231-97d1-4435-b9d3-954304d47828',
 'ended': None,
 'id': 1420145243,
 'metadata': {'worker': 'loader_high_priority@worker11.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 8, 9, 8, 11, 52, 167411, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 8, 9, 8, 11, 52, 204201, tzinfo=datetime.timezone.utc),
 'status': 'started',
 'task': 397470396}

<SaveOriginRequest: {'id': 87544, 'origin_url': 'https://android.googlesource.com/platform/system/core', 'visit_type': 'git', 'save_request_date': '2021-07-26T13:34:16.987000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': '2021-07-26T13:34:25.182000+00:00', 'loading_task_id': 396985908}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://android.googlesource.com/platform/system/core'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 396985908,
 'next_run': datetime.datetime(2021, 8, 3, 16, 15, 35, 306511, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 2,
 'status': 'completed',
 'type': 'load-git'}
{'backend_id': 'bf89d8df-209d-4cd3-acaf-1f85dc081830',
 'ended': datetime.datetime(2021, 7, 26, 13, 38, 53, 217421, tzinfo=datetime.timezone.utc),
 'id': 1419432281,
 'metadata': {'worker': 'loader_high_priority@worker16.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 7, 26, 13, 34, 25, 144379, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 7, 26, 13, 34, 25, 174690, tzinfo=datetime.timezone.utc),
 'status': 'failed',
 'task': 396985908}

<SaveOriginRequest: {'id': 87538, 'origin_url': 'https://svn.r-project.org/R-dev-web/trunk/', 'visit_type': 'svn', 'save_request_date': '2021-07-26T12:59:50.038000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': '2021-07-26T12:59:52.299000+00:00', 'loading_task_id': 396985198}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://svn.r-project.org/R-dev-web/trunk/'}},
 'current_interval': datetime.timedelta(days=1),
 'id': 396985198,
 'next_run': datetime.datetime(2021, 8, 3, 16, 15, 35, 306511, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 2,
 'status': 'completed',
 'type': 'load-svn'}
{'backend_id': 'afd1f724-faa4-4b7b-ab8b-60e64feff3c7',
 'ended': datetime.datetime(2021, 7, 26, 13, 38, 22, 325391, tzinfo=datetime.timezone.utc),
 'id': 1419432271,
 'metadata': {'worker': 'loader_high_priority@worker14.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 7, 26, 12, 59, 52, 9739, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 7, 26, 12, 59, 52, 152890, tzinfo=datetime.timezone.utc),
 'status': 'failed',
 'task': 396985198}

<SaveOriginRequest: {'id': 87536, 'origin_url': 'svn://svn.code.sf.net/p/codeblocks/code/trunk', 'visit_type': 'svn', 'save_request_date': '2021-07-26T12:54:25.784000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': '2021-07-26T12:54:31.857000+00:00', 'loading_task_id': 396984987}>
{'arguments': {'args': [],
               'kwargs': {'url': 'svn://svn.code.sf.net/p/codeblocks/code/trunk'}},
 'current_interval': datetime.timedelta(days=1),
 'id': 396984987,
 'next_run': datetime.datetime(2021, 8, 3, 19, 5, 9, 45631, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 0,
 'status': 'disabled',
 'type': 'load-svn'}
{'backend_id': '240683e6-a753-48e5-8362-f2f5804acee0',
 'ended': datetime.datetime(2021, 7, 26, 13, 38, 26, 1067, tzinfo=datetime.timezone.utc),
 'id': 1419432270,
 'metadata': {'worker': 'loader_high_priority@worker15.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 7, 26, 12, 54, 31, 529652, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 7, 26, 12, 54, 31, 565497, tzinfo=datetime.timezone.utc),
 'status': 'failed',
 'task': 396984987}

<SaveOriginRequest: {'id': 86401, 'origin_url': 'https://github.com/keybase/client', 'visit_type': 'git', 'save_request_date': '2021-07-17T22:43:29.870000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': '2021-07-17T22:43:35.725000+00:00', 'loading_task_id': 396435047}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://github.com/keybase/client'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 396435047,
 'next_run': datetime.datetime(2021, 8, 3, 17, 22, 21, 784019, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 2,
 'status': 'next_run_scheduled',
 'type': 'load-git'}
{'backend_id': 'c71b79dc-dadb-4edb-beaf-845b23b915bd',
 'ended': None,
 'id': 1419394665,
 'metadata': None,
 'scheduled': datetime.datetime(2021, 7, 17, 22, 43, 34, 594211, tzinfo=datetime.timezone.utc),
 'started': None,
 'status': 'scheduled',
 'task': 396435047}

<SaveOriginRequest: {'id': 84583, 'origin_url': 'git://git.archlinux.org/svntogit/packages.git', 'visit_type': 'git', 'save_request_date': '2021-06-29T17:08:37.262000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': '2021-06-29T17:08:42.461000+00:00', 'loading_task_id': 381575775}>
{'arguments': {'args': [],
               'kwargs': {'url': 'git://git.archlinux.org/svntogit/packages.git'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 381575775,
 'next_run': datetime.datetime(2021, 6, 29, 19, 16, 33, 490182, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 3,
 'status': 'completed',
 'type': 'load-git'}
{'backend_id': '63b8a186-dae4-4078-a5f8-a94eb849da26',
 'ended': None,
 'id': 1419224462,
 'metadata': {'worker': 'loader_high_priority@worker11.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 6, 29, 17, 8, 42, 398128, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 6, 29, 17, 8, 42, 495148, tzinfo=datetime.timezone.utc),
 'status': 'started',
 'task': 381575775}

<SaveOriginRequest: {'id': 84095, 'origin_url': 'https://anonhg.netbsd.org/pkgsrc-public/', 'visit_type': 'hg', 'save_request_date': '2021-06-24T18:56:00.441000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': '2021-06-28T09:06:30.064000+00:00', 'loading_task_id': 381575556}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://anonhg.netbsd.org/pkgsrc-public/'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 381575556,
 'next_run': datetime.datetime(2021, 9, 4, 5, 31, 27, 427868, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 2,
 'status': 'next_run_scheduled',
 'type': 'load-hg'}
{'backend_id': '22111e49-08ec-48ab-904f-f225893ab3d0',
 'ended': None,
 'id': 1418915294,
 'metadata': None,
 'scheduled': datetime.datetime(2021, 6, 28, 9, 6, 29, 728178, tzinfo=datetime.timezone.utc),
 'started': None,
 'status': 'scheduled',
 'task': 381575556}

<SaveOriginRequest: {'id': 84094, 'origin_url': 'https://anonhg.netbsd.org/pkgsrc-draft/', 'visit_type': 'hg', 'save_request_date': '2021-06-24T18:55:55.386000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': '2021-06-28T09:06:29.908000+00:00', 'loading_task_id': 381575557}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://anonhg.netbsd.org/pkgsrc-draft/'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 381575557,
 'next_run': datetime.datetime(2021, 9, 2, 20, 29, 56, 629485, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 0,
 'status': 'disabled',
 'type': 'load-hg'}
{'backend_id': '8cb08c19-1401-44a1-aeff-fbb7a368c4ed',
 'ended': None,
 'id': 1418915295,
 'metadata': None,
 'scheduled': datetime.datetime(2021, 6, 28, 9, 6, 29, 728184, tzinfo=datetime.timezone.utc),
 'started': None,
 'status': 'scheduled',
 'task': 381575557}

<SaveOriginRequest: {'id': 84093, 'origin_url': 'https://anonhg.netbsd.org/pkgsrc/', 'visit_type': 'hg', 'save_request_date': '2021-06-24T18:55:51.537000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': '2021-06-28T09:06:30.028000+00:00', 'loading_task_id': 381575558}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://anonhg.netbsd.org/pkgsrc/'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 381575558,
 'next_run': datetime.datetime(2021, 9, 6, 3, 6, 12, 88036, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 0,
 'status': 'next_run_scheduled',
 'type': 'load-hg'}
{'backend_id': '47e11e34-fbe7-456a-97c5-debd6f7d4df6',
 'ended': None,
 'id': 1418915296,
 'metadata': None,
 'scheduled': datetime.datetime(2021, 6, 28, 9, 6, 29, 728189, tzinfo=datetime.timezone.utc),
 'started': None,
 'status': 'scheduled',
 'task': 381575558}

<SaveOriginRequest: {'id': 84091, 'origin_url': 'https://anonhg.netbsd.org/src-public/', 'visit_type': 'hg', 'save_request_date': '2021-06-24T18:55:41.589000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': '2021-06-28T09:06:30.060000+00:00', 'loading_task_id': 381575560}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://anonhg.netbsd.org/src-public/'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 381575560,
 'next_run': datetime.datetime(2021, 9, 2, 18, 0, 51, 767822, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 2,
 'status': 'next_run_scheduled',
 'type': 'load-hg'}
{'backend_id': '82de4ed7-25a8-4258-a926-0afd8e7fd20c',
 'ended': None,
 'id': 1418915298,
 'metadata': None,
 'scheduled': datetime.datetime(2021, 6, 28, 9, 6, 29, 728199, tzinfo=datetime.timezone.utc),
 'started': None,
 'status': 'scheduled',
 'task': 381575560}

<SaveOriginRequest: {'id': 84090, 'origin_url': 'https://anonhg.netbsd.org/src-draft/', 'visit_type': 'hg', 'save_request_date': '2021-06-24T18:55:35.637000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': '2021-06-28T09:06:30.003000+00:00', 'loading_task_id': 381575561}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://anonhg.netbsd.org/src-draft/'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 381575561,
 'next_run': datetime.datetime(2021, 9, 5, 23, 1, 12, 801373, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 0,
 'status': 'next_run_scheduled',
 'type': 'load-hg'}
{'backend_id': 'ac424ab5-9ecd-4dd9-87cd-452bbd39170b',
 'ended': None,
 'id': 1418915299,
 'metadata': None,
 'scheduled': datetime.datetime(2021, 6, 28, 9, 6, 29, 728204, tzinfo=datetime.timezone.utc),
 'started': None,
 'status': 'scheduled',
 'task': 381575561}

<SaveOriginRequest: {'id': 84080, 'origin_url': 'git://git.archlinux.org/svntogit/community.git', 'visit_type': 'git', 'save_request_date': '2021-06-24T16:53:31.099000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': '2021-06-24T16:53:34.576000+00:00', 'loading_task_id': 381575242}>
{'arguments': {'args': [],
               'kwargs': {'url': 'git://git.archlinux.org/svntogit/community.git'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 381575242,
 'next_run': datetime.datetime(2021, 6, 29, 19, 15, 49, 417085, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 3,
 'status': 'completed',
 'type': 'load-git'}
{'backend_id': 'a21a7341-d924-43e6-927d-839f547fc89d',
 'ended': None,
 'id': 1418855740,
 'metadata': None,
 'scheduled': datetime.datetime(2021, 6, 24, 16, 53, 34, 383036, tzinfo=datetime.timezone.utc),
 'started': None,
 'status': 'scheduled',
 'task': 381575242}

<SaveOriginRequest: {'id': 83883, 'origin_url': 'https://github.com/jlippold/tweakCompatible', 'visit_type': 'git', 'save_request_date': '2021-06-22T21:56:29.522000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': '2021-06-22T21:56:32.973000+00:00', 'loading_task_id': 381575019}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://github.com/jlippold/tweakCompatible'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 381575019,
 'next_run': datetime.datetime(2021, 9, 2, 20, 50, 39, 890407, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 0,
 'status': 'disabled',
 'type': 'load-git'}
{'backend_id': '9dfef3fd-7514-47ad-8209-e2c8a7219c4b',
 'ended': None,
 'id': 1418844860,
 'metadata': None,
 'scheduled': datetime.datetime(2021, 6, 22, 21, 56, 32, 898764, tzinfo=datetime.timezone.utc),
 'started': None,
 'status': 'scheduled',
 'task': 381575019}

<SaveOriginRequest: {'id': 79867, 'origin_url': 'https://git.savannah.gnu.org/git/gnulib.git', 'visit_type': 'git', 'save_request_date': '2021-05-23T12:18:02.213000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': None, 'loading_task_id': 381570771}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://git.savannah.gnu.org/git/gnulib.git'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 381570771,
 'next_run': datetime.datetime(2021, 9, 2, 19, 3, 52, 236645, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 2,
 'status': 'completed',
 'type': 'load-git'}
{'backend_id': 'a17f4353-1170-48cf-b388-2cba67ae435e',
 'ended': None,
 'id': 1416730029,
 'metadata': None,
 'scheduled': datetime.datetime(2021, 5, 23, 12, 22, 2, 875906, tzinfo=datetime.timezone.utc),
 'started': None,
 'status': 'scheduled',
 'task': 381570771}

<SaveOriginRequest: {'id': 78750, 'origin_url': 'https://git.libreoffice.org/translations/', 'visit_type': 'git', 'save_request_date': '2021-05-15T06:09:05.969000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': None, 'loading_task_id': 381569600}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://git.libreoffice.org/translations/'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 381569600,
 'next_run': datetime.datetime(2021, 9, 2, 19, 13, 6, 171503, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 2,
 'status': 'next_run_scheduled',
 'type': 'load-git'}
{'backend_id': '1a8f722b-45c8-4467-8228-44b0ad513b5d',
 'ended': None,
 'id': 1411999145,
 'metadata': {'worker': 'loader_high_priority@worker01.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 5, 15, 6, 10, 32, 99965, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 5, 15, 7, 54, 2, 782756, tzinfo=datetime.timezone.utc),
 'status': 'started',
 'task': 381569600}

<SaveOriginRequest: {'id': 75668, 'origin_url': 'https://github.com/unitystation/unitystation', 'visit_type': 'git', 'save_request_date': '2021-04-20T11:40:21.612000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': None, 'visit_date': None, 'loading_task_id': 380617630}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://github.com/unitystation/unitystation'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 380617630,
 'next_run': datetime.datetime(2021, 9, 2, 21, 5, 13, 500429, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 0,
 'status': 'disabled',
 'type': 'load-git'}
{'backend_id': '952b3637-ff95-4be2-9e23-29689f7b2f24',
 'ended': None,
 'id': 1385436041,
 'metadata': {'worker': 'loader_high_priority@worker05.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 4, 20, 11, 40, 24, 478619, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 4, 20, 12, 3, 36, 665888, tzinfo=datetime.timezone.utc),
 'status': 'started',
 'task': 380617630}

<SaveOriginRequest: {'id': 74433, 'origin_url': 'https://github.com/coreutils/gnulib', 'visit_type': 'git', 'save_request_date': '2021-04-20T08:52:13.105000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': None, 'visit_date': None, 'loading_task_id': 380606658}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://github.com/coreutils/gnulib'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 380606658,
 'next_run': datetime.datetime(2021, 9, 2, 20, 22, 29, 459136, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 0,
 'status': 'next_run_scheduled',
 'type': 'load-git'}
{'backend_id': '3be46e59-2cb2-4319-b85e-0dbc3abf17f9',
 'ended': None,
 'id': 1385261535,
 'metadata': {'worker': 'loader_high_priority@worker09.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 4, 20, 8, 52, 16, 886847, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 4, 20, 8, 53, 59, 286505, tzinfo=datetime.timezone.utc),
 'status': 'started',
 'task': 380606658}

<SaveOriginRequest: {'id': 71357, 'origin_url': 'https://scm.gforge.inria.fr/anonscm/git/simty/simty.git', 'visit_type': 'git', 'save_request_date': '2021-04-01T16:57:50.205000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': None, 'visit_date': None, 'loading_task_id': 379069991}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://scm.gforge.inria.fr/anonscm/git/simty/simty.git'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 379069991,
 'next_run': datetime.datetime(2021, 9, 2, 21, 18, 44, 983099, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 0,
 'status': 'disabled',
 'type': 'load-git'}
{'backend_id': '10538c74-07ab-48d7-9b8c-8d7354dbf471',
 'ended': None,
 'id': 1357070596,
 'metadata': {'worker': 'loader_git@worker07.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 4, 1, 17, 53, 56, 874230, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 4, 1, 23, 46, 57, 829277, tzinfo=datetime.timezone.utc),
 'status': 'started',
 'task': 379069991}

<SaveOriginRequest: {'id': 70508, 'origin_url': 'https://git.php.net/repository/web/php-distributions.git', 'visit_type': 'git', 'save_request_date': '2021-03-30T09:45:14.138000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': None, 'visit_date': None, 'loading_task_id': 378877888}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://git.php.net/repository/web/php-distributions.git'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 378877888,
 'next_run': datetime.datetime(2021, 9, 2, 22, 33, 45, 640445, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 0,
 'status': 'disabled',
 'type': 'load-git'}
{'backend_id': 'a423ccd1-572a-4fb6-b373-7ebfcee6a9d4',
 'ended': None,
 'id': 1353606625,
 'metadata': {'worker': 'loader_git@worker06.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 3, 30, 10, 47, 41, 37168, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 3, 30, 14, 13, 23, 269112, tzinfo=datetime.timezone.utc),
 'status': 'started',
 'task': 378877888}

<SaveOriginRequest: {'id': 69864, 'origin_url': 'https://github.com/kusl/wgeteveryday', 'visit_type': 'git', 'save_request_date': '2021-03-22T07:03:38.046000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': None, 'visit_date': None, 'loading_task_id': 378218248}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://github.com/kusl/wgeteveryday'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 378218248,
 'next_run': datetime.datetime(2021, 9, 3, 0, 57, 10, 295640, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 0,
 'status': 'disabled',
 'type': 'load-git'}
{'backend_id': '4f354c97-938b-4376-bddb-5569353412b0',
 'ended': None,
 'id': 1341798458,
 'metadata': {'worker': 'loader_git@worker07.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 3, 22, 8, 43, 57, 547826, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 3, 22, 12, 47, 27, 684610, tzinfo=datetime.timezone.utc),
 'status': 'started',
 'task': 378218248}

<SaveOriginRequest: {'id': 6930, 'origin_url': 'https://github.com/CambridgeSemiticsLab/BH_time_collocations', 'visit_type': 'git', 'save_request_date': '2020-01-24T00:29:45.320000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': None, 'visit_date': '2020-01-24T00:35:04.555000+00:00', 'loading_task_id': 269843604}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://github.com/CambridgeSemiticsLab/BH_time_collocations'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 269843604,
 'next_run': datetime.datetime(2021, 4, 20, 7, 20, 52, 350358, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 0,
 'status': 'completed',
 'type': 'load-git'}
{'backend_id': '36f7efb9-b342-493d-8bf5-6f765c5c135e',
 'ended': datetime.datetime(2020, 1, 24, 2, 24, 38, 697068, tzinfo=datetime.timezone.utc),
 'id': 789488522,
 'metadata': {'worker': 'celery@loader_git.worker03'},
 'scheduled': datetime.datetime(2020, 1, 24, 0, 29, 57, 694726, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2020, 1, 24, 2, 24, 40, 987615, tzinfo=datetime.timezone.utc),
 'status': 'started',
 'task': 269843604}

      Below are the save request states once they have been processed, some of them got their statuses updated to failed or scheduled but most of them are still in running state.

      [{'id': 90358,
  'loading_task_id': 398327718,
  'origin_url': 'https://github.com/mozilla/gecko-dev',
  'save_request_date': '2021-08-27T18:05:28.837000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'running',
  'visit_date': '2021-08-27T18:05:31.152715+00:00',
  'visit_status': 'created',
  'visit_type': 'git'},
 {'id': 89569,
  'loading_task_id': 398036169,
  'origin_url': 'https://github.com/404-not-find/client',
  'save_request_date': '2021-08-21T08:06:47.263000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'running',
  'visit_date': '2021-08-21T08:06:48.337363+00:00',
  'visit_status': 'created',
  'visit_type': 'git'},
 {'id': 88780,
  'loading_task_id': 397492293,
  'origin_url': 'https://github.com/CocoaPods/Specs',
  'save_request_date': '2021-08-10T09:31:51.290000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'running',
  'visit_date': '2021-08-10T09:31:59.346891+00:00',
  'visit_status': 'created',
  'visit_type': 'git'},
 {'id': 88681,
  'loading_task_id': 397470397,
  'origin_url': 'https://github.com/y20k/transistor',
  'save_request_date': '2021-08-09T08:11:51.846000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'running',
  'visit_date': '2021-08-09T08:11:52.233412+00:00',
  'visit_status': 'created',
  'visit_type': 'git'},
 {'id': 88680,
  'loading_task_id': 397470396,
  'origin_url': 'https://github.com/TheLastProject/Catima',
  'save_request_date': '2021-08-09T08:11:43.631000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'running',
  'visit_date': '2021-08-09T08:11:52.192663+00:00',
  'visit_status': 'created',
  'visit_type': 'git'},
 {'id': 87544,
  'loading_task_id': 396985908,
  'origin_url': 'https://android.googlesource.com/platform/system/core',
  'save_request_date': '2021-07-26T13:34:16.987000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'failed',
  'visit_date': None,
  'visit_status': None,
  'visit_type': 'git'},
 {'id': 87538,
  'loading_task_id': 396985198,
  'origin_url': 'https://svn.r-project.org/R-dev-web/trunk/',
  'save_request_date': '2021-07-26T12:59:50.038000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'failed',
  'visit_date': None,
  'visit_status': None,
  'visit_type': 'svn'},
 {'id': 87536,
  'loading_task_id': 396984987,
  'origin_url': 'svn://svn.code.sf.net/p/codeblocks/code/trunk',
  'save_request_date': '2021-07-26T12:54:25.784000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'failed',
  'visit_date': None,
  'visit_status': None,
  'visit_type': 'svn'},
 {'id': 86401,
  'loading_task_id': 396435047,
  'origin_url': 'https://github.com/keybase/client',
  'save_request_date': '2021-07-17T22:43:29.870000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'scheduled',
  'visit_date': None,
  'visit_status': None,
  'visit_type': 'git'},
 {'id': 84583,
  'loading_task_id': 381575775,
  'origin_url': 'git://git.archlinux.org/svntogit/packages.git',
  'save_request_date': '2021-06-29T17:08:37.262000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'running',
  'visit_date': None,
  'visit_status': None,
  'visit_type': 'git'},
 {'id': 84095,
  'loading_task_id': 381575556,
  'origin_url': 'https://anonhg.netbsd.org/pkgsrc-public/',
  'save_request_date': '2021-06-24T18:56:00.441000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'scheduled',
  'visit_date': None,
  'visit_status': None,
  'visit_type': 'hg'},
 {'id': 84094,
  'loading_task_id': 381575557,
  'origin_url': 'https://anonhg.netbsd.org/pkgsrc-draft/',
  'save_request_date': '2021-06-24T18:55:55.386000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'scheduled',
  'visit_date': None,
  'visit_status': None,
  'visit_type': 'hg'},
 {'id': 84093,
  'loading_task_id': 381575558,
  'origin_url': 'https://anonhg.netbsd.org/pkgsrc/',
  'save_request_date': '2021-06-24T18:55:51.537000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'scheduled',
  'visit_date': None,
  'visit_status': None,
  'visit_type': 'hg'},
 {'id': 84091,
  'loading_task_id': 381575560,
  'origin_url': 'https://anonhg.netbsd.org/src-public/',
  'save_request_date': '2021-06-24T18:55:41.589000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'scheduled',
  'visit_date': None,
  'visit_status': None,
  'visit_type': 'hg'},
 {'id': 84090,
  'loading_task_id': 381575561,
  'origin_url': 'https://anonhg.netbsd.org/src-draft/',
  'save_request_date': '2021-06-24T18:55:35.637000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'scheduled',
  'visit_date': None,
  'visit_status': None,
  'visit_type': 'hg'},
 {'id': 84080,
  'loading_task_id': 381575242,
  'origin_url': 'git://git.archlinux.org/svntogit/community.git',
  'save_request_date': '2021-06-24T16:53:31.099000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'scheduled',
  'visit_date': None,
  'visit_status': None,
  'visit_type': 'git'},
 {'id': 83883,
  'loading_task_id': 381575019,
  'origin_url': 'https://github.com/jlippold/tweakCompatible',
  'save_request_date': '2021-06-22T21:56:29.522000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'scheduled',
  'visit_date': None,
  'visit_status': None,
  'visit_type': 'git'},
 {'id': 79867,
  'loading_task_id': 381570771,
  'origin_url': 'https://git.savannah.gnu.org/git/gnulib.git',
  'save_request_date': '2021-05-23T12:18:02.213000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'scheduled',
  'visit_date': None,
  'visit_status': None,
  'visit_type': 'git'},
 {'id': 78750,
  'loading_task_id': 381569600,
  'origin_url': 'https://git.libreoffice.org/translations/',
  'save_request_date': '2021-05-15T06:09:05.969000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'running',
  'visit_date': None,
  'visit_status': None,
  'visit_type': 'git'},
 {'id': 75668,
  'loading_task_id': 380617630,
  'origin_url': 'https://github.com/unitystation/unitystation',
  'save_request_date': '2021-04-20T11:40:21.612000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'running',
  'visit_date': None,
  'visit_status': None,
  'visit_type': 'git'},
 {'id': 74433,
  'loading_task_id': 380606658,
  'origin_url': 'https://github.com/coreutils/gnulib',
  'save_request_date': '2021-04-20T08:52:13.105000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'running',
  'visit_date': None,
  'visit_status': None,
  'visit_type': 'git'},
 {'id': 71357,
  'loading_task_id': 379069991,
  'origin_url': 'https://scm.gforge.inria.fr/anonscm/git/simty/simty.git',
  'save_request_date': '2021-04-01T16:57:50.205000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'running',
  'visit_date': None,
  'visit_status': None,
  'visit_type': 'git'},
 {'id': 70508,
  'loading_task_id': 378877888,
  'origin_url': 'https://git.php.net/repository/web/php-distributions.git',
  'save_request_date': '2021-03-30T09:45:14.138000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'running',
  'visit_date': None,
  'visit_status': None,
  'visit_type': 'git'},
 {'id': 69864,
  'loading_task_id': 378218248,
  'origin_url': 'https://github.com/kusl/wgeteveryday',
  'save_request_date': '2021-03-22T07:03:38.046000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'running',
  'visit_date': None,
  'visit_status': None,
  'visit_type': 'git'},
 {'id': 6930,
  'loading_task_id': 269843604,
  'origin_url': 'https://github.com/CambridgeSemiticsLab/BH_time_collocations',
  'save_request_date': '2020-01-24T00:29:45.320000+00:00',
  'save_request_status': 'accepted',
  'save_task_status': 'running',
  'visit_date': None,
  'visit_status': None,
  'visit_type': 'git'}]

      Below are the state of those save requests still marked as running when I relaunch the refresh_savecodenow_statuses django command.

      <SaveOriginRequest: {'id': 90358, 'origin_url': 'https://github.com/mozilla/gecko-dev', 'visit_type': 'git', 'save_request_date': '2021-08-27T18:05:28.837000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': '2021-08-27T18:05:31.152715+00:00', 'loading_task_id': 398327718}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://github.com/mozilla/gecko-dev'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 398327718,
 'next_run': datetime.datetime(2021, 8, 27, 20, 48, 31, 653577, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 2,
 'status': 'completed',
 'type': 'load-git'}
{'backend_id': '7d17cf6d-1de5-43e7-af25-da5c96b19f3d',
 'ended': datetime.datetime(2021, 8, 27, 19, 48, 31, 653577, tzinfo=datetime.timezone.utc),
 'id': 1422313746,
 'metadata': {'worker': 'loader_high_priority@worker01.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 8, 27, 18, 5, 31, 125969, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 8, 27, 18, 5, 31, 151958, tzinfo=datetime.timezone.utc),
 'status': 'failed',
 'task': 398327718}

<SaveOriginRequest: {'id': 89569, 'origin_url': 'https://github.com/404-not-find/client', 'visit_type': 'git', 'save_request_date': '2021-08-21T08:06:47.263000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': '2021-08-21T08:06:48.337363+00:00', 'loading_task_id': 398036169}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://github.com/404-not-find/client'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 398036169,
 'next_run': datetime.datetime(2021, 8, 21, 8, 6, 47, 232838, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 3,
 'status': 'next_run_scheduled',
 'type': 'load-git'}
{'backend_id': 'd8c57ab6-01ed-43f2-aed0-644c7e40288b',
 'ended': None,
 'id': 1421317589,
 'metadata': None,
 'scheduled': datetime.datetime(2021, 8, 21, 8, 6, 48, 309219, tzinfo=datetime.timezone.utc),
 'started': None,
 'status': 'scheduled',
 'task': 398036169}

<SaveOriginRequest: {'id': 88780, 'origin_url': 'https://github.com/CocoaPods/Specs', 'visit_type': 'git', 'save_request_date': '2021-08-10T09:31:51.290000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': '2021-08-10T09:31:59.346891+00:00', 'loading_task_id': 397492293}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://github.com/CocoaPods/Specs'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 397492293,
 'next_run': datetime.datetime(2021, 8, 10, 15, 1, 10, 715052, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 0,
 'status': 'disabled',
 'type': 'load-git'}
{'backend_id': '1a3f1b06-21cd-428b-acd9-45cf9f8c7e36',
 'ended': datetime.datetime(2021, 8, 10, 10, 13, 9, 354302, tzinfo=datetime.timezone.utc),
 'id': 1420171853,
 'metadata': {'worker': 'loader_high_priority@worker10.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 8, 10, 9, 31, 59, 315942, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 8, 10, 9, 31, 59, 340566, tzinfo=datetime.timezone.utc),
 'status': 'failed',
 'task': 397492293}

<SaveOriginRequest: {'id': 88681, 'origin_url': 'https://github.com/y20k/transistor', 'visit_type': 'git', 'save_request_date': '2021-08-09T08:11:51.846000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': '2021-08-09T08:11:52.233412+00:00', 'loading_task_id': 397470397}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://github.com/y20k/transistor'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 397470397,
 'next_run': datetime.datetime(2021, 8, 9, 8, 11, 51, 823350, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 3,
 'status': 'next_run_scheduled',
 'type': 'load-git'}
{'backend_id': '0e0d4b91-a5ae-41d7-a2cc-b61db12ff335',
 'ended': None,
 'id': 1420145244,
 'metadata': None,
 'scheduled': datetime.datetime(2021, 8, 9, 8, 11, 52, 167431, tzinfo=datetime.timezone.utc),
 'started': None,
 'status': 'scheduled',
 'task': 397470397}

<SaveOriginRequest: {'id': 88680, 'origin_url': 'https://github.com/TheLastProject/Catima', 'visit_type': 'git', 'save_request_date': '2021-08-09T08:11:43.631000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': 'created', 'visit_date': '2021-08-09T08:11:52.192663+00:00', 'loading_task_id': 397470396}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://github.com/TheLastProject/Catima'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 397470396,
 'next_run': datetime.datetime(2021, 8, 9, 8, 11, 43, 585929, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 3,
 'status': 'next_run_scheduled',
 'type': 'load-git'}
{'backend_id': 'df168231-97d1-4435-b9d3-954304d47828',
 'ended': None,
 'id': 1420145243,
 'metadata': {'worker': 'loader_high_priority@worker11.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 8, 9, 8, 11, 52, 167411, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 8, 9, 8, 11, 52, 204201, tzinfo=datetime.timezone.utc),
 'status': 'started',
 'task': 397470396}

<SaveOriginRequest: {'id': 84583, 'origin_url': 'git://git.archlinux.org/svntogit/packages.git', 'visit_type': 'git', 'save_request_date': '2021-06-29T17:08:37.262000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': None, 'visit_date': None, 'loading_task_id': 381575775}>
{'arguments': {'args': [],
               'kwargs': {'url': 'git://git.archlinux.org/svntogit/packages.git'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 381575775,
 'next_run': datetime.datetime(2021, 6, 29, 19, 16, 33, 490182, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 3,
 'status': 'completed',
 'type': 'load-git'}
{'backend_id': '63b8a186-dae4-4078-a5f8-a94eb849da26',
 'ended': None,
 'id': 1419224462,
 'metadata': {'worker': 'loader_high_priority@worker11.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 6, 29, 17, 8, 42, 398128, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 6, 29, 17, 8, 42, 495148, tzinfo=datetime.timezone.utc),
 'status': 'started',
 'task': 381575775}

<SaveOriginRequest: {'id': 78750, 'origin_url': 'https://git.libreoffice.org/translations/', 'visit_type': 'git', 'save_request_date': '2021-05-15T06:09:05.969000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': None, 'visit_date': None, 'loading_task_id': 381569600}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://git.libreoffice.org/translations/'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 381569600,
 'next_run': datetime.datetime(2021, 9, 2, 19, 13, 6, 171503, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 2,
 'status': 'next_run_scheduled',
 'type': 'load-git'}
{'backend_id': '1a8f722b-45c8-4467-8228-44b0ad513b5d',
 'ended': None,
 'id': 1411999145,
 'metadata': {'worker': 'loader_high_priority@worker01.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 5, 15, 6, 10, 32, 99965, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 5, 15, 7, 54, 2, 782756, tzinfo=datetime.timezone.utc),
 'status': 'started',
 'task': 381569600}

<SaveOriginRequest: {'id': 75668, 'origin_url': 'https://github.com/unitystation/unitystation', 'visit_type': 'git', 'save_request_date': '2021-04-20T11:40:21.612000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': None, 'visit_date': None, 'loading_task_id': 380617630}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://github.com/unitystation/unitystation'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 380617630,
 'next_run': datetime.datetime(2021, 9, 2, 21, 5, 13, 500429, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 0,
 'status': 'disabled',
 'type': 'load-git'}
{'backend_id': '952b3637-ff95-4be2-9e23-29689f7b2f24',
 'ended': None,
 'id': 1385436041,
 'metadata': {'worker': 'loader_high_priority@worker05.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 4, 20, 11, 40, 24, 478619, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 4, 20, 12, 3, 36, 665888, tzinfo=datetime.timezone.utc),
 'status': 'started',
 'task': 380617630}

<SaveOriginRequest: {'id': 74433, 'origin_url': 'https://github.com/coreutils/gnulib', 'visit_type': 'git', 'save_request_date': '2021-04-20T08:52:13.105000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': None, 'visit_date': None, 'loading_task_id': 380606658}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://github.com/coreutils/gnulib'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 380606658,
 'next_run': datetime.datetime(2021, 9, 2, 20, 22, 29, 459136, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 0,
 'status': 'next_run_scheduled',
 'type': 'load-git'}
{'backend_id': '3be46e59-2cb2-4319-b85e-0dbc3abf17f9',
 'ended': None,
 'id': 1385261535,
 'metadata': {'worker': 'loader_high_priority@worker09.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 4, 20, 8, 52, 16, 886847, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 4, 20, 8, 53, 59, 286505, tzinfo=datetime.timezone.utc),
 'status': 'started',
 'task': 380606658}

<SaveOriginRequest: {'id': 71357, 'origin_url': 'https://scm.gforge.inria.fr/anonscm/git/simty/simty.git', 'visit_type': 'git', 'save_request_date': '2021-04-01T16:57:50.205000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': None, 'visit_date': None, 'loading_task_id': 379069991}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://scm.gforge.inria.fr/anonscm/git/simty/simty.git'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 379069991,
 'next_run': datetime.datetime(2021, 9, 2, 21, 18, 44, 983099, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 0,
 'status': 'disabled',
 'type': 'load-git'}
{'backend_id': '10538c74-07ab-48d7-9b8c-8d7354dbf471',
 'ended': None,
 'id': 1357070596,
 'metadata': {'worker': 'loader_git@worker07.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 4, 1, 17, 53, 56, 874230, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 4, 1, 23, 46, 57, 829277, tzinfo=datetime.timezone.utc),
 'status': 'started',
 'task': 379069991}

<SaveOriginRequest: {'id': 70508, 'origin_url': 'https://git.php.net/repository/web/php-distributions.git', 'visit_type': 'git', 'save_request_date': '2021-03-30T09:45:14.138000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': None, 'visit_date': None, 'loading_task_id': 378877888}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://git.php.net/repository/web/php-distributions.git'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 378877888,
 'next_run': datetime.datetime(2021, 9, 2, 22, 33, 45, 640445, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 0,
 'status': 'disabled',
 'type': 'load-git'}
{'backend_id': 'a423ccd1-572a-4fb6-b373-7ebfcee6a9d4',
 'ended': None,
 'id': 1353606625,
 'metadata': {'worker': 'loader_git@worker06.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 3, 30, 10, 47, 41, 37168, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 3, 30, 14, 13, 23, 269112, tzinfo=datetime.timezone.utc),
 'status': 'started',
 'task': 378877888}

<SaveOriginRequest: {'id': 69864, 'origin_url': 'https://github.com/kusl/wgeteveryday', 'visit_type': 'git', 'save_request_date': '2021-03-22T07:03:38.046000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': None, 'visit_date': None, 'loading_task_id': 378218248}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://github.com/kusl/wgeteveryday'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 378218248,
 'next_run': datetime.datetime(2021, 9, 3, 0, 57, 10, 295640, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 0,
 'status': 'disabled',
 'type': 'load-git'}
{'backend_id': '4f354c97-938b-4376-bddb-5569353412b0',
 'ended': None,
 'id': 1341798458,
 'metadata': {'worker': 'loader_git@worker07.internal.softwareheritage.org'},
 'scheduled': datetime.datetime(2021, 3, 22, 8, 43, 57, 547826, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2021, 3, 22, 12, 47, 27, 684610, tzinfo=datetime.timezone.utc),
 'status': 'started',
 'task': 378218248}

<SaveOriginRequest: {'id': 6930, 'origin_url': 'https://github.com/CambridgeSemiticsLab/BH_time_collocations', 'visit_type': 'git', 'save_request_date': '2020-01-24T00:29:45.320000+00:00', 'save_request_status': 'accepted', 'save_task_status': 'running', 'visit_status': None, 'visit_date': None, 'loading_task_id': 269843604}>
{'arguments': {'args': [],
               'kwargs': {'url': 'https://github.com/CambridgeSemiticsLab/BH_time_collocations'}},
 'current_interval': datetime.timedelta(days=64),
 'id': 269843604,
 'next_run': datetime.datetime(2021, 4, 20, 7, 20, 52, 350358, tzinfo=datetime.timezone.utc),
 'policy': 'oneshot',
 'priority': 'high',
 'retries_left': 0,
 'status': 'completed',
 'type': 'load-git'}
{'backend_id': '36f7efb9-b342-493d-8bf5-6f765c5c135e',
 'ended': datetime.datetime(2020, 1, 24, 2, 24, 38, 697068, tzinfo=datetime.timezone.utc),
 'id': 789488522,
 'metadata': {'worker': 'celery@loader_git.worker03'},
 'scheduled': datetime.datetime(2020, 1, 24, 0, 29, 57, 694726, tzinfo=datetime.timezone.utc),
 'started': datetime.datetime(2020, 1, 24, 2, 24, 40, 987615, tzinfo=datetime.timezone.utc),
 'status': 'started',
 'task': 269843604}

      We can see that some save requests have failed but are not correctly reported, this needs to be fixed. Otherwise the remaining requests have their scheduler task marked as scheduled or started but their execution or termination date is missing so it seems something went wrong on the scheduler reporting side.

    • Phabricator Migration user marked this issue as related to swh/devel/swh-scheduler#3583 (closed)

      marked this issue as related to swh/devel/swh-scheduler#3583 (closed)

    • Antoine R. Dumont
      Antoine R. Dumont @ardumont ·
      Author Maintainer

      very old comment not pushed for some reason

      heads up, worker17 (larger machine) did a pass on the origins with status issues. Most got ingested without issues [1].

      They could not be ingested normally by other workers due to missing either memory or disk space. Some other are failing for other reasons [2]

      • [1] (not exhaustive)
      INFO:swh.loader.git.loader.GitLoader:Listed 1241 refs for repo https://github.com/mozilla/gecko-dev {'status': 'eventful'}
real    523m7.053s
user    323m13.147s
sys     19m3.428s
INFO:swh.loader.git.loader.GitLoader:Listed 5 refs for repo https://github.com/globalbioticinteractions/jsonld-template-dataset {'status': 'uneventful'}
real    0m2.249s
user    0m0.986s
sys     0m0.143s
INFO:swh.loader.git.loader.GitLoader:Listed 352 refs for repo https://github.com/coreutils/gnulib {'status': 'eventful'}
real    38m1.260s
user    32m49.223s
sys     0m47.782s
INFO:swh.loader.git.loader.GitLoader:Listed 4567 refs for repo https://github.com/unitystation/unitystation {'status': 'eventful'}
real    108m20.898s
user    43m11.780s
sys     3m49.296s
INFO:swh.loader.git.loader.GitLoader:Listed 14031 refs for repo https://github.com/CocoaPods/Specs {'status': 'eventful'}
real    1428m25.351s
user    1274m9.850s
sys     22m53.308s
INFO:swh.loader.git.loader.GitLoader:Listed 19845 refs for repo https://github.com/keybase/client {'status': 'eventful'}
real    59m17.546s
user    47m59.797s
sys     3m12.392s
      • [2]
      INFO:swh.loader.git.loader.GitLoader:Load origin 'https://scm.gforge.inria.fr/anonscm/git/simty/simty.git' with type 'git'
...
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='scm.gforge.inria.fr', port=443): Max retries exceeded with url: /anonscm/git/simty/simty.git/info/refs?service=git-upload-pack (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0
x7f5c5f48b9e8>: Failed to establish a new connection: [Errno 111] Connection refused')) {'status': 'failed'}

real    0m1.671s
user    0m0.954s
sys     0m0.219s
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='scm.gforge.inria.fr', port=443): Max retries exceeded with url: /anonscm/git/simty/simty.git/info/refs?service=git-upload-pack (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0
x7f5c5f48b9e8>: Failed to establish a new connection: [Errno 111] Connection refused'))
{'status': 'failed'}

real    0m1.671s
user    0m0.954s
sys     0m0.219s

OSError: Pack file too big for repository https://git.libreoffice.org/translations/, limit is 4294967296 bytes, current size is 4294966867, would write 65515 {'status': 'failed'}
real    80m23.198s
user    2m12.409s
sys     1m26.351s
INFO:swh.loader.git.loader.GitLoader:Load origin 'git://git.archlinux.org/svntogit/packages.git' with type 'git'
ERROR:swh.loader.git.loader.GitLoader:Loading failure, updating to `failed` status
...
OSError: [Errno 101] Network is unreachable {'status': 'failed'}
real    0m3.037s
user    0m1.060s
sys     0m0.154s

INFO:swh.loader.git.loader.GitLoader:Load origin 'https://git.php.net/repository/web/php-distributions.git' with type 'git' <- This ones just hang in there indefinitely, try a git clone which does the same thing
    • Antoine R. Dumont changed the description

      changed the description

    Please register or sign in to reply