save code now: Failure to ingest new 'archives' type when head response is incomplete
The following task got submitted but it's missing some mandatory information [1] Which is missing a timestamp (null in db) [2]
- [1]
Jun 07 10:21:27 worker10 python3[2047318]: [2021-06-07 10:21:27,717: INFO/MainProcess] Received task: swh.loader.package.archive.tasks.LoadArchive[f8644e5c-9149-4e60-a112-9e0c23c1f824]
Jun 07 10:21:32 worker10 python3[2047323]: [2021-06-07 10:21:32,126: ERROR/ForkPoolWorker-1] Failed loading branch releases/eqp-09d for https://www.cs.unm.edu/~mccune/old-ftp/
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/swh/loader/package/loader.py", line 576, in load
res = self._load_revision(p_info, origin)
File "/usr/lib/python3/dist-packages/swh/loader/package/loader.py", line 719, in _load_revision
p_info, uncompressed_path, directory=directory.hash
File "/usr/lib/python3/dist-packages/swh/loader/package/archive/loader.py", line 159, in build_revision
normalized_time = TimestampWithTimezone.from_datetime(parsed_time)
File "/usr/lib/python3/dist-packages/swh/model/model.py", line 258, in from_datetime
return cls.from_dict(dt)
File "/usr/lib/python3/dist-packages/swh/model/model.py", line 251, in from_dict
timestamp=Timestamp.from_dict(d["timestamp"]),
TypeError: 'NoneType' object is not subscriptable
Jun 07 10:21:34 worker10 python3[2047323]: [2021-06-07 10:21:34,227: ERROR/ForkPoolWorker-1] Failed loading branch releases/eqp-09e for https://www.cs.unm.edu/~mccune/old-ftp/
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/swh/loader/package/loader.py", line 576, in load
res = self._load_revision(p_info, origin)
File "/usr/lib/python3/dist-packages/swh/loader/package/loader.py", line 719, in _load_revision
p_info, uncompressed_path, directory=directory.hash
File "/usr/lib/python3/dist-packages/swh/loader/package/archive/loader.py", line 159, in build_revision
normalized_time = TimestampWithTimezone.from_datetime(parsed_time)
File "/usr/lib/python3/dist-packages/swh/model/model.py", line 258, in from_datetime
return cls.from_dict(dt)
File "/usr/lib/python3/dist-packages/swh/model/model.py", line 251, in from_dict
timestamp=Timestamp.from_dict(d["timestamp"]),
TypeError: 'NoneType' object is not subscriptable
Jun 07 10:21:34 worker10 python3[2047323]: [2021-06-07 10:21:34,273: WARNING/ForkPoolWorker-1] 2 failed branches
Jun 07 10:21:34 worker10 python3[2047323]: [2021-06-07 10:21:34,273: WARNING/ForkPoolWorker-1] Failed branches: releases/eqp-09d
Jun 07 10:21:34 worker10 python3[2047323]: [2021-06-07 10:21:34,273: WARNING/ForkPoolWorker-1] releases/eqp-09e- [2]
softwareheritage-scheduler=> select * from task where type like 'load-archive-files' and arguments#>>'{kwargs,url}' = 'https://www.cs.unm.edu/~mccune/old-ftp/';
+-[ RECORD 1 ]-----+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| id | 381572629 |
| type | load-archive-files |
| arguments | {"args": [], "kwargs": {"url": "https://www.cs.unm.edu/~mccune/old-ftp/", "artifacts": [{"url": "https://web.archive.org/web/20190318010151/https://www.cs.unm.edu/~mccune/old-ftp/eqp-09d.tar.gz", "time": null, "length": 0, "version": "eqp-09d"}, {"url": "https://web.archive.org/web/20190318010151/https://www.cs.unm.edu/~mccune/old-ftp/eqp-09e.tar.gz", "time": null, "length": 0, "version": "eqp-09e"}], "snapshot_append": true}} |
| next_run | 2021-06-07 10:21:19.069381+00 |
| current_interval | 1 day |
| status | completed |
| policy | oneshot |
| retries_left | 2 |
| priority | high |
+------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+Migrated from T3365 (view on Phabricator)
Edited by Phabricator Migration user