Skip to content

Only filters snapshot's revisions to retrieve artifact

Antoine R. Dumont requested to merge generated-differential-D727-source into generated-differential-D727-target

In current production, there are multiple issues about this:

Nov 28 13:02:40 worker15 python3[8850]: [2018-11-28 13:02:40,897: INFO/MainProcess] Received task: swh.loader.pypi.tasks.LoadPyPI[27a370b7-92b3-49ea-b307-4f1c5c9c8597]
Nov 28 13:02:41 worker15 python3[8883]: [2018-11-28 13:02:41,254: ERROR/Worker-10] Loading failure, updating to `partial` status
                                        Traceback (most recent call last):
                                          File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 886, in load
                                            self.prepare(*args, **kwargs)
                                          File "/usr/lib/python3/dist-packages/swh/loader/pypi/loader.py", line 147, in prepare
                                            self._prepare_state()
                                          File "/usr/lib/python3/dist-packages/swh/loader/pypi/loader.py", line 156, in _prepare_state
                                            self.known_artifacts = self._known_artifacts(last_snapshot)
                                          File "/usr/lib/python3/dist-packages/swh/loader/pypi/loader.py", line 117, in _known_artifacts
                                            if 'original_artifact' in revision['metadata']:
                                        TypeError: 'NoneType' object is not subscriptable

That's why the loader got a speed boost earlier this afternoon (T.T)

;)

Test Plan

tox

Migrated from D727 (view on Phabricator)

Merge request reports