package/loader: Handle errors when retrieving package version info

Some package loaders might encounter errors while attempting to get package info for a given version (HTTP error for instance).

So handle that case and ensure partial visit when such an error occurs.

Related to T4124

Depends on !319 (closed)

---

Migrated from D8505 (view on Phabricator)

Build is green

Patch application report for D8505 (id=30628)

Could not rebase; Attempt merge onto 0c599bdc...

Updating 0c599bd..47c931d
Fast-forward
 swh/loader/package/loader.py                       |  61 +++++++---
 .../package_example_example-v1.0.tar.gz            | Bin 0 -> 181 bytes
 .../package_example_example-v2.0.tar.gz            | Bin 0 -> 181 bytes
 .../package_example_example-v3.0.tar.gz            | Bin 0 -> 181 bytes
 .../package_example_example-v4.0.tar.gz            | Bin 0 -> 182 bytes
 swh/loader/package/tests/test_loader.py            | 134 ++++++++++++++++++---
 6 files changed, 157 insertions(+), 38 deletions(-)
 create mode 100644 swh/loader/package/tests/data/https_example.org/package_example_example-v1.0.tar.gz
 create mode 100644 swh/loader/package/tests/data/https_example.org/package_example_example-v2.0.tar.gz
 create mode 100644 swh/loader/package/tests/data/https_example.org/package_example_example-v3.0.tar.gz
 create mode 100644 swh/loader/package/tests/data/https_example.org/package_example_example-v4.0.tar.gz

Changes applied before test

commit 47c931d1d6061353159d148791ef2a0d346ebbf8
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Sep 16 15:19:29 2022 +0200

    package/loader: Handle errors when retrieving package version info
    
    Some package loaders might encounter errors while attempting to get
    package info for a given version (HTTP error for instance).
    
    So handle that case and ensure partial visit when such an error occurs.
    
    Related to [T4124](https://forge.softwareheritage.org/T4124 'view original for T4124 on Phabricator')

commit e12ca02f894e5769e576d3b799a7c1bd1645a695
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Sep 19 16:26:32 2022 +0200

    package/loader: Implement load_status and visit_status
    
    Those methods were not reimplemented and could return incorrect statuses
    once the load method called.
    
    As they are useful for tests, implement them in PackageLoader class.

commit 99e56d3f8746cf38c90e3e658e8937018c08a57a
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Sep 19 15:29:12 2022 +0200

    package/tests: Add a visit success test for StubPackageLoader
    
    Such test was missing so add a couple of tiny tarballs in tests
    data directory and simulate a successful visit for origin example.
    
    Also update origin URL and remove a couple of hardcoded litterals.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/878/ for more details.

Merge request was accepted

approved this merge request

Some references in the commit message have been migrated:

• T4124 is now swh/meta#4124 (closed)

Rebase

Build is green

Patch application report for D8505 (id=30637)

Could not rebase; Attempt merge onto c4370009...

Updating c437000..d471cca
Fast-forward
 swh/loader/package/loader.py                       |  61 +++++++---
 .../package_example_example-v1.0.tar.gz            | Bin 0 -> 181 bytes
 .../package_example_example-v2.0.tar.gz            | Bin 0 -> 181 bytes
 .../package_example_example-v3.0.tar.gz            | Bin 0 -> 181 bytes
 .../package_example_example-v4.0.tar.gz            | Bin 0 -> 182 bytes
 swh/loader/package/tests/test_loader.py            | 134 ++++++++++++++++++---
 6 files changed, 157 insertions(+), 38 deletions(-)
 create mode 100644 swh/loader/package/tests/data/https_example.org/package_example_example-v1.0.tar.gz
 create mode 100644 swh/loader/package/tests/data/https_example.org/package_example_example-v2.0.tar.gz
 create mode 100644 swh/loader/package/tests/data/https_example.org/package_example_example-v3.0.tar.gz
 create mode 100644 swh/loader/package/tests/data/https_example.org/package_example_example-v4.0.tar.gz

Changes applied before test

commit d471cca9f2a32b9e5ac2f0a752e64b593d6dfd8f
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Sep 16 15:19:29 2022 +0200

    package/loader: Handle errors when retrieving package version info
    
    Some package loaders might encounter errors while attempting to get
    package info for a given version (HTTP error for instance).
    
    So handle that case and ensure partial visit when such an error occurs.
    
    Related to [T4124](https://forge.softwareheritage.org/T4124 'view original for T4124 on Phabricator')

commit 598873f00392e75f085aa0b9e957b7261127aaa2
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Sep 19 16:26:32 2022 +0200

    package/loader: Implement load_status and visit_status
    
    Those methods were not reimplemented and could return incorrect statuses
    once the load method called.
    
    As they are useful for tests, implement them in PackageLoader class.

commit 63ccbd9a3a671a4b218529be7d12d2fb1542b5cd
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Mon Sep 19 15:29:12 2022 +0200

    package/tests: Add a visit success test for StubPackageLoader
    
    Such test was missing so add a couple of tiny tarballs in tests
    data directory and simulate a successful visit for origin example.
    
    Also update origin URL and remove a couple of hardcoded litterals.

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/882/ for more details.

Merge request was merged

closed