Skip to content
Snippets Groups Projects
  1. Dec 09, 2021
  2. Dec 08, 2021
  3. Dec 07, 2021
  4. Dec 06, 2021
    • Antoine Lambert's avatar
      debian: Add md5 sum fallback when sha* checksum is missing in metadata · 2d9e93a2
      Antoine Lambert authored
      In order to check successful download of a package file, the debian loader
      will compare sha256 or sha1 checksum of the file with the one located in
      debian dsc file.
      
      However for old debian-based distributions (some ubuntu old releases for
      instance) the only available checksum in the dsc file is a md5 sum.
      
      So add a fallback to use md5 sum to check successful download when sha*
      checksum is missing in the dsc file.
      
      Related to T2400
      v1.3.0
      2d9e93a2
    • Boris Baldassari's avatar
      loader: add new maven-jar loader · 89f5ccc7
      Boris Baldassari authored
      The maven loader loads jar and zip files as Maven artefacts into the software heritage archive.
      
      Note:
      Supersedes D6158 and addresses the review done in that diff.
      
      Related to T1724
      89f5ccc7
  5. Dec 03, 2021
  6. Dec 02, 2021
  7. Dec 01, 2021
  8. Nov 22, 2021
    • vlorentz's avatar
      Package loaders: Add a newline at the end of the message · 021880d0
      vlorentz authored
      To be consistent with Git.
      v1.1.0
      021880d0
    • vlorentz's avatar
      Package loader: Uniformize author and message · 2ab367ba
      vlorentz authored
      Authors: use the empty string '' instead of placeholders
      Message: use the same message format (inspired by the Debian loader)
       for all loaders, instead of the empty string / the version /
       something else; except for PyPI and Deposit (which have a better
       format because we have more metadata available).
      
      Additionally, this commit adds test of each release object,
      instead of only relying on its hash.
      2ab367ba
  9. Nov 10, 2021
  10. Nov 09, 2021
  11. Nov 08, 2021
    • vlorentz's avatar
      Make package loaders write releases instead of revisions · 89417bb0
      vlorentz authored
      The artifacts they load match the semantics of a Release, but we used Revisions
      so far because of technical details (we needed the 'metadata' field of Revision
      that Release lacks) that is no longer relevant (thanks to the metadata storage).
      
      Packages that were loaded by previous versions of the package loader (as revs)
      will be converted to releases. In order to avoid fetching them from the origin,
      the loader will look for an existing extid pointing to a revision (like it used
      to), fetch that revision, extract some fields (directory id, author, date, ...)
      and build a new release using this information.
      
      This commit is unfortunately very large because of all changes in tests, mostly
      just new hashes and renaming 'revision' to 'release' (and various abbreviations
      and capitalizations).
      
      The only meaningful changes are in swh/loader/package/tests/test_loader.py and
      swh/loader/package/loader.py.
      
      To keep this commit as short as possible, I did not yet change individual loaders
      to create releases: they still create revisions, but are converted by the base
      loader. The next commit will refactor them to remove this conversion layer.
      89417bb0
  12. Nov 04, 2021
    • vlorentz's avatar
      tests: Remove duplicate checks · c0a98a5c
      vlorentz authored
      All the '*_missing' tests are already done automatically by check_snapshot
      (it recursively checks all objects are present in the storage).
      c0a98a5c
    • vlorentz's avatar
      tests: Hide utilities from stack traces · 2311ad9b
      vlorentz authored
      They clutter the test output because pytest prints the whole code
      of the function raising the assertionerror.
      
      With this magic variable, the error is shown as if it was raised
      directly in the caller's body.
      2311ad9b
    • vlorentz's avatar
      package loaders: Make test failures more helpful · 551c55ff
      vlorentz authored
      Some tests did the following:
      
      1. build a snapshot
      2. get the snapshot from the storage
      3. compare it with the expected snapshot
      4. get the origin visit from the storage and check it
      
      If the loader built a wrong snapshot, the test fails at step 2,
      and the only information displayed is that the expected snapshot id
      does not exist, which is very unhelpful.
      
      Instead, I reordered them as: 1, 4, 2, 3. This way, if a wrong
      snapshot is build by the loader, it is detected when comparing
      the visit, and pytest shows the two hashes.
      Then, the test can be modified to use the hash that is actually
      generated to show the actual snapshot.
      
      This is consistent with what was already done in the pypi loader.
      
      Additionally, I made the following changes:
      
      1. always check stats last (because a difference in numbers is
         hardly actionable without testing other objects)
      2. add a few more snapshot id checks in visits
      3. deduplicated a hardcoded snapshot id.
      551c55ff
    • vlorentz's avatar
      deposit: Remove 'parent' deposit · 89a0bfee
      vlorentz authored
      The parent is computed by the deposit as the revision of the latest deposit
      in the same origin before the current one.
      Therefore, it is redundant, as it can be recomputed from metadata
      + revision date.
      
      This is a preliminary change needed to make package loaders produce
      releases instead of revisions, as releases don't have parent relationships
      89a0bfee
    • vlorentz's avatar
    • vlorentz's avatar
      18bbbae7
  13. Nov 03, 2021
    • vlorentz's avatar
      Revert "deposit: Remove 'parent' deposit" · 5063082e
      vlorentz authored
      This reverts commit f6905cdf.
      
      That commit was a first step toward making loaders write releases
      instead of revisions.
      
      Unfortunately, we will still write revisions for a non-negligeable time,
      so I prefer to defer the removal of parent deposit revisions to the
      moment we actually make that switch, so we don't end up with inconsistent
      revisions.
      5063082e
  14. Oct 21, 2021
Loading