Skip to content
Snippets Groups Projects
  1. Oct 26, 2022
  2. Oct 25, 2022
  3. Oct 21, 2022
    • Franck Bret's avatar
      Conda: Anaconda packages archive loader · e7ba6316
      Franck Bret authored
      For each origin it takes advantage of 'artifacts' data send through
      'extra_loader_arguments' of the conda lister, providing versions,
      archive url, checksum, etc.
      Author extracted from intrinsic metadata.
      
      Related T4579
      e7ba6316
  4. Oct 18, 2022
  5. Oct 17, 2022
  6. Oct 11, 2022
  7. Oct 07, 2022
  8. Oct 05, 2022
    • Antoine R. Dumont's avatar
      {Cnt|Dir}Loader: Fix standard/nar hash mismatch behavior to fail loading · 8aa6dab7
      Antoine R. Dumont authored
      Prior to this commit, there was a discrepancy between the hash mismatch computations
      with "standard" and "nar" computations. This commit fixes the gap between those.
      
      When a hash mismatch occurs, either "nar" or "standard", the issue is caught and the
      next mirror url is checked. At the end of it all, if nothing is loaded and errors
      exist, this is raised. This fails the visit.
      
      This also adds the missing tests.
      
      Related to T3781
      8aa6dab7
    • Antoine R. Dumont's avatar
      DirectoryLoader: Check nar hashes when provided · 4d51ad99
      Antoine R. Dumont authored
      The lister now provides the "checksums_computation". This is either "standard" (for most
      cases as in bare checksums on the object retrieved) or "nar" for some edge case. In that
      case the computation is delegated to the "nix-store" command (which should be present in
      the system running the loading).
      
      This adapts the directory loader to deal with this case.
      
      No work has been done for the ContentLoader yet besides failing the case if a call
      happens with such case.
      
      Related to T3781
      4d51ad99
  9. Oct 04, 2022
  10. Oct 03, 2022
  11. Sep 30, 2022
    • Antoine Lambert's avatar
      Use tarball checksum to check download integrity in package loaders · 5482a48e
      Antoine Lambert authored
      When one or multiple tarball checksums are available, either from listers
      output or from Web APIs calls perfomed by some loaders, use them to check
      integrity of downloaded tarballs.
      5482a48e
    • Antoine R. Dumont's avatar
      Add Content Loader to ingest raw content file · f774aba5
      Antoine R. Dumont authored
      In some marginal listing cases (Nix or Guix for now), we can receive raw file to ingest.
      This commit adds a loader to ingest those. The output of the ingestion is a snapshot
      with 1 branch, one HEAD branch targetting the file content ingested.
      
      This expects to receive a mandatory 'integrity' field. It is used to check the content
      match the declaration.
      
      This can also optionally receive a list of mirror urls in case the main origin url is no
      longer available. Those mirror urls are solely used as fallback to retrieve the content.
      
      Related to T3781
      f774aba5
  12. Sep 29, 2022
    • Franck Bret's avatar
      Puppet: The puppet loader loads origins from https://forge.puppet.com · 6299c091
      Franck Bret authored
      For each origin it takes advantage of 'artifacts' data send through
      'extra_loader_arguments' from the Puppet lister, providing versions,
      archive url, last_update, filename.
      Author and description are extracted from intrinsic metadata.
      
      Related T4580
      6299c091
    • Franck Bret's avatar
      Cpan: Cpan loader loads Perl modules from cpan.org · 2db1a754
      Franck Bret authored
      For each origin it calls an http api endpoint to retrieve extrinsic
      metadata for each version of a module.
      Author and package description are extracted from intrinsic metadata
      parsing data from META.json or META.yml at the root of the archive.
      
      Related T2833
      2db1a754
  13. Sep 28, 2022
    • Antoine Lambert's avatar
      discovery: Fix compatibility with storage RPC API · 7375a83c
      Antoine Lambert authored
      Software Heritage homemade RPC layer does not known how to serialize
      set objects so we need to pass lists as parameters of *_missing
      methods from storage API.
      7375a83c
    • Raphaël Gomès's avatar
      Setup async interface for discovery module · 1facea3c
      Raphaël Gomès authored
      This will allow us to use this interface in async code like ``swh-scanner``.
      
      Unfortunately, this means calling ``asyncio.run`` for sync code, but the
      performance impact should be negligible.
      
      The ``swh_storage.*missing*`` APIs are inconsistent for each type, which
      requires a lot of boilerplate code. This should be addressed in a
      follow-up.
      1facea3c
  14. Sep 26, 2022
    • Raphaël Gomès's avatar
      Use a Merkle discovery algorithm with archives · 798f749e
      Raphaël Gomès authored
      "Discovery" is the term used to find out the differences between two
      Merkle graphs. Using such an algorithm is useful in that it drastically
      reduces the amount of data that needs to be transferred.
      
      This commit introduces an efficient but simple algorithm that is a good
      starting point for improved performance: random sampling of directories,
      the details of which are explained in the docstrings.
      
      Mercurial uses a more sophisticated algorithm for its discovery, but it
      is quite a bit more involved and would introduce too much complexity at
      once. Also, the constraints for speed that Mercurial has (in the order
      of milliseconds) don't apply as obviously to this context without
      further investigation.
      
      Benchmarks
      ==========
      
      Setup
      -----
      - With a local postgresql storage (so no network overhead), a local
        tmpfs obstorage on a fast NVME SSD, all of which should make this
        improvement look less good than it will be in production
      - With a tarball of the linux kernel at commit
        d96d875ef5dd372f533059a44f98e92de9cf0d42 already loaded
      - Loading a tarball of 20 commits earlier
        (bf3f401db6cbe010095fe3d1e233a5fde54e8b78)
      - Only taking into account the loading (not the downloading of the
        tarball, or its decompression)
      
      Result
      ------
      
      before: ~30s
      after: ~17s
      
      Reproduced 4 times.
      798f749e
    • Antoine Lambert's avatar
      26fe954b
  15. Sep 21, 2022
Loading