Skip to content
Snippets Groups Projects
  1. Dec 12, 2022
  2. Dec 08, 2022
  3. Dec 07, 2022
    • Antoine Lambert's avatar
      replay: Copy dir states and external paths in copy_from operations · b016f654
      Antoine Lambert authored
      Copied directories might have externals so we also need to copy states
      and update external paths in case externals list is later modified.
      b016f654
    • Antoine Lambert's avatar
      svn: Use urllib.parse.quote to percent encode svn URLs · fc78f574
      Antoine Lambert authored
      In order to detect all ascii characters that must be percent encoded
      in svn URLs, add a brute force test and use urllib.parse.quote in
      quote_svn_url function.
      fc78f574
    • Antoine Lambert's avatar
      utils: Raise ValueError when external definition could not be parsed · f377e9f7
      Antoine Lambert authored
      Such case can happen when an external definition is malformed.
      
      Previously, the parsed malformed external was added to the directory state
      with an empty external URL which could lead to unexpected side effects like
      removing all previously exported valid externals.
      f377e9f7
    • Antoine Lambert's avatar
      replay: Simplify FileEditor implementation · 301b31e9
      Antoine Lambert authored
      Instead of maintaining file state based on svn properties across revisions
      replay and trying to reconstruct the same file as with a svn export operation
      after applying text deltas, prefer to simply export the file from the currently
      processed revision when closing the associated file editor.
      
      This greatly simplify the replay module implementation while approximatively
      keeping the same performance as before.
      
      Also add a test that would fail without these changes.
      
      Related to T4673
      301b31e9
  4. Dec 05, 2022
  5. Nov 25, 2022
    • Antoine Lambert's avatar
      replay: Add more debug logs · e35f800a
      Antoine Lambert authored
      Add more debug logs to the replay module to ease detection of issues.
      Nevertheless, as those are quite verbose, only display them when setting
      debug parameter of the loader to True.
      e35f800a
  6. Nov 23, 2022
  7. Nov 22, 2022
    • Antoine Lambert's avatar
      loader: Add logs displaying path differences after revision replay · a843858b
      Antoine Lambert authored
      When a tree computation divergence is detected after replaying a revision
      add debug logs displaying the paths that differ or are missing between the
      reconstructed repository filesystem and the exported one at that specific
      revision.
      
      It should help to gain some time when debugging such issues.
      a843858b
  8. Oct 31, 2022
    • Jenkins for Software Heritage's avatar
    • Jenkins for Software Heritage's avatar
    • Antoine Lambert's avatar
      replay: Ensure copyfrom operations are properly handled · 04566a7f
      Antoine Lambert authored
      A subversion revision can contain new directories and files copied from
      ancestor revisions but those were not perfectly handled in the commit
      editor used to reconstruct the repository filesystem when replaying
      revisions.
      
      In particular previous implementation could not handle the case where a
      path copied from an ancestor revision is replaced in a same commit (for
      instance replacing a directory by a file with the same name).
      
      These changes ensure that info about source path and source revision from
      which a path is copied is passed to the commit editor methods as paramaters
      in order to let them handle the copies but also that the replace operations
      will be correctly replayed.
      
      It also prevents OS error "Too many open files" when a really large files
      tree is copied from an ancestor revision.
      v1.4.0
      04566a7f
    • Antoine Lambert's avatar
      loader: Compress dump file and rework truncated dump handling · d24ba1a5
      Antoine Lambert authored
      When dumping a subversion repository to file before loading it, compress
      that file using gzip while producing it. It enables to save significant
      disk space while dumping a large repository.
      
      Also rework the way how truncated dump is handled now dump file is
      compressed by providing the expected max revision number to be loaded
      by svnadmin. If the number of loaded revisions matches, we can safely
      continue the partial loading of the repository.
      d24ba1a5
  9. Oct 28, 2022
  10. Oct 25, 2022
  11. Oct 19, 2022
  12. Oct 18, 2022
  13. Oct 17, 2022
    • Antoine Lambert's avatar
      replay: Use swh.model.from_disk.Directory.collect · aaa82617
      Antoine Lambert authored
      Instead of maintaining a set of modified paths for each replayed revision,
      use the swh.model.from_disk.Directory.collect method which performs the same
      task by returning added or modified contents and directories since the last
      collect operation.
      aaa82617
  14. Oct 01, 2022
  15. Sep 30, 2022
    • Antoine Lambert's avatar
      svn: Add support to basic subversion authentication in SvnRepo · 8e735c95
      Antoine Lambert authored
      Some subversion servers in the wild disabled anonymous access and require
      to authenticate using a read only user account, typically with credentials
      anonymous/anonymous.
      
      So add support for basic authentication in the subversion loader, credentials
      must be provided in the repository URL as with basic HTTP authentication.
      v1.3.5
      8e735c95
  16. Sep 15, 2022
  17. Jun 17, 2022
    • Antoine Lambert's avatar
      utils: Ensure to return correct revision in parse_external_definition · 651e2964
      Antoine Lambert authored
      An external definition can be of the following form (where XXX and YYY
      are revision numbers):
      
      	-r XXX <repo_url>@YYY
      
      In that case, the official subversion client will export the revision
      XXX of the external repository.
      
      So ensure to have the same behavior when the subversion loader processes
      a repository with such external defintion in it.
      v1.3.4
      651e2964
  18. Jun 01, 2022
  19. May 20, 2022
  20. May 09, 2022
  21. May 05, 2022
  22. May 03, 2022
  23. Apr 29, 2022
  24. Apr 27, 2022
  25. Apr 26, 2022
    • Antoine Lambert's avatar
      tasks: Fix and simplify implementation · b4bca94f
      Antoine Lambert authored
      Recent changes in swh-scheduler add new parameters to the celery tasks
      produced from swh.scheduler.model.ListedOrigin instances.
      
      Those new parameters were not properly handled in the celery tasks
      implementation for svn loading but also in some svn loader ones.
      
      So ensure to handle any new parameters by not hardcoding the expected
      ones in task signatures but also allowing any extra keyword parameters
      in loader constructors.
      
      Also remove explicit setting of visit_date if none has been provided
      as task parameter as it is already handled in loader constructor.
      
      Related to T4187
      b4bca94f
    • vlorentz's avatar
      Bump mypy to v0.942 · 91dc7ad9
      vlorentz authored
      91dc7ad9
  26. Apr 21, 2022
Loading