Skip to content
Snippets Groups Projects
  1. May 07, 2021
    • Raphaël Gomès's avatar
      Move `os.environ` manipulation to pre_cleanup · 773d872a
      Raphaël Gomès authored
      Simply initializing a loader would empty the environment, which can
      cause seemingly unrelated things to break. Moving the environment
      handling to the `pre_cleanup` phase ensures that `cleanup` will also
      be called and the environment will not be left in a broken state.
      
      We also add the `HGRCSKIPREPO` variable that I forgot to add in the
      test environment. This is still needed because the tests invoke
      `hg` directly. We could potentially have a wrapper util that uses a
      context-manager to do the environment manipulation closer to the issue,
      but we'd have to make sure that no other bare `hg` invocations can
      happen, even in random subprocesses.
      773d872a
  2. Apr 30, 2021
    • Raphaël Gomès's avatar
      Handle more cases of corruption · 88847148
      Raphaël Gomès authored
      Some corrupted repos have missing files or broken logical links in the
      underlying Mercurial datastructure, which means that say sometimes fail
      for a given revision. This does not mean we should throw away the rest
      of the repository. (Tested on repos of various levels and flavors of
      corruption in the Boatbucket archive)
      88847148
    • Raphaël Gomès's avatar
      Ignore the repository's config · f73d960b
      Raphaël Gomès authored
      `HGRCPATH` only tells Mercurial to ignore the user's config files, but
      some repositories have a `.hg/hgrc` file (only in the case that you copy
      the files instead of cloning, if present) that is usually used for server-side
      configuration. We want to ignore this, since it might affect loading
      and ask for hooks that are not there or are otherwise annoying/dangerous,
      for example.
      f73d960b
    • Raphaël Gomès's avatar
      Also use minimal env in the new Mercurial loader · aa80a360
      Raphaël Gomès authored
      The old loader (bundle2 loader) already received this treatment which
      ensures Mercurial doesn't pick up on any user customization, but I
      apparently forgot to apply the same changes to the new one.
      aa80a360
    • Raphaël Gomès's avatar
      Use billiard instead of stdlib multiprocessing · 23260277
      Raphaël Gomès authored
      This circumvents a few celery-related issues, and is consistent with
      what the rest of the codebase does.
      
      stdlib multiprocessing is not able to spawn children from daemonic
      processes, and even says so plainly if you try:
      
      `AssertionError: daemonic processes are not allowed to have children`
      
      This is incompatible with the SWH infrastructure which needs to do this
      exactly. Fortunately, we're already using billiard and celery. I'm
      assuming that there could be other blocking or annoying differences
      between stdlib and billiard, but we will save ourselves the trouble of
      finding out.
      23260277
  3. Apr 26, 2021
    • Antoine Lambert's avatar
      tox: Add sphinx environments to check sane doc build · 504ee123
      Antoine Lambert authored
      Enable to check package documentation can be built without producing
      sphinx warnings.
      
      The sphinx environment is designed to be used in continuous integration
      in order to prevent breaking documentation build when committing changes.
      
      The sphinx-dev environment is designed to be used inside a full swh
      development environment.
      
      Related to T3258
      v0.5.0
      504ee123
  4. Apr 13, 2021
  5. Apr 06, 2021
  6. Mar 30, 2021
  7. Mar 29, 2021
  8. Mar 05, 2021
  9. Feb 26, 2021
  10. Feb 25, 2021
    • Antoine R. Dumont's avatar
      mercurial.cli: Deprecate cli in favor of the generic `swh loader run` · d8572187
      Antoine R. Dumont authored
      This module is:
      - not referenced in the setup.py (so no simple cli call)
      - not tested
      - using an in-memory storage
      - no longer the canonical way of triggering a mercurial ingestion
      
      Note that this also fixes the visit date cli input. When user provided, this date should
      be parsed as a datetime prior to being passed to the loader constructor.
      d8572187
  11. Feb 23, 2021
  12. Feb 17, 2021
  13. Feb 15, 2021
  14. Feb 09, 2021
  15. Feb 03, 2021
  16. Feb 01, 2021
  17. Dec 01, 2020
  18. Nov 30, 2020
  19. Nov 12, 2020
    • Antoine Cezar's avatar
      Add new example repository generated from script · c8c91ab6
      Antoine Cezar authored
      First updatable example repository documented by its generation script.
      c8c91ab6
    • Antoine Cezar's avatar
      Add `Hg20BundleLoader` tests from json files · bc32e128
      Antoine Cezar authored
      Generated json files with `swh/loader/mercurial/tests/data/build.py` for
      existing repositories and added them to `Hg20BundleLoader` tests.
      
      Introduce `LoaderChecker` as a standardized way to test repositories
      against json files.
      bc32e128
    • Antoine Cezar's avatar
      Add testing repository builder · ff11f77f
      Antoine Cezar authored
      This build script purpose is to create example repositories from bash scripts
      and extract assertion data from them into json files.
      
      Advantages:
      
          - the bash script documents the repository creation
          - automating creation allow easy repository update
          - automation extraction allow easier update of assertion data
      ff11f77f
Loading