Skip to content
Snippets Groups Projects
  1. Jul 29, 2021
  2. Jul 28, 2021
  3. Jun 16, 2021
  4. Jun 15, 2021
    • Raphaël Gomès's avatar
      loader: add an hg-specific mapping for branching · 2877eb3c
      Raphaël Gomès authored
      As discussed in T3352, the branching mechanism of Mercurial is more
      featureful than that of Git's. The Snapshot model was not designed
      with multiple heads, closed heads, bookmarks, etc. in mind, but with
      only branches being "pointers" to (mostly) revisions.
      As a workaround for a possible re-design of the Snapshot model (though
      nothing of the sort is planned for now), we define a mapping that better
      represents Mercurial's branching system.
      
      This allows for handling multiple heads per branch and closed branches,
      whose revisions (if not already covered by another branch) would
      previously have been lost to the ether. Additionally, bookmarks are now
      saved to get a better representation of the projects that do use them.
      2877eb3c
  5. Jun 09, 2021
  6. Jun 03, 2021
    • Raphaël Gomès's avatar
      Fix data build script · f9fa0dd3
      Raphaël Gomès authored
      The subprocess call (which should have been using run) uses the -u
      bash option, which stops the script at the first sight of an unset
      variable... but also doesn't set the variable expected by the very
      docstring example.
      
      We also reset the environment so that the hg script is not
      influenced by user configuration, which was a second source of
      breakage.
      f9fa0dd3
  7. May 28, 2021
  8. May 27, 2021
    • Raphaël Gomès's avatar
      loader: don't ignore closed branches · 44fc0133
      Raphaël Gomès authored
      The existing util for getting a repo's branches skips closed branches
      but did not leave any explanation why, either in the code or in the
      commit message. I cannot think of a good reason for ignoring closed
      branches, so we're removing this exception, which in turn fixes the
      incremental issue detailed in T3336.
      
      This has affected existing tests of the two repositories that had closed
      branches. A test for the incremental behavior was added as well.
      44fc0133
  9. May 21, 2021
  10. May 20, 2021
  11. May 19, 2021
    • Raphaël Gomès's avatar
      Replace old loader with the new one · 92441e90
      Raphaël Gomès authored
      This is the minimal amount of code needed to switch from the old one to
      the new one. If the new loader proves to be good enough, we may remove
      the old one entirely.
      92441e90
  12. May 07, 2021
    • Raphaël Gomès's avatar
      Make the Mercurial loader incremental · 4630de8a
      Raphaël Gomès authored
      Before this change, if a previous snapshot of a given origin existed
      and new changes were detected, we would start from scratch.
      
      This change leverages the recent new db mapping for external ids (like
      Mercurial's node ids) to internal SWH ids to compute what has changed
      from the latest snapshot, now that it is possible to find an SWH id from
      a Mercurial node id.
      
      For revisions, the logic is simple: look at the heads we've saved and
      ask Mercurial for all the revisions that are not ancestors of these
      heads (themselves excluded). This is not as "clever" as the full
      Mercurial discovery algorithm, but is much simpler and good enough for
      the kinds of scales we're operating at on a single repository.
      
      For tags, the previous logic assumed that all possible target revisions
      were done in the same run. Here, we look at the difference between the
      tags Mercurial reports and the one form the previous snapshot; any new
      tag will either have its corresponding release in cache (because it was
      processed in the same run) or fetched from the database using the
      aforementioned mapping.
      4630de8a
    • Raphaël Gomès's avatar
      Move `os.environ` manipulation to pre_cleanup · 773d872a
      Raphaël Gomès authored
      Simply initializing a loader would empty the environment, which can
      cause seemingly unrelated things to break. Moving the environment
      handling to the `pre_cleanup` phase ensures that `cleanup` will also
      be called and the environment will not be left in a broken state.
      
      We also add the `HGRCSKIPREPO` variable that I forgot to add in the
      test environment. This is still needed because the tests invoke
      `hg` directly. We could potentially have a wrapper util that uses a
      context-manager to do the environment manipulation closer to the issue,
      but we'd have to make sure that no other bare `hg` invocations can
      happen, even in random subprocesses.
      773d872a
  13. Apr 30, 2021
    • Raphaël Gomès's avatar
      Handle more cases of corruption · 88847148
      Raphaël Gomès authored
      Some corrupted repos have missing files or broken logical links in the
      underlying Mercurial datastructure, which means that say sometimes fail
      for a given revision. This does not mean we should throw away the rest
      of the repository. (Tested on repos of various levels and flavors of
      corruption in the Boatbucket archive)
      88847148
    • Raphaël Gomès's avatar
      Ignore the repository's config · f73d960b
      Raphaël Gomès authored
      `HGRCPATH` only tells Mercurial to ignore the user's config files, but
      some repositories have a `.hg/hgrc` file (only in the case that you copy
      the files instead of cloning, if present) that is usually used for server-side
      configuration. We want to ignore this, since it might affect loading
      and ask for hooks that are not there or are otherwise annoying/dangerous,
      for example.
      f73d960b
    • Raphaël Gomès's avatar
      Also use minimal env in the new Mercurial loader · aa80a360
      Raphaël Gomès authored
      The old loader (bundle2 loader) already received this treatment which
      ensures Mercurial doesn't pick up on any user customization, but I
      apparently forgot to apply the same changes to the new one.
      aa80a360
    • Raphaël Gomès's avatar
      Use billiard instead of stdlib multiprocessing · 23260277
      Raphaël Gomès authored
      This circumvents a few celery-related issues, and is consistent with
      what the rest of the codebase does.
      
      stdlib multiprocessing is not able to spawn children from daemonic
      processes, and even says so plainly if you try:
      
      `AssertionError: daemonic processes are not allowed to have children`
      
      This is incompatible with the SWH infrastructure which needs to do this
      exactly. Fortunately, we're already using billiard and celery. I'm
      assuming that there could be other blocking or annoying differences
      between stdlib and billiard, but we will save ourselves the trouble of
      finding out.
      23260277
  14. Apr 29, 2021
  15. Apr 26, 2021
    • Antoine Lambert's avatar
      tox: Add sphinx environments to check sane doc build · 504ee123
      Antoine Lambert authored
      Enable to check package documentation can be built without producing
      sphinx warnings.
      
      The sphinx environment is designed to be used in continuous integration
      in order to prevent breaking documentation build when committing changes.
      
      The sphinx-dev environment is designed to be used inside a full swh
      development environment.
      
      Related to T3258
      v0.5.0
      504ee123
  16. Apr 13, 2021
  17. Apr 06, 2021
  18. Mar 30, 2021
Loading