Skip to content

Add script to mirror an upstream repository to our forge

Antoine R. Dumont requested to merge generated-differential-D7850-source into generated-differential-D7850-target

This reuses the new common library [1] to mirror a new phabricator repo (e.g [2]).

This:

  • creates a new repository in our forge
  • clones locally a new repository
  • pushes the local clone to the new swh forge repository
  • configures the remote properly

An example of mirrored repository bootstraped with this.

Related to T2225

Test Plan

$ workon swh
$ python mirror-upstream-repo https://code.librehq.com/ots/forgerie/
Repository 263 still importing, sleeping for 0.5 seconds...
Repository 263 still importing, sleeping for 1.0 seconds...
Repository 263 still importing, sleeping for 2.0 seconds...
Repository 263 still importing, sleeping for 4.0 seconds...
... some failure here ...
...
$ python bin/mirror-upstream-repo --clone https://code.librehq.com/ots/forgerie/ --repo-id 263
Cloning into 'forgerie'...
warning: You appear to have cloned an empty repository.
warning: redirecting to https://code.librehq.com/ots/forgerie.git/
remote: Enumerating objects: 957, done.
remote: Counting objects: 100% (253/253), done.
remote: Compressing objects: 100% (175/175), done.
remote: Total 957 (delta 137), reused 94 (delta 52), pack-reused 704
Receiving objects: 100% (957/957), 156.04 KiB | 739.00 KiB/s, done.
Resolving deltas: 100% (539/539), done.
From https://code.librehq.com/ots/forgerie
 * [new branch]      bug-trackers-ph-gl-mapping -> upstream/bug-trackers-ph-gl-mapping
 * [new branch]      main                       -> upstream/main
error: pathspec 'master' did not match any file(s) known to git
Traceback (most recent call last):
  File "/home/tony/work/inria/repo/swh/puppet-environment/bin/mirror-upstream-repo", line 63, in <module>
    main()
  File "/home/tony/.virtualenvs/swh/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/tony/.virtualenvs/swh/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/tony/.virtualenvs/swh/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/tony/.virtualenvs/swh/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/tony/work/inria/repo/swh/puppet-environment/bin/mirror-upstream-repo", line 53, in main
    clone_and_push_repo(
  File "/home/tony/work/inria/repo/swh/puppet-environment/bin/common.py", line 92, in clone_and_push_repo
    subprocess.check_call(["git", "checkout", default_branch])
  File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'checkout', 'master']' returned non-zero exit status 1.
$ python bin/mirror-upstream-repo --clone https://code.librehq.com/ots/forgerie/ --repo-id 263 --default-branch main
fatal: destination path 'forgerie' already exists and is not an empty directory.
Traceback (most recent call last):
  File "/home/tony/work/inria/repo/swh/puppet-environment/bin/mirror-upstream-repo", line 63, in <module>
    main()
  File "/home/tony/.virtualenvs/swh/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/tony/.virtualenvs/swh/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/tony/.virtualenvs/swh/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/tony/.virtualenvs/swh/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/tony/work/inria/repo/swh/puppet-environment/bin/mirror-upstream-repo", line 53, in main
    clone_and_push_repo(
  File "/home/tony/work/inria/repo/swh/puppet-environment/bin/common.py", line 84, in clone_and_push_repo
    subprocess.check_call(["git", "clone", phabricator_repo, module_name])
  File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['git', 'clone', 'https://forge.softwareheritage.org/source/forgerie.git', 'forgerie']' returned non-zero exit status 128.
$ python bin/mirror-upstream-repo --no-clone https://code.librehq.com/ots/forgerie/ --repo-id 263 --default-branch main
error: remote upstream already exists.
warning: redirecting to https://code.librehq.com/ots/forgerie.git/
branch 'main' set up to track 'upstream/main'.
Switched to a new branch 'main'
Already up to date.
Enumerating objects: 954, done.
Counting objects: 100% (954/954), done.
Delta compression using up to 8 threads
Compressing objects: 100% (312/312), done.
Writing objects: 100% (954/954), 155.29 KiB | 155.29 MiB/s, done.
Total 954 (delta 539), reused 954 (delta 539), pack-reused 0
remote: Resolving deltas: 100% (539/539), done.
To forge.softwareheritage.org:/source/forgerie.git
 * [new branch]      main -> main
branch 'main' set up to track 'origin/main'.

Migrated from D7850 (view on Phabricator)

Merge request reports