Skip to content
Snippets Groups Projects

Test full phabricator migration to gitlab

    • Closed Issue created by Antoine R. Dumont @ardumont

    #4064 (closed) used a sysadm subset of repositories, we need to check the overall migration on all repositories from phabricator.

    Migrated from T4383 (view on Phabricator)

    Designs

      Child items 0

      2. No child items are currently assigned. Use child items to break down this issue into smaller parts.

      Activity

      • Phabricator Migration user marked this issue as related to #4064 (closed)

        marked this issue as related to #4064 (closed)

      • Antoine R. Dumont added GitLab migration Roadmap 2020 System administration priority:High labels

        added GitLab migration Roadmap 2020 System administration priority:High labels

      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Author Owner

        I just reused the current last migration run and added the swh-objstorage to the list of repositories (not yet migrated). And the migration just broke on that repository.

        Which does not bode well for testing the full migration... I don't get the issue just yet.

        Forgerie run backtrace:

        Backtrace for: #<SB-THREAD:THREAD RUNNING {1008090E93}>
0: (SB-DEBUG::DEBUGGER-DISABLED-HOOK #<SIMPLE-ERROR "Got error running git command ~A with args ~S in dir ~A" {1015039663}> #<unused argument> :QUIT T)
1: (SB-DEBUG::RUN-HOOK *INVOKE-DEBUGGER-HOOK* #<SIMPLE-ERROR "Got error running git command ~A with args ~S in dir ~A" {1015039663}>)
2: (INVOKE-DEBUGGER #<SIMPLE-ERROR "Got error running git command ~A with args ~S in dir ~A" {1015039663}>)
3: (ERROR "Got error running git command ~A with args ~S in dir ~A" "push" ("gitlab" "--all") "/tmp/forgerie/gitlab/swh-objstorage")
4: (FORGERIE-CORE:GIT-CMD "/tmp/forgerie/gitlab/swh-objstorage" "push" ("gitlab" "--all") :ERROR T :DEBUG NIL :INPUT NIL)
5: (FORGERIE-GITLAB::CREATE-PROJECT #S(FORGERIE-CORE::VC-REPOSITORY :NAME "Object storage" :SLUG "swh-objstorage" :PRIMARY-PROJECTS (#S(FORGERIE-CORE::PROJECT :NAME "Object storage" :TAGS (#1="object_storage")) #S(FORGERIE-CORE::PROJECT :N
AME "Software Heritage Archiver" :TAGS ("software_heritage_archiver"))) :GIT-LOCATION "/srv/phabricator/repos/72/" :PROJECTS (#S(FORGERIE-CORE::PROJECT :NAME "Has debian packaging branches" :TAGS ("has_debian_packaging_branches" "debpkg"))
 #S(FORGERIE-CORE::PROJECT :NAME "Language-Python" :TAGS ("python" "language-python")) #S(FORGERIE-CORE::PROJECT :NAME "Sync to GitHub" :TAGS ("sync_to_github")) #S(FORGERIE-CORE::PROJECT :NAME "Object storage" :TAGS (#1#)) #S(FORGERIE-COR
E::PROJECT :NAME "Developers" :TAGS ("developers" "devel"))) :PRIVATE NIL :COMMITS (#S(FORGERIE-CORE:COMMIT :SHA "e712f28684a1a17233a700dd22606b594a3b617b" :PARSED-COMMENT ("Make WineryWriter.add return None

Like WineryObjStorage and other ObjStorage backends.

")) #S(FORGERIE-CORE:COMMIT :SHA "2caa05e869c952a3e8c4f3418d41ea17955f5447" :PARSED-COMMENT ("Remove get_random()

It is not used anywhere (and swh-storage provides content_get_random() that
feels the same need), and cloud backends do not implement it anyway.

")) #S(FORGERIE-CORE:COMMIT :SHA "d7f4daa242a55c00f18f4bf3d0bc2832ab4789d4" :PARSED-COMMENT ("Remove deprecated 'args' argument of get_objstorage

")) #S(FORGERIE-CORE:COMMIT :SHA "847285a28da8164a3ea618d9c14c14280058c4f8" :PARSED-COMMENT ("Add return type to get_objstorage"

        Connecting to the container and doing the same action on the same repository gives a bit more insight:

        forgerie@bespin:/tmp/forgerie/gitlab/swh-objstorage$ git push gitlab --all
Warning: Permanently added 'gitlab-staging.swh.network,20.103.140.206' (ECDSA) to the list of known hosts.
Enumerating objects: 4345, done.
Counting objects: 100% (4345/4345), done.
Delta compression using up to 12 threads
Compressing objects: 100% (1756/1756), done.
Writing objects: 100% (4345/4345), 822.75 KiB | 39.18 MiB/s, done.
Total 4345 (delta 2849), reused 3603 (delta 2370), pack-reused 0
remote: Resolving deltas: 100% (2849/2849), done.
remote: GitLab: 401 Unauthorized
To gitlab-staging.swh.network:migrated/swh-objstorage.git
 ! [remote rejected] azure-tryout -> azure-tryout (pre-receive hook declined)
 ! [remote rejected] debian/buster-swh -> debian/buster-swh (pre-receive hook declined)
 ! [remote rejected] debian/stretch-swh -> debian/stretch-swh (pre-receive hook declined)
 ! [remote rejected] debian/unstable-swh -> debian/unstable-swh (pre-receive hook declined)
 ! [remote rejected] debian/upstream -> debian/upstream (pre-receive hook declined)
 ! [remote rejected] master -> master (pre-receive hook declined)
 ! [remote rejected] pristine-tar -> pristine-tar (pre-receive hook declined)
 ! [remote rejected] wip/jenkins -> wip/jenkins (pre-receive hook declined)
error: failed to push some refs to 'gitlab-staging.swh.network:migrated/swh-objstorage.git'

        but... the ssh connection is ok... (same .ssh/config configuration as the git push...)

        forgerie@bespin:~$ ssh gitlab-staging.swh.network
Warning: Permanently added 'gitlab-staging.swh.network,20.103.140.206' (ECDSA) to the list of known hosts.
PTY allocation request failed on channel 0
Welcome to GitLab, @root!
Connection to gitlab-staging.swh.network closed.

        Gitaly server logs does not say much beside 401...

        ***/var/log/gitaly/gitaly.log***
{"correlation_id":"01G79S0PBQTAKKN4DRDDER1248","duration_ms":25,"error":"401 Unauthorized","level":"error","method":"POST","msg":"Internal API error","status":401,"time":"2022-07-06T12:51:01.990Z","url":"http://gitlab-webservice-default.gitlab-system.svc:8181//api/v4/internal/allowed"}
{"correlation_id":"01G79S0PBQTAKKN4DRDDER1248","error":"GitLab: 401 Unauthorized","grpc.meta.auth_version":"v2","grpc.meta.deadline_type":"none","grpc.meta.method_type":"bidi_stream","grpc.method":"PreReceiveHook","grpc.request.fullMethod":"/gitaly.HookService/PreReceiveHook","grpc.request.glProjectPath":"migrated/swh-objstorage","grpc.request.glRepository":"project-79","grpc.request.repoPath":"@hashed/98/a3/98a3ab7c340e8a033e7b37b6ef9428751581760af67bbab2b9e05d4964a8874a.git","grpc.request.repoStorage":"default","grpc.service":"gitaly.HookService","grpc.start_time":"2022-07-06T12:51:01.964","level":"warning","msg":"stopping transaction because pre-receive hook failed","peer.address":"@","pid":1,"span.kind":"server","system":"grpc","time":"2022-07-06T12:51:01.990Z"}
{"correlation_id":"01G79S0PBQTAKKN4DRDDER1248","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.deadline_type":"none","grpc.meta.method_type":"bidi_stream","grpc.method":"PreReceiveHook","grpc.request.fullMethod":"/gitaly.HookService/PreReceiveHook","grpc.request.glProjectPath":"migrated/swh-objstorage","grpc.request.glRepository":"project-79","grpc.request.payload_bytes":3323,"grpc.request.repoPath":"@hashed/98/a3/98a3ab7c340e8a033e7b37b6ef9428751581760af67bbab2b9e05d4964a8874a.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":30,"grpc.service":"gitaly.HookService","grpc.start_time":"2022-07-06T12:51:01.964","grpc.time_ms":26.11,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":1,"span.kind":"server","system":"grpc","time":"2022-07-06T12:51:01.991Z"}
{"correlation_id":"01G79S0PBQTAKKN4DRDDER1248","diskcache":"f4dca393-aaf5-4eeb-980a-5f5863f064ec","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-shell-git-receive-pack","grpc.meta.deadline_type":"none","grpc.meta.method_type":"bidi_stream","grpc.method":"SSHReceivePack","grpc.request.fullMethod":"/gitaly.SSHService/SSHReceivePack","grpc.request.glProjectPath":"migrated/swh-objstorage","grpc.request.glRepository":"project-79","grpc.request.repoPath":"@hashed/98/a3/98a3ab7c340e8a033e7b37b6ef9428751581760af67bbab2b9e05d4964a8874a.git","grpc.request.repoStorage":"default","grpc.service":"gitaly.SSHService","grpc.start_time":"2022-07-06T12:51:01.691","level":"info","msg":"diskcache state change","peer.address":"10.244.6.76:55044","pid":1,"remote_ip":"10.244.6.78","span.kind":"server","system":"grpc","time":"2022-07-06T12:51:02.006Z","user_id":"user-1","username":"root"}
{"command.count":1,"command.cpu_time_ms":148,"command.inblock":0,"command.majflt":0,"command.maxrss":205344,"command.minflt":3201,"command.oublock":1896,"command.real_time_ms":303,"command.system_time_ms":29,"command.user_time_ms":119,"correlation_id":"01G79S0PBQTAKKN4DRDDER1248","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-shell-git-receive-pack","grpc.meta.deadline_type":"none","grpc.meta.method_type":"bidi_stream","grpc.method":"SSHReceivePack","grpc.request.fullMethod":"/gitaly.SSHService/SSHReceivePack","grpc.request.glProjectPath":"migrated/swh-objstorage","grpc.request.glRepository":"project-79","grpc.request.payload_bytes":843812,"grpc.request.repoPath":"@hashed/98/a3/98a3ab7c340e8a033e7b37b6ef9428751581760af67bbab2b9e05d4964a8874a.git","grpc.request.repoStorage":"default","grpc.response.payload_bytes":4982,"grpc.service":"gitaly.SSHService","grpc.start_time":"2022-07-06T12:51:01.691","grpc.time_ms":314.529,"level":"info","msg":"finished streaming call with code OK","peer.address":"10.244.6.76:55044","pid":1,"remote_ip":"10.244.6.78","span.kind":"server","system":"grpc","time":"2022-07-06T12:51:02.006Z","user_id":"user-1","username":"root"}
      • Antoine R. Dumont changed title from Test full phabricator migratio to gitlab to Test full phabricator migration to gitlab

        changed title from Test full phabricator migratio to gitlab to Test full phabricator migration to gitlab

      • Vincent Sellier
        Vincent Sellier @vsellier ·
        Owner

        I suppose this one can be closed?

      • Antoine R. Dumont closed

        closed

      Please register or sign in to reply