Add LIP6 gitlab instance to regular crawling list

The LIP6 laboratory formally asked for its GitLab instance to be archived. Here is the link to the forge https://gitlab.lip6.fr/

---

Migrated from T3095 (view on Phabricator)

added Archive coverage Scientific Community Building priority:Normal labels

We would like to see this in prod as soon as reasonably possible.

added priority:High label and removed priority:Normal label

changed title from Ad LIP6 gitlab instance to regular crawling list to Add LIP6 gitlab instance to regular crawling list

Checking in staging first, with:

swhworker@worker0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ task add list-gitlab-incremental url="https://gitlab.lip6.fr/api/v4/" instance=lip6
Created 1 tasks

Task 19957696
  Next run: today (2021-03-15T10:36:57.115547+00:00)
  Interval: 1 day, 0:00:00
  Type: list-gitlab-incremental
  Policy: recurring
  Args:
  Keyword args:
    instance: 'lip6'
    url: 'https://gitlab.lip6.fr/api/v4/'

Everything went fine:

worker1.internal.staging.swh.network: Mar 15 10:37:01 worker1 python3[2277003]: [2021-03-15 10:37:01,800: INFO/MainProcess] Received task: swh.lister.gitlab.tasks.IncrementalGitLabLister[86f12806-f321-4ea1-8438-83c6fd0c457b]
worker1.internal.staging.swh.network: Mar 15 10:37:06 worker1 python3[2277067]: [2021-03-15 10:37:06,017: INFO/ForkPoolWorker-4] Task swh.lister.gitlab.tasks.IncrementalGitLabLister[86f12806-f321-4ea1-8438-83c6fd0c457b] succeeded in 4.2116
25785101205s: {'pages': 5, 'origins': 64}

Listing deployed in production:

swhscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ task add list-gitlab-incremental url="https://gitlab.lip6.fr/api/v4/" instance=lip6
Created 1 tasks

Task 377656882
  Next run: today (2021-03-15T10:47:18.408016+00:00)
  Interval: 1 day, 0:00:00
  Type: list-gitlab-incremental
  Policy: recurring
  Args:
  Keyword args:
    instance: 'lip6'
    url: 'https://gitlab.lip6.fr/api/v4/'

swhscheduler@saatchi:~$ swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ task add list-gitlab-full url="https://gitlab.lip6.fr/api/v4/" instance=lip6
Created 1 tasks

Task 377656889
  Next run: today (2021-03-15T10:47:24.752195+00:00)
  Interval: 90 days, 0:00:00
  Type: list-gitlab-full
  Policy: recurring
  Args:
  Keyword args:
    instance: 'lip6'
    url: 'https://gitlab.lip6.fr/api/v4/'

Note: I may have pushed the scheduling of both incremental and full listing task a bit too close.

It worked nonetheless (as in same result than staging):

worker09: Mar 15 10:47:21 worker09 python3[2184142]: [2021-03-15 10:47:21,376: INFO/MainProcess] Received task: swh.lister.gitlab.tasks.IncrementalGitLabLister[85bc5001-ab0c-493c-9e6a-c28e7c216859]
worker09: Mar 15 10:47:22 worker09 python3[2855486]: [2021-03-15 10:47:22,550: INFO/ForkPoolWorker-6] Task swh.lister.gitlab.tasks.IncrementalGitLabLister[85bc5001-ab0c-493c-9e6a-c28e7c216859] succeeded in 0.8740711617283523s: {'pages': 5, 'origins': 64}
worker09: Mar 15 10:47:34 worker09 python3[2184142]: [2021-03-15 10:47:34,733: INFO/MainProcess] Received task: swh.lister.gitlab.tasks.FullGitLabRelister[e7760b44-a48e-4ef6-abd2-f090ec7f16ba]
worker09: Mar 15 10:47:35 worker09 python3[2855486]: [2021-03-15 10:47:35,320: INFO/ForkPoolWorker-6] Task swh.lister.gitlab.tasks.FullGitLabRelister[e7760b44-a48e-4ef6-abd2-f090ec7f16ba] succeeded in 0.5820938930846751s: {'pages': 5, 'origins': 64}

I believe this is done: https://archive.softwareheritage.org/browse/search/?q=gitlab.lip6.fr&visit_type=git&with_content=true&with_visit=true @ardumont: can you confirm?

closed

reopened

I randomly opened a few repositories which were 'grey' with the browser exetnsion. @ardumont can you check if the lip6 gitlab is regularly crawled or was it a one time save?

@moranegg it is done, thx for the ping.

The closed status got missed on that task apparently (the forge one is closed).

closed

@ardumont Thanks!