ingest git.eclipse.org repositories

We want to ingest all Git repositories of the Eclipse foundation (and they want us to do that too!).

They are using cgit, and the full repo listing is here: https://git.eclipse.org/c/ It also comes with a "idle" column, which is a good substitute for an actual push feed.

The only annoying thing about that cgit listing is that it's paginated and html based, we might want to fix that and work with them to deploy the change (and push it upstream).

---

Migrated from T376 (view on Phabricator)

added Archive coverage priority:Normal labels

added priority:Low label and removed priority:Normal label

Now that we have a cgit lister, this should be a no brainer. If that's the case, we need it up and running quickly.

added priority:High label and removed priority:Low label

assigned to @ardumont

marked this issue as related to swh/devel/swh-lister#2999 (closed)

In the context of deploying the next gen lister in staging (swh/infra/sysadm-environment#2998 (closed)), i also tried the eclipse cgit instance

That's not ready for that instance yet:

swhworker@worker0:~$ SWH_CONFIG_FILENAME=lister.yml swh lister run --lister cgit url=https://git.eclipse.org/c/ instance=eclipse

WARNING:swh.lister.cgit.lister:Unexpected HTTP status code 500 on https://git.eclipse.org/c/osbp/org.eclipse.osbp.runtime.functionlibrary.validation.git/  # <- on their side
Traceback (most recent call last):
...
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))   # <- most probably the lister is too agressive

($928 for the full stacktrace)

Listed only 900 origins:

swh-scheduler=> select count(*) from listed_origins lo inner join listers l on lo.lister_id=l.id and l.name='cgit' and l.instance_name='eclipse';
 count
-------
   900
(1 row)

(I'm not sold yet on the no-brainer ;)

We have also swh/devel/swh-lister#2999 (closed) to attend to which should definitely help.

Thanks @ardumont for experimenting with this. The 500 seems normal: we need to tell Eclipse about us first, I'll put you in touch. So maybe it's still a no-brainer, and we just need to document the "contant the owner to get whitelisted" human step :-)

yes, agreed.

The 500 seems normal

yes, it can happen and the lister is able to deal with it.

Listed only 900 origins:

By the way, after implementing swh/devel/swh-lister#2999 (closed), the test revealed that we did 2/3 of the listing in the listing (900 origins out of 1340)..

So I think after swh/devel/swh-lister!394 (closed) is deployed, we should be able to list it in one request. We'll now ending up with only 1 http request.

So from the lister standpoint, we'll be good.

With the latest improvment, we listed the instance in one request [1]

• [1] swh/infra/sysadm-environment#3013 (closed)

Thanks @ardumont , that's great! If you think this does not need any more support on the Eclipse side, may you let them know?

! In #376 (closed), @rdicosmo wrote: Thanks @ardumont , that's great! If you think this does not need any more support on the Eclipse side, may you let them know?

Oh, you did it already, faster than thought :-)

Instance cgit scheduled [1]

And listed:

softwareheritage-scheduler=> \conninfo
You are connected to database "softwareheritage-scheduler" as user "guest" on host "belvedere.internal.softwareheritage.org" (address "192.168.100.210") at port "5432".
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
softwareheritage-scheduler=> select count(*) from listed_origins where lister_id='7a775770-2b2f-4139-aacb-ad715c022b9d';
 count
-------
  1340
(1 row)

Note that does not mean this is or will be ingested anytime soon though. We are still missing at least the one cog to actually schedule those listed origins.

• [1] swh/infra/sysadm-environment#3024 (closed)

Note that does not mean this is or will be ingested anytime soon though. We are still missing at least the one cog to actually schedule those listed origins.

More details in swh/devel/swh-scheduler#2345

unassigned @ardumont

! In #376 (closed), @ardumont wrote: Note that does not mean this is or will be ingested anytime soon though. We are still missing at least the one cog to actually schedule those listed origins.

More details in swh/devel/swh-scheduler#2345

If I understand well, everything is ready, and eclipse will be ingested when the new listers will be in production: thanks!

new listers

new scheduler, yes (new listers are already deployed and running in production)

It's been done for a while. We can see it appearing in the main archive page under cgit.

closed