Add forge now - Process https://git.neil.brown.name/

https://archive.softwareheritage.org/admin/add-forge/request/163/

Type: gitweb

changed milestone to %Extend archive coverage [Roadmap - Collect]

added AddForgeNow label

assigned to @guillaume

On staging environment the first ingestion is completed with one failure, see add-forge-now-requests#43 (closed).

On production environment:

swh@swh-toolbox-8499ff5477-ghgpp:~$ swh scheduler -C $SWH_CONFIG_FILENAME \
  add-forge-now --preset production \
  register-lister gitweb \
    instance=git.neil.brown.name
Created 1 tasks

Task 415916172
  Next run: today (2023-11-10T08:29:13.904035+00:00)
  Interval: 1 day, 0:00:00
  Type: list-gitweb
  Policy: recurring
  Args:
  Keyword args:
    instance: 'git.neil.brown.name'

Check the listed origins:

swh@swh-toolbox-8499ff5477-ghgpp:~$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-listed-origins gitweb git.neil.brown.name -l
url                                   last_seen                         last_update
------------------------------------  --------------------------------  --------------------------------
git://git.neil.brown.name/LaFS.git    2023-11-10 08:31:35.206778+00:00  2011-11-10 08:29:29.044019+00:00
[...]
git://git.neil.brown.name/wiggle.git  2023-11-10 08:31:35.206778+00:00  2023-09-15 08:31:34.362032+00:00

Forge git.neil.brown.name (gitweb) has 21 listed origins in the scheduler database.

Schedule the first ingestion:

swh@swh-toolbox-8499ff5477-ghgpp:~$ swh scheduler -C $SWH_CONFIG_FILENAME \
  add-forge-now --preset production \
    schedule-first-visits \
      --type-name git \
      --lister-name gitweb \
      --lister-instance-name git.neil.brown.name

10000 slots available in celery queue
21 visits to send to celery

The first ingestion is completed with a few failures:

swh@swh-toolbox-8499ff5477-ghgpp:~$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-ingested-origins gitweb git.neil.brown.name

Forge git.neil.brown.name (gitweb) has 21 scheduled ingests in the scheduler.
failed       : 2
None         : 0
not_found    : 0
successful   : 19
total        : 21
success rate : 90.47619047619048%

swh@swh-toolbox-8499ff5477-ghgpp:~$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-ingested-origins gitweb git.neil.brown.name -l | awk '/failed/'
git://git.neil.brown.name/git.git                      failed               2023-11-10 08:34:11.032735+00:00
git://git.neil.brown.name/linux.git                    failed               2023-11-10 08:34:18.141960+00:00
failed       : 2

added 30m of time spent

closed