Skip to content

Add Forge Now - Process https://git.joeyh.name/

https://archive.softwareheritage.org/admin/add-forge/request/14/

staging:

  • Schedule oneshot forge instance listing [1]
  • Disable origins listed [2]
  • Schedule some origins for ingestion [3]
  • Stop once some passed (meaning purge rabbitmq queue) [4]

production (same m.o. than ^):

  • Schedule recurring forge instance listing
  • Schedule origins for ingestion
  • Ensure ingestion started [5]

[1]

swhscheduler@scheduler0:~$ swh scheduler --url http://scheduler0.internal.staging.swh.network:5008/ task add list-cgit -p oneshot url=https://git.joeyh.name/
Created 1 tasks

Task 33419910
  Next run: today (2022-11-08T10:10:42.502228+00:00)
  Interval: 1 day, 0:00:00
  Type: list-cgit
  Policy: oneshot
  Args:
  Keyword args:
    url: 'https://git.joeyh.name/'

[2]

11:12:52 swh-scheduler@db1:5432=> select * from listers where name ='cgit' and instance_name='git.joeyh.name' order by created desc limit 1;
+--------------------------------------+------+----------------+-------------------------------+---------------+-------------------------------+
|                  id                  | name | instance_name  |            created            | current_state |            updated            |
+--------------------------------------+------+----------------+-------------------------------+---------------+-------------------------------+
| 1224be51-6b53-4bc0-bc8c-7fdfff8975db | cgit | git.joeyh.name | 2022-11-08 10:10:54.338087+00 | {}            | 2022-11-08 10:10:54.338087+00 |
+--------------------------------------+------+----------------+-------------------------------+---------------+-------------------------------+
(1 row)
11:13:27 swh-scheduler@db1:5432=> update listed_origins set enabled=false where lister_id='1224be51-6b53-4bc0-bc8c-7fdfff8975db';
UPDATE 65
Time: 815.715 ms

[3]

swhscheduler@scheduler0:~$ ./git-joeyh-name.sh                                                                                                                                                                                         [14/795]
Tue Nov  8 10:16:44 UTC 2022 scheduling git origins with policy never_visited_oldest_update_first to queue add_forge_now:swh.loader.git.tasks.UpdateGitRepository for lister git.joeyh.name (tablesample 1)
100 slots available in celery queue
65 visits to send to celery
...

[4]

loaders [2022-11-08 10:17:15,762: INFO/MainProcess] loader@loader-addforgenow-df9784495-slmbx ready.
loaders [2022-11-08 10:17:15,767: INFO/MainProcess] Task swh.loader.git.tasks.UpdateGitRepository[8dc0fa4e-a2c0-4953-bea7-bfde686e4809] received
loaders [2022-11-08 10:17:16,775: INFO/ForkPoolWorker-1] Load origin 'https://git.joeyh.name/git/zzattic/alien.git' with type 'git'
loaders [2022-11-08 10:17:18,437: INFO/ForkPoolWorker-1] Listed 39 refs for repo https://git.joeyh.name/git/zzattic/alien.git
loaders [2022-11-08 10:18:23,792: INFO/ForkPoolWorker-1] Fetched 2059 objects; 258 are new
loaders [2022-11-08 10:18:23,920: INFO/ForkPoolWorker-1] Task swh.loader.git.tasks.UpdateGitRepository[8dc0fa4e-a2c0-4953-bea7-bfde686e4809] succeeded in 68.02570837800158s: {'status': 'eventful'}
loaders [2022-11-08 10:18:23,925: INFO/MainProcess] Task swh.loader.git.tasks.UpdateGitRepository[025276dd-5e16-4925-8f98-77b9db436af9] received
loaders [2022-11-08 10:18:24,277: INFO/ForkPoolWorker-1] Load origin 'https://git.joeyh.name/git/zzattic/jetring.git' with type 'git'
loaders [2022-11-08 10:18:25,677: INFO/ForkPoolWorker-1] Listed 21 refs for repo https://git.joeyh.name/git/zzattic/jetring.git

[5]

loaders [2022-11-08 10:32:13,388: INFO/ForkPoolWorker-1] Task swh.loader.git.tasks.UpdateGitRepository[e5b98e40-bb54-43b9-ba2a-d97610980be5] succeeded in 1.9239774579182267s: {'status': 'uneventful'}
loaders [2022-11-08 10:32:13,414: INFO/MainProcess] Task swh.loader.git.tasks.UpdateGitRepository[4e8a983c-f987-4027-b21e-c017884f9780] received
loaders [2022-11-08 10:32:13,599: INFO/ForkPoolWorker-1] Load origin 'https://git.joeyh.name/git/mpdtoys.git' with type 'git'
loaders [2022-11-08 10:32:16,881: INFO/ForkPoolWorker-1] Listed 29 refs for repo https://git.joeyh.name/git/mpdtoys.git
loaders [2022-11-08 10:32:23,556: INFO/ForkPoolWorker-1] Fetched 582 objects; 47 are new
loaders [2022-11-08 10:32:24,052: INFO/ForkPoolWorker-1] Task swh.loader.git.tasks.UpdateGitRepository[e4d80b08-9ab1-425c-ac79-8f946b0a5920] succeeded in 10.62666466878727s: {'status': 'eventful'}
loaders [2022-11-08 10:32:24,062: INFO/MainProcess] Task swh.loader.git.tasks.UpdateGitRepository[99b184a8-ffeb-43d5-9496-52006292a5c9] received
loaders [2022-11-08 10:32:24,242: INFO/ForkPoolWorker-1] Load origin 'https://git.joeyh.name/git/quarantimer.git' with type 'git'
loaders [2022-11-08 10:32:40,409: INFO/ForkPoolWorker-1] Listed 5 refs for repo https://git.joeyh.name/git/quarantimer.git
loaders [2022-11-08 10:33:26,485: INFO/ForkPoolWorker-1] Fetched 266 objects; 235 are new
loaders [2022-11-08 10:33:26,684: INFO/ForkPoolWorker-1] Task swh.loader.git.tasks.UpdateGitRepository[4e8a983c-f987-4027-b21e-c017884f9780] succeeded in 62.619401288218796s: {'status': 'eventful'}
Edited by Antoine R. Dumont