Add forge now - process https://git.afpy.org

mentioned in commit swh-apps@20800c9e

After patching the scheduler to simplify the calls [1], I finally triggered some origins to ingest (and crafted a generic script around it [2]):

swhscheduler@scheduler0:~/addforgenow$ ./addforge-now-schedule-with-url-lister-and-optional-visit-type-and-queue.sh gitea git.afpy.org
Tue Dec  6 17:16:20 UTC 2022 scheduling git origins with policy never_visited_oldest_update_first to queue add_forge_now:swh.loader.git.tasks.UpdateGitRepository for lister gitea (instance git.afpy.org) (tablesample 1)
100 slots available in celery queue
15 visits to send to celery

They got ingested [3]

We'll probably want to puppetize this script at some point.

Related to https://forge.softwareheritage.org/P1540

[1] https://forge.softwareheritage.org/D8922

[2] addforge-now-schedule-with-url-lister-and-optional-visit-type-and-queue.sh

[3]

2022-12-06 18:47:43 swh-scheduler@db1:5432 λ select * from origin_visit_stats where visit_type='git' and url like 'https://git.afpy.org%';
+-----------------------------------------------------------+------------+--------------------------------------------+-------------------------------+---------------------------+----------------------+-------------------+-------------------------------+-------------------------------+-------------------+
|                            url                            | visit_type |               last_snapshot                |        last_scheduled         | next_visit_queue_position | next_position_offset | successive_visits |        last_successful        |          last_visit           | last_visit_status |
+-----------------------------------------------------------+------------+--------------------------------------------+-------------------------------+---------------------------+----------------------+-------------------+-------------------------------+-------------------------------+-------------------+
| https://git.afpy.org/fcode/boilerplate-python.git         | git        | \x812e8ff75a6424c51cfd22a9202503f21cccf13d | 2022-12-06 17:16:22.384523+00 |              167043832784 |                    4 |                 1 | 2022-12-06 17:17:28.58761+00  | 2022-12-06 17:17:28.58761+00  | successful        |
| https://git.afpy.org/AFPy/infra.git                       | git        | \x2696466a836f7411e35d3802d0de31fb5a3a1c5d | 2022-12-06 17:16:22.384523+00 |              167043839992 |                    4 |                 1 | 2022-12-06 17:27:31.499761+00 | 2022-12-06 17:27:31.499761+00 | successful        |
| https://git.afpy.org/mdk/git-xss-locator.git              | git        | \xa55b9be6a67299b668084a68c2775081f3cfe255 | 2022-12-06 17:16:22.384523+00 |              167043822872 |                    4 |                 1 | 2022-12-06 17:17:11.161265+00 | 2022-12-06 17:17:11.161265+00 | successful        |
| https://git.afpy.org/HARFANG3D/harfang3d.git              | git        | \x140a4581ac338e0196f71634eb5c8a388afaf419 | 2022-12-06 17:16:22.384523+00 |              167043836254 |                    4 |                 1 | 2022-12-06 17:25:33.642212+00 | 2022-12-06 17:25:33.642212+00 | successful        |
| https://git.afpy.org/mdk/horrlang.git                     | git        | \x88039a3ad3630e246880707ac7b5052879f78810 | 2022-12-06 17:16:22.384523+00 |              167043844551 |                    4 |                 1 | 2022-12-06 17:17:31.745645+00 | 2022-12-06 17:17:31.745645+00 | successful        |
| https://git.afpy.org/mdk/infra.git                        | git        | \x6d4974ee86a729ad36c512b8ca5caaa4314c2106 | 2022-12-06 17:16:22.384523+00 |              167043829585 |                    4 |                 1 | 2022-12-06 17:20:56.603273+00 | 2022-12-06 17:20:56.603273+00 | successful        |
| https://git.afpy.org/foxmask/dj-diabetes.git              | git        | \x7c7a184719d6b66a93e283b6711b79677c514fb9 | 2022-12-06 17:16:22.384523+00 |              167043844794 |                    4 |                 1 | 2022-12-06 17:18:30.15001+00  | 2022-12-06 17:18:30.15001+00  | successful        |
| https://git.afpy.org/foxmask/baeuda.git                   | git        | \x07d1cfe8e163562f035473c420eb45acb48a359d | 2022-12-06 17:16:22.384523+00 |              167043853107 |                    4 |                 1 | 2022-12-06 17:17:34.682106+00 | 2022-12-06 17:17:34.682106+00 | successful        |
| https://git.afpy.org/mdk/hl-vt100.git                     | git        | \x7a649953bce78849007625bddc8dfc69b152f1a8 | 2022-12-06 17:16:22.384523+00 |              167043834780 |                    4 |                 1 | 2022-12-06 17:17:38.292726+00 | 2022-12-06 17:17:38.292726+00 | successful        |
| https://git.afpy.org/freezed/free_zed.gitlab.io.git       | git        | \x8a5941dd3ec9a4350ed8fadfd3486531fe57c4ff | 2022-12-06 17:16:22.384523+00 |              167043839901 |                    4 |                 1 | 2022-12-06 17:20:07.322192+00 | 2022-12-06 17:20:07.322192+00 | successful        |
| https://git.afpy.org/AFPy/afpy.org.git                    | git        | \x8e7d398a34ed2a2a084397660733e08216770daa | 2022-12-06 17:16:22.384523+00 |              167043853574 |                    4 |                 1 | 2022-12-06 17:23:14.834908+00 | 2022-12-06 17:23:14.834908+00 | successful        |
| https://git.afpy.org/AFPy/gestion.git                     | git        | \xb8464eef6f90f6d2e836e25122c9b506b01f81f4 | 2022-12-06 17:16:22.384523+00 |              167043841956 |                    4 |                 1 | 2022-12-06 17:31:49.301271+00 | 2022-12-06 17:31:49.301271+00 | successful        |
| https://git.afpy.org/mdk/hkis-au-campus-du-libre-2022.git | git        | \x0a2116e2b561f881ca9d6b5ef73b20fddbcd5f48 | 2022-12-06 17:16:22.384523+00 |              167043833627 |                    4 |                 1 | 2022-12-06 17:25:38.914527+00 | 2022-12-06 17:25:38.914527+00 | successful        |
| https://git.afpy.org/foxmask/django-th.git                | git        | \x004bc22e7b2e2a1158fceed1a86a7209936859c8 | 2022-12-06 17:16:22.384523+00 |              167043824244 |                    4 |                 1 | 2022-12-06 17:27:21.051499+00 | 2022-12-06 17:27:21.051499+00 | successful        |
| https://git.afpy.org/fcode/delarte.git                    | git        | \x76517eca13c0858067d391e082b86033d16ab17e | 2022-12-06 17:16:22.384523+00 |              167043856375 |                    4 |                 1 | 2022-12-06 17:27:37.400123+00 | 2022-12-06 17:27:37.400123+00 | successful        |
+-----------------------------------------------------------+------------+--------------------------------------------+-------------------------------+---------------------------+----------------------+-------------------+-------------------------------+-------------------------------+-------------------+
(15 rows)

changed the description

Production:

swhscheduler@saatchi:~/addforgenow$ ./addforge-now-schedule-with-url-lister-and-optional-visit-type-and-queue.sh gitea git.afpy.org
Wed Dec  7 11:51:22 UTC 2022 scheduling git origins with policy never_visited_oldest_update_first to queue add_forge_now:swh.loader.git.tasks.UpdateGitRepository for lister gitea (instance git.afpy.org) (tablesample 1)
10000 slots available in celery queue
37 visits to send to celery

marked the checklist item Trigger full listing as completed

marked the checklist item Trigger origin addforgenow scheduling as completed

Ingestion done.

2022-12-07 13:49:23 softwareheritage-scheduler@belvedere:5432 λ select count(*) from origin_visit_stats where visit_type='git' and url like 'https://git.afpy.org%';
+-------+
| count |
+-------+
|    37 |
+-------+
(1 row)

Time: 128600.255 ms (02:08.600)

cc @bchauvet

closed

mentioned in issue #4694 (closed)

added activity::Processing label

Add forge now - process https://git.afpy.org

Designs

Child items ...

Activity