Skip to content

Deploy swh-loader-metadata v0.0.3

It adds metadata collection for Gitea.

(Reminder: swh-loader-metadata is a package that hooks into actual loaders; so the upgrade needs to happen on loaders)

Staging:

* [ ] Configure Gitea credentials in loader configuration, to match listers if any (there should already be GitHub credentials, same as the listers)

  • Upgrade package on workers (swh-apps: Bump requirements with new swh.loader.metadata in swh-loader-git)
    • swh-loader-git
    • swh-loader-highpriority
  • Restart workers
  • Reset journal client on the swh.journal.objects.raw_extrinsic_metadata topic (to actually run the new SWORD metadata mapping)
  • Schedule a run (with lister_name=gitea and lister_instance_name set) [1]
  • Make sure the metadata is loaded in swh-storage: select * from raw_extrinsic_metadata where target = 'swh:1:ori:...' and format = 'gitea-repository-json' limit 1;) [2]

Production:

* [ ] Configure Gitea credentials in loader configuration, to match listers if any (there should already be GitHub credentials, same as the listers)

  • Upgrade package on workers
  • Restart workers
  • so previous puppet changes can be applied
  • Reset journal client on the swh.journal.objects.raw_extrinsic_metadata topic (to actually run the new SWORD metadata mapping)
  • Schedule a run (with lister_name=gitea and lister_instance_name set) [3]
  • Make sure the metadata is loaded in swh-storage: select * from raw_extrinsic_metadata where target = 'swh:1:ori:...' and format = 'gitea-repository-json' limit 1;) [4]

[1]

swh@loader-git-6b7f97f86-lj7ss:~$ swh loader run git https://git.fsfe.org/fsfe-system-hackers/tedective lister_name=gitea lister_instance_name='git.fsfe.org'
INFO:swh.loader.git.loader.GitLoader:Load origin 'https://git.fsfe.org/fsfe-system-hackers/tedective' with type 'git'
WARNING:swh.loader.metadata.gitea:No authentication token set in configuration, using anonymous mode
Enumerating objects: 175, done.
Counting objects: 100% (175/175), done.
Compressing objects: 100% (172/172), done.
Total 800 (delta 61), reused 0 (delta 0), pack-reused 625
INFO:swh.loader.git.loader:Listed 2 refs for repo https://git.fsfe.org/fsfe-system-hackers/tedective
INFO:swh.loader.git.loader.GitLoader:Fetched 801 objects; 778 are new
{'status': 'eventful'} for origin 'https://git.fsfe.org/fsfe-system-hackers/tedective'
swh@loader-highpriority-6f4b86c9d4-chrlh:~$ swh loader run git https://git.fsfe.org/fsfe-system-hackers/staff-laptop lister_name=gitea lister_instance_name='git.fsfe.org'
INFO:swh.loader.git.loader.GitLoader:Load origin 'https://git.fsfe.org/fsfe-system-hackers/staff-laptop' with type 'git'
WARNING:swh.loader.metadata.gitea:No authentication token set in configuration, using anonymous mode
Enumerating objects: 67, done.
Total 67 (delta 0), reused 0 (delta 0), pack-reused 67
INFO:swh.loader.git.loader:Listed 2 refs for repo https://git.fsfe.org/fsfe-system-hackers/staff-laptop
INFO:swh.loader.git.loader.GitLoader:Fetched 68 objects; 66 are new
{'status': 'eventful'} for origin 'https://git.fsfe.org/fsfe-system-hackers/staff-laptop'

[2]

10:51:34 swh@db1:5432=> select now(), count(*) from raw_extrinsic_metadata where format = 'gitea-repository-json' and target='swh:1:ori:1c843fcc484c2717486b269603fe0507bdd29bf8';  -- tedective
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2022-10-20 08:58:29.455219+00 |     1 |
+-------------------------------+-------+
(1 row)

Time: 132.468 ms
10:51:30 swh@db1:5432=> select now(), count(*) from raw_extrinsic_metadata where format = 'gitea-repository-json' and target='swh:1:ori:56cdfaa0e3eaf6555c27a61ae5f9f186b9986ada';  -- staff-laptop
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2022-10-20 08:51:34.214996+00 |     1 |
+-------------------------------+-------+
(1 row)

Time: 91.929 ms

[3]

tony@yavin4 $ swhid-origin https://git.fsfe.org/fsfe-system-hackers/innernet
swh:1:ori:20ee4381fd802d0b551a91da0c5dd5ae8e0d4c63
swhworker@worker03:~$ swh loader -C /etc/softwareheritage/loader_git.yml run git https://git.fsfe.org/fsfe-system-hackers/innernet lister_name=gitea lister_instance_name='git.fsfe.org'
INFO:swh.loader.git.loader.GitLoader:Load origin 'https://git.fsfe.org/fsfe-system-hackers/innernet' with type 'git'
WARNING:swh.loader.metadata.gitea:No authentication token set in configuration, using anonymous mode
Enumerating objects: 313, done.
Counting objects: 100% (313/313), done.
Compressing objects: 100% (6/6), done.
Total 4436 (delta 308), reused 307 (delta 307), pack-reused 4123
INFO:swh.loader.git.loader:Listed 107 refs for repo https://git.fsfe.org/fsfe-system-hackers/innernet
INFO:swh.loader.git.loader.GitLoader:Fetched 4437 objects; 4023 are new
{'status': 'eventful'} for origin 'https://git.fsfe.org/fsfe-system-hackers/innernet'

[4]

16:46:51 softwareheritage@belvedere:5432=> select now(), count(*) from raw_extrinsic_metadata where format = 'gitea-repository-json' and target='swh:1:ori:20ee4381fd802d0b551a91da0c5dd5ae8e0d4c63';
+------------------------------+-------+
|             now              | count |
+------------------------------+-------+
| 2022-10-20 14:48:36.25583+00 |     1 |
+------------------------------+-------+
(1 row)

Time: 19.857 ms

Migrated from T4605 (view on Phabricator)

Edited by Antoine R. Dumont