Skip to content

launchpad: Allow bzr origins listing

Related to swh-loader-bzr#3945 (closed)

Test Plan

tox

and docker is happy too:

Lister log (with a pprint on the collection it lists)

swh-lister_1                        | [2022-02-16 16:33:50,268: INFO/MainProcess] Task swh.lister.launchpad.tasks.FullLaunchpadLister[f3e3f3aa-8f4a-4e2c-8821-facd4952e53e] received
swh-lister_1                        | ('git', <lazr.restfulclient.resource.Collection object at 0x7ff52ebef290>)
swh-lister_1                        | ('bzr', <lazr.restfulclient.resource.Collection object at 0x7ff52d1a9450>)
...
  • [1] scheduler db in docker, it's listing new bzr origins (no bzr prior to the run):
17:58:57 swh-scheduler@localhost:5433=# select now(), count(*) from listed_origins where visit_type='bzr';
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2022-02-16 16:59:07.267496+00 | 21000 |
+-------------------------------+-------+
(1 row)

Time: 5.236 ms
17:59:07 swh-scheduler@localhost:5433=# select now(), count(*) from listed_origins where visit_type='bzr';
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2022-02-16 16:59:46.291344+00 | 22000 |
+-------------------------------+-------+
(1 row)

Time: 5.584 ms
18:00:23 swh-scheduler@localhost:5433=# select now(), * from listed_origins where visit_type='bzr' order by last_update desc limit 1;
+-[ RECORD 1 ]-----------+-------------------------------------------------------------------------------+
| now                    | 2022-02-16 17:00:34.779024+00                                                 |
| lister_id              | 9290a3f8-6896-47ea-81b3-e3adc9df21be                                          |
| url                    | https://code.launchpad.net/~ubuntu-branches/ubuntu/karmic/libvncserver/karmic |
| visit_type             | bzr                                                                           |
| extra_loader_arguments | {}                                                                            |
| enabled                | t                                                                             |
| first_seen             | 2022-02-16 16:59:53.309055+00                                                 |
| last_seen              | 2022-02-16 16:59:53.309055+00                                                 |
| last_update            | 2009-06-27 00:56:06.928908+00                                                 |
+------------------------+-------------------------------------------------------------------------------+

Time: 11.362 ms

After an incremental run:

19:59:26 swh-scheduler@localhost:5433=# select now(), count(*) from listed_origins where visit_type='bzr';
+-------------------------------+--------+
|              now              | count  |
+-------------------------------+--------+
| 2022-02-17 08:18:45.201575+00 | 168000 |
+-------------------------------+--------+
(1 row)

Time: 20.536 ms
09:18:45 swh-scheduler@localhost:5433=# select * from listers where name='launchpad';
+-[ RECORD 1 ]--+-----------------------------------------------------------------------------------------------------------------------+
| id            | 9290a3f8-6896-47ea-81b3-e3adc9df21be                                                                                  |
| name          | launchpad                                                                                                             |
| instance_name | launchpad                                                                                                             |
| created       | 2022-02-16 16:24:45.466527+00                                                                                         |
| current_state | {"bzr_date_last_modified": "2009-09-10T10:21:25+00:00", "git_date_last_modified": "2022-02-16T19:07:16.970183+00:00"} |
| updated       | 2022-02-16 21:25:33.628123+00                                                                                         |
+---------------+-----------------------------------------------------------------------------------------------------------------------+

Time: 0.414 ms

Migrated from D7193 (view on Phabricator)

Merge request reports