Skip to content

listers: Open cli to allow a first listing pass on new forge instance

Antoine R. Dumont requested to merge generated-differential-D1919-source into generated-differential-D1919-target

The output of that listing results in scheduled tasks with policy 'oneshot' and some priority (well depending on the input cli call). Thus ingesting those faster and without manual intervention as we currently do [1]

Example use case (for now is for small forges):

$ export SWH_CONFIG_FILENAME=/etc/softwareheritage/lister.yml
$ swh lister run --lister gitlab \
                 --priority high \
                 --db-url postgresql://postgres@localhost:5432/swh-listers \
                 api_baseurl=https://gitlab.ow2.org/api/v4/

Related T1919

  • [1] along the lines of psql update "the listed tasks with priority to high", then wait for it to finish somehow, and then at some point, remove that priority with psql update yet again... tedious!

Also, to explicit this. This does not replace a usual deployment which triggers recurring tasks with no priority. This is to ease a first pass on new (small?) forges without having to plunge within the scheduler and drag a whole new task (we discussed about some refactoring, with schema impacts, on the scheduler but that's for another time).

Test Plan

doco up -d swh-lister

Then trigger the sample cli use case and check the scheduler/lister db:

 policy  | priority |                                              arguments                                          $
---------+----------+-------------------------------------------------------------------------------------------------$
 oneshot | high     | {"args": ["https://gitlab.ow2.org/sat4j/sat4j.git"], "kwargs": {}}
 oneshot | high     | {"args": ["https://gitlab.ow2.org/stamp/unit-test-ampli.git"], "kwargs": {}}
 oneshot | high     | {"args": ["https://gitlab.ow2.org/stamp/conf-test-ampli.git"], "kwargs": {}}
 ...

A usual deployment would have seen the policy set to recurring and the priority to null.

Note:

docker-compose.override.yml:

version: '2'

services:
  swh-listers-db:
    ports:
      - "5432:5432"

  swh-scheduler-db:
    ports:
      - "5433:5432"

  swh-lister:
    volumes:
      - "$SWH_ENVIRONMENT_HOME/swh-lister:/src/swh-lister"

Migrated from D1919 (view on Phabricator)

Merge request reports