Deploy new listers (gnu, phabricator)

Deploy and run those:

• gnu lister
• phabricator listers (Full, Incremental) with our instance https://forge.softwareheritage.org.

Steps:

• swh/infra/puppet/puppet-swh-site!184 (closed): Update defaults.yaml to reference the new listers

• Tag swh-lister (v0.0.24)

• Check the debian build package is fine on jenkins [1]

• Deploy latest listers

• Initialize the db with phabricator/gnu lister models [2]

• Give read-only access to new db to guest user

• Insert new task types (list-gnu-full, load-gnu, list-phab, list-phab-incremental) [3] [4]

• Deploy latest version changes

• !68 (closed): Fix phabricator credential setup

• Deploy latest changes

• Insert new tasks:

  • 1 recurring list-gnu-full task [5]
  • 1 recurring list-phab (instance swh)
  • 1 recurring list-phab-incremental (instance swh)

• [1] https://jenkins.softwareheritage.org/view/Debian%20packages/job/debian/job/packages/job/DLS/job/gbp-buildpackage/

• [2] Initialize db stanza:

$ workon swh
$ swh lister db-init --db-url postgres://swh-lister:<redacted>@belvedere.internal.softwareheritage.org:5432/swh-lister gnu
INFO:swh.lister.cli:Initializing lister gnu
INFO:swh.lister.cli:Creating tables for gnu
$ swh lister db-init --db-url postgres://swh-lister:<redacted>@belvedere.internal.softwareheritage.org:5432/swh-lister phabricator
INFO:swh.lister.cli:Initializing lister phabricator
INFO:swh.lister.cli:Creating tables for phabricator

• [3] Initialize task-type stanza:

$ SCHEDULER_API_URL=http://saatchi.internal.softwareheritage.org:5008/
$ swh scheduler --url $SCHEDULER_API_URL task-type add \
    list-gnu-full "swh.lister.gnu.tasks.GNUListerTask" "Full GNU Lister" \
    --default-interval '90 days' \
    --backoff-factor 1
$ swh scheduler --url $SCHEDULER_API_URL task-type add \
    list-phabricator-full "swh.lister.phabricator.tasks.FullPhabricatorLister" "Full Phabricator instance Lister" \
    --default-interval '90 days' \
    --backoff-factor 1
$ swh scheduler --url $SCHEDULER_API_URL task-type add \
    list-phabricator-incremental "swh.lister.phabricator.tasks.IncrementalPhabricatorLister" "Incremental Phabricator instance Lister" \
    --default-interval '1 week' \
    --backoff-factor 1

• [4] load-gnu does not exist but we need something in the db for the listing to not fail. That task-type will need to be completed when we actually have something for it.

• [5]

swh scheduler --url http://saatchi.internal.softwareheritage.org:5008/ task add list-gnu-full --policy recurring
Created 1 tasks

Task 167876594
  Next run: just now (2019-06-14 14:40:31+00:00)
  Interval: 90 days, 0:00:00
  Type: list-gnu-full
  Policy: recurring
  Args:
  Keyword args:

---

Migrated from T1797 (view on Phabricator)