Skip to content

swh-web: Migrate from sqlite to postgresql

Antoine R. Dumont requested to merge generated-differential-D5392-source into generated-differential-D5392-target

This adapts the production settings to allow the use of the postgresql backend.

This does not touch the development or testing settings module though.

Related to swh/infra/sysadm-environment#2945 (closed)

Test Plan

docker tryouts.

Checked migration part is running with the django-admin-migrate dumpdata/loaddata stanza.

Out of the retrieved staging webapp json dump:

$ swh-doco exec swh-web /bin/bash
swh@3f80177ef10a:/$ django-admin loaddata /webapp-schema.staging.json
Installed 132 object(s) from 1 fixture(s)

psql service=swh-web-dev
swh-web=# select * from save_origin_request;
swh-web=# \conninfo
You are connected to database "swh-web" as user "postgres" on host "localhost" (address "::1") at port "5437".
 id |        request_date        | visit_type |                             origin_url                              |  status  | loading_task_id |         visit_date         | loading_task_status
----+----------------------------+------------+---------------------------------------------------------------------+----------+-----------------+----------------------------+---------------------
  1 | 2020-10-30 18:44:05.807+00 | git        | https://github.com/kubernetes/kubernetes.git                        | accepted |               1 | 2020-10-30 22:49:34.565+00 | succeeded
  2 | 2020-10-30 18:55:31.97+00  | git        | https://gitlab.com/ardumont/home                                    | accepted |               2 | 2020-10-30 19:44:19.999+00 | succeeded
  3 | 2020-10-30 18:55:42.809+00 | git        | https://github.com/NixOS/nixpkgs                                    | accepted |               3 | 2020-10-30 19:45:27.076+00 | failed
  4 | 2020-10-30 18:59:10.378+00 | hg         | https://hg.sr.ht/~zimoun/hello-example                              | accepted |              10 | 2020-11-02 14:34:22.766+00 | failed
  5 | 2020-10-31 10:10:12.777+00 | git        | https://github.com/torvalds/linux                                   | accepted |               4 | 2020-11-01 03:02:47.823+00 | failed
  6 | 2020-11-02 08:33:41.303+00 | git        | https://github.com/videolan/vlc                                     | accepted |               5 | 2020-11-02 09:44:37.583+00 | succeeded
  7 | 2020-11-02 09:54:45.282+00 | git        | https://github.com/torvalds/linux                                   | accepted |               6 |                            | failed
  8 | 2020-11-02 13:53:20.516+00 | git        | https://github.com/torvalds/linux                                   | accepted |               7 |                            | failed
  9 | 2020-11-02 14:01:46.448+00 | git        | https://github.com/torvalds/linux                                   | accepted |               8 | 2020-11-02 16:37:06.489+00 | succeeded
 10 | 2020-11-02 14:03:34.946+00 | git        | https://github.com/NixOS/nixpkgs                                    | accepted |               9 |                            | failed
 11 | 2020-11-02 15:53:18.317+00 | git        | https://github.com/NixOS/nixpkgs                                    | accepted |              12 | 2020-11-03 00:14:12.155+00 | succeeded
 12 | 2020-11-02 17:12:48.265+00 | git        | https://gitorious.org/parmap/parmap.git                             | accepted |              13 | 2020-11-02 17:13:34.731+00 | failed
 13 | 2020-11-03 09:40:50.622+00 | git        | https://github.com/vsellier/easy-cozy                               | accepted |           34059 | 2020-11-03 09:40:56.756+00 | succeeded
 14 | 2020-11-03 22:38:27.531+00 | git        | https://github.com/kubernetes/kubernetes.git                        | accepted |           34069 | 2020-11-03 22:49:50.343+00 | succeeded
 15 | 2020-11-03 22:40:01.333+00 | git        | https://github.com/torvalds/linux                                   | accepted |           34070 | 2020-11-04 00:57:04.078+00 | succeeded
 16 | 2020-11-04 11:08:01.602+00 | git        | https://forge.softwareheritage.org/source/swh-core.git              | pending  |              -1 |                            | not created
 17 | 2020-11-10 08:15:04.852+00 | git        | https://github.com/videolan/vlc                                     | accepted |           34073 | 2020-11-10 08:28:04.666+00 | succeeded

production like success as well but a bit more tedious:

swh@c32204adf7fa:/$ django-admin loaddata --settings=swh.web.settings.production /swh-web.production.json
Installed 71460 object(s) from 1 fixture(s)

In docker-compose.override.yaml

...
  swh-web:
    volumes:
      - "$SWH_ENVIRONMENT_HOME/swh-web:/src/swh-web"
      - "/tmp/webapp-schema.staging.json:/webapp-schema.staging.json:ro"
      - "/tmp/swh-web.production.json:/swh-web.production.json:ro"

Migrated from D5392 (view on Phabricator)

Merge request reports