Skip to content

Migrate swh-web production database from SQLite to PostgreSQL

Now that the "Save code now" service gets more and more used, we started to encounter some SQLite concurrency issue, see #2891 (closed) and sentry related issue.

This calls for the use of a better Django database backend in production, PostgreSQL being the obvious choice.

This task will track the actions that need to be done to perform that migration process, notably:

  • the creation of a swh-web database in a PostgreSQL server somewhere in swh staging / production environment
  • the adaptation of Django production settings and swh-web configuration to use that database
  • the dump of current swh-web SQLite database content to JSON
  • the load from those JSON data into the new swh-web PostgreSQL database

Plan or action so far:

  • D5392: Migrate swh.web settings to declare the use of postgres
  • D5391: Ensure migration tools and setup works in docker
  • infra/puppet/puppet-swh-site!336: staging: Prepare db (puppet)
  • production: Deactivate puppet on moma, webapp1
  • infra/puppet/puppet-swh-site!337: staging: Prepare swh-site setup change to webapps
  • Land, tag package swh-web (so the migration tools is available on nodes to migrate)
  • Check everything is fine in webapp.staging.swh.network

  • production: Prepare swh-site setup change to webapps
  • production: Prepare new swh-web db (manual creation)
  • production: Run swh-web upgrade and migrate previous data dump on new db
  • production: Run upgrade on moma (reactivate puppet)

Migrated from T2945 (view on Phabricator)

Edited by Antoine R. Dumont