For consistency with swh.model v0.7.2, to prepare for the addition of
an (intrinsic) id field to RawExtrinsicMetadata objects.

Related to T2734

To match the new behavior of package loaders.

The old attribute was deprecated in swh.model 0.7.2

swh.model.model.ModelObject.compute_hash was changed to a method instead of a
staticmethod.

migrate_extrinsic_metadata: Make pypi_origin_from_filename fix project names when possible using PyPI's API.

Instead of bloating handle_row, which is already way too long.

migrate_extrinsic_metadata: add support for guessing the origin of more PyPI packages from filenames.

It now supports all pypi revisions with an id starting with an hex digit
from 0 to 5.

This is actually what starts the server, so it sounds more reasonable to test
that part.

This also drops the type parameter from load_and_check_config which is never
used.

A future version of swh-journal will add a mandatory argument to JournalWriter,
whic get_journal_writer sets by default.

migrate_extrinsic_metadata: add support for the new deposit metadata formats introduced in late september.

* https://forge.softwareheritage.org/D4065
* https://forge.softwareheritage.org/D4105

Notably, `get_objstorage`'s `args` has been deprecated as of swh.objstorage
0.2.2.

This makes the backfiller configuration compatible with all other modules.

This allows more settings than the previous hardcoded three, e.g. the
`privileged` flag to backfill a journal containing anonymous topics.

Otherwise, Ctrl-C is ignored if pressed while sending a request.

migrate_extrinsic_metadata: add support for guessing the origin of more PyPI packages from filenames.

It now supports all pypi revisions with an id starting with 0, 1, or 2.

It was renamed in 4d72d1be529a568784842f5c0864e862a4b4705c.

migrate_extrinsic_metadata: Add support for the current format of original_artifacts written by the CRAN loader.

Default argument object_types was not properly declared in StorageInterface
and concrete implmentations PostgreSQL and Cassandra.
Reverted unnecessary fix in storage tests.

migrate_extrinsic_metadata: add support for guessing the origin of more PyPI packages from filenames.

It now supports all pypi revisions with an id starting with 0 or 1.

migrate_extrinsic_metadata: add support for guessing the origin of more PyPI packages from filenames.

This is used by swh.loader.core, regressed in v0.15.0 but wasn't caught by local
tests.

We stopped using it months ago.

- use more generic collection types, so that parametrized types can be
made stricter (e.g. str, in the next revision)
- remove Optionals that are not needed and provide better defaults

This should fix the currently failing documentation build.

Tenma authored 4 years ago and Nicolas Dandrimont committed 4 years ago

Use a more simple default value and do not identity check against it.

This uses a new database table and some psql conditionals to introduce three
different flavors for the swh.storage Postgres database:

 - the 'default' flavor has all the deduplication features, foreign keys and
 read indexes
 - the 'mirror' flavor has all the deduplication features and read indexes; it
 drops some foreign keys to allow for out of order addition of some object types
 - the 'read_replica' flavor has the minimal set of indexes to support
 read queries, and replication using the PostgreSQL logical replication feature

Related to T2604.