- Only read the storage key once.
- Improve the logging error.

Also, adds a force_copy flag in the configuration file to avoid checking
preexistence of sha1.

This is to be efficient for the first time copy in a new backend.

The initial director and worker had split configuration files.
Now it's unified to be both archiver/worker.yml file

Drop unused archive.url column.

Simplify associated tests setup on the archiver with retention policy.

Actions storage:
- Open cache_content_get to retrieve contents in cache
- sql/upgrades/080: Add stored procedure to read contents from cache

Actions archiver:
- d/control: Add dependency to archiver on python3-swh.objstorage.cloud
- Renamed Archiver(Director|Worker) to
ArchiverWithRetentionPolicy(Director|Worker)
- Add ArchiverToBackend(Director|Worker)
- Add new celery task dedicated for new workers
- Update docstring details

Related T555

As for python3-swh.storage.archiver, the runtime dependency on
python3-swh.core is the implicit one from python3-swh.storage.

The same note about splitting repository and adding explicit dependency
applies.

Related T557

content_archive.num_present act like a cache for the number of copies
where the content is present. This allow to make a much faster request
when retrieving the contents that have fewer copies than expected.

Additional configuration was passed from director to the worker at
initialization. We remove it so the queued messages in celery are only
lists of sha1 with no dict to pack/unpack.

The arguments for these objects are now parsed from a configuration file instead of being passed as rguments

Those changes allow the archiver to use any storage as a source or a destination making it the central point of all archival.
Also, the archiver's mecanisms and code are simplified.

See T512 for new architecture.
Closes T512

Reviewers: #reviewers!, olasd, zack

Maniphest Tasks: T512

Differential Revision: https://forge.softwareheritage.org/D88

Before the archiver copy a content, it have to check if it is valid. If it's not the case, the content status is updated to reflect his invalid state.

Summary:
As the api of RemoteObjStorage have been fixed (See D83) and is now
aligned to the base ObjStorage one, places where the remote API were
used directly must be updated.

Related D83

Reviewers: #reviewers!, olasd

Maniphest Tasks: T513

Differential Revision: https://forge.softwareheritage.org/D84

Make the required changes into the db and keep up-to-date the archiver's tests.

Related T412

Related D78
Closes T499

Quentin Campos authored 8 years ago and Antoine R. Dumont committed 8 years ago

Summary:
Related T499

test.archiver: update the tests to follow changes in the archiver

Reviewers: #reviewers, qcampos

Reviewed By: #reviewers

Maniphest Tasks: T499

Differential Revision: https://forge.softwareheritage.org/D78

Closes D78

Also, fix typos in sql schema and data definition