Related P106
Related P107

To limit the dependencies on where it's needed

Quentin Campos authored 8 years ago and Antoine R. Dumont committed 8 years ago

Summary:
Create mock classes for the libcloud driver so the objstorage behavior can be checked.

Also, catch a libcloud exception and raise swh exception instead at the lower possible level in the code.

Reviewers: #reviewers, ardumont

Reviewed By: #reviewers, ardumont

Subscribers: ardumont

Differential Revision: https://forge.softwareheritage.org/D100

Related T545

Related T545

Related T545

Related T545

- remove --check-type and use check-type
- is_daemon to daemon

As this method is required in the ObjStorage abstract base class, the
test belongs to the fixture test class of this abstract class.

Make use of the newly factorized method compute_hash in ObjStorage to
compute hash in the cloud-based implementation.

At documentation generation, the docstring of the superclass will be used so there is no need to duplicate it or make reference to it.

The 'content' argument in the PathSlicingObjStorage.add method was
incorrectly named 'bytes'.

Hash computation code was duplicated in the add methods of ObjStorage
implementations. They are now factorized in a function.

ObjStorage is the main API of the object storage interface. Making it an
Abstract Base Class is highly usefull to catch mistakes when adding a
new implementation.

Also, some subclasses may use this feature to make their extension more easy.

The docstring of the ObjStorage implementations have been updated to
reference the ones present in the base class.

Amazon's S3 and OpenStack swift now have their own CloudObjStorage implementation.

Add an content integrity checker that update archive database : this implementation
set the status of the archival database corresponding to the real state
of the object. This way, next archival scheduling, the storage may be
used as a destination for a file copy.

The integrity checker now depends on an abstract class that contains the
base code that is necessary to perform the check on a storage.

This abstract class can be inherited by concret classes that will define
the checker behavior when a corrupted or missing content is find
(Note that for now, the checker runs on a local storage so the list of
content to check is made from walking the filesystem. So a missing
content shouldn't happend. But another version could get this list from
another source that is not the filesystem).

The PathSlicingObjStorage was exposed in the swh.objstorage top-level,
and is no longer a default implementation.
Classes using an ObjStorage should now use the get_objstorage method.

The previous api for the client was not aligned with the more recent
base API for locals objstorages.

Add the get_batch method to the base API so it is now possible to
perform a get of multiple contents at the same time.
This should be an improvement for the remote objstorage.

The method get_objstorage at the module top_level allows to easily
create an ObjStorage given a unique class key and a dictionary of
matching arguments.