Implement object check in base ObjStorage class instead of duplicating that code
in every object storage backend.

Add an ObjCorruptedError exception raised when an object corruption is detected.

Improve exception messages.

Normalize imports of exception classes.

Related to swh/devel/swh-scrubber#4694.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

Add a pytest plugin enabling to create a new object storage instance
for tests using the swh_objstorage fixture.

The type of object storage to create is determined by the configuration
returned by the swh_objstorage_config fixture.

Use these new fixtures to simplify a bit some tests implementation.

Related to swh/devel/swh-scrubber#4694.

This allows disabling ceph-based tests on systems where ceph is
available, by setting USE_CEPH=no.

First, wait for the image to reappear read-only before marking it as
mapped; secondd, raising the ShardNotMapped error if the Shard open
operation fails with a FileNotFoundError.

This allows printing internal winery stats at a given interval

This matches the production behavior.

While this was done, the improper mock of the rw_shard_cleaner return
value was noticed and fixed.

This cuts the operation into smaller batches, and each batch calls a
cheaper function.

This allows the benchmark to shorten the workload of the workers, rather
than wait for them to complete (potentially for a long time).

With a singleton PoolManager, with proper fork tracking and reference
counting to avoid leaking ConnectionPools

By keeping track of created databases, created tables, and connection
pools, we cut down drastically on the admin operations performed while
the code is running.

The transaction state of the PostgreSQL connection is attached to the
db-api connection object, not to the cursor (which is just a lightweight
object to represent the back-and-forth between a query and the results
of the queries). To share the transaction state, across functions, when
using connection pooling, we need to share the connection object.

In that specific case, the db attribute of the SharedBase object is
already reused between calls and holds the transaction state, so the
cursor passing is unnecessary.

This helped track down the various connection leaks that this code was
subject to when the benchmarks would run.

Try to avoid leaking database connections and open files.

This avoids interfering with the production settings if any.

Forcing a change of the EC profile has catastrophic data availability
consequences, so avoid doing that at all

The monotonic clock is decorrelated with real life, so it can't be
matched to statistics.

We run the benchmark workers in a process pool with fixed pids, so the
files would get clobbered on every worker restart. Instead, append to
the file by outputting a line of zeros to mark the reset.

These tables are append-only and short-lived, there's not much point
using resources to autovacuum them.

Remove the use of unittest.TestCase class.

Replace use of unittest assertions by pytest ones.

Use pytest autouse fixtures instead of setup/teardown methods.

Use tmpdir fixture to manage temporary directories.

Especially the main one in swh/objstorage needs to be removed since this
later is a namespace, in addition to being a package, in which
swh-objstorage-replayer installs itself. So for the namespace importing
mechanism to work properly, it must be an actual namespace (handled by
the NamespaceLoader, not a package backed by the __init__ file).

The idea is to allow using Azure's secondary endpoint
(BlobSecondaryEndpoint) to craft the download url the storage used as
public download URL.

This is needed for the integration test where the endpoint used to add
content in an azure-backend objstorage is different from the public URL
that can be used to download the blob directly from azure (azurite in the
context of integration tests).

Related to swh/meta#5075.

Jérémy Bobbio (Lunar) authored 1 year ago and Nicolas Dandrimont committed 1 year ago

We now allow to delete objects from Winery. The object is first marked
as deleted in the shared base. This makes it inaccessible from further
`get()` calls. If the object still in a RWShard, the relevant line in
the `objects` table is deleted as well.

To take care of removing the data in ROShards, a dedicated host able to
map images in read-write mode can then call:

    swh objstorage winery clean-deleted-objects

This will zero out any deleted object present in the images and remove
their key from the indices. Object keys will then be removed from the
shared database.

Thanks to olasd for proposing several improvements over the initial
submission.

Addresses swh-alter#4

Jérémy Bobbio (Lunar) authored 1 year ago and Nicolas Dandrimont committed 1 year ago

This paves the way to implement object deletion as another status.
(So we can later cleanup the shards in a batch process.)

Thanks to olasd for proposing several improvements over the initial
submission.

Jérémy Bobbio (Lunar) authored 1 year ago and Nicolas Dandrimont committed 1 year ago

As an alternative to running the tests with a Ceph backend, we
implement an alternative image pool based on simple files.
We represent unmapped images by setting their permissions to 0o000.

The tests are modified to run with both the file-backed image pool
and the Ceph pool. Those latter ones will be skipped if Ceph is not
available.

moto v5 rejigged the fixtures in a way that's incompatible with
swh.objstorage, pin it down for now.

Running `test_list_content*` on cloud implementations is pretty useless
as we won’t be listing 17 billion contents from Azure or S3 in a single
thread operation. As each one takes a minute per cloud, we skip these
tests to improve the runtime of the test suite.

The test suite is now reaching the C.I. timeout. In order to reduce the
overall runtime, we now skip testing most compression methods (bzip2,
zlib, lzma) in cloud objstorages by default. Testing all compression
methods can still be done by specifying `--all-compression-methods` on
pytest command-line.

diff --git a/swh/objstorage/tests/conftest.py b/swh/objstorage/tests/conftest.py
index 377778e..664aade 100644
--- a/swh/objstorage/tests/conftest.py
+++ b/swh/objstorage/tests/conftest.py
@@ -1,5 +1,7 @@
 import sys

+import pytest
+

 def pytest_configure(config):
     config.addinivalue_line("markers", "shard_max_size: winery backend")
@@ -14,6 +16,13 @@ def pytest_configure(config):
     config.addinivalue_line(
         "markers", "use_benchmark_flags: use the --winery-bench-* CLI flags"
     )
+    config.addinivalue_line(
+        "markers",
+        (
+            "all_compression_methods: "
+            "test all compression methods instead of only the most common ones"
+        ),
+    )

 def pytest_addoption(parser):
@@ -106,3 +115,15 @@ def pytest_addoption(parser):
         help="Maximum number of bytes per second write",
         default=100 * 1024 * 1024,
     )
+    parser.addoption(
+        "--all-compression-methods",
+        action="store_true",
+        default=False,
+        help="Test all compression methods",
+    )
+
+
+def pytest_runtest_setup(item):
+    if item.get_closest_marker("all_compression_methods"):
+        if not item.config.getoption("--all-compression-methods"):
+            pytest.skip("`--all-compression-methods` has not been specified")
diff --git a/swh/objstorage/tests/test_objstorage_azure.py b/swh/objstorage/tests/test_objstorage_azure.py
index b78e3d4..3679258 100644
--- a/swh/objstorage/tests/test_objstorage_azure.py
+++ b/swh/objstorage/tests/test_objstorage_azure.py
@@ -269,14 +269,17 @@ class TestMockedAzureCloudObjStorageGzip(TestMockedAzureCloudObjStorage):
     compression = "gzip"

+@pytest.mark.all_compression_methods
 class TestMockedAzureCloudObjStorageZlib(TestMockedAzureCloudObjStorage):
     compression = "zlib"

+@pytest.mark.all_compression_methods
 class TestMockedAzureCloudObjStorageLzma(TestMockedAzureCloudObjStorage):
     compression = "lzma"

+@pytest.mark.all_compression_methods
 class TestMockedAzureCloudObjStorageBz2(TestMockedAzureCloudObjStorage):
     compression = "bz2"

diff --git a/swh/objstorage/tests/test_objstorage_cloud.py b/swh/objstorage/tests/test_objstorage_cloud.py
index 49cbd62..1c50850 100644
--- a/swh/objstorage/tests/test_objstorage_cloud.py
+++ b/swh/objstorage/tests/test_objstorage_cloud.py
@@ -148,6 +148,7 @@ class TestCloudObjStorage(ObjStorageTestFixture, unittest.TestCase):
         pass

+@pytest.mark.all_compression_methods
 class TestCloudObjStorageBz2(TestCloudObjStorage):
     compression = "bz2"

@@ -156,10 +157,12 @@ class TestCloudObjStorageGzip(TestCloudObjStorage):
     compression = "gzip"

+@pytest.mark.all_compression_methods
 class TestCloudObjStorageLzma(TestCloudObjStorage):
     compression = "lzma"

+@pytest.mark.all_compression_methods
 class TestCloudObjStorageZlib(TestCloudObjStorage):
     compression = "zlib"

diff --git a/swh/objstorage/tests/test_objstorage_pathslicing.py b/swh/objstorage/tests/test_objstorage_pathslicing.py
index d1f5568..72c3305 100644
--- a/swh/objstorage/tests/test_objstorage_pathslicing.py
+++ b/swh/objstorage/tests/test_objstorage_pathslicing.py
@@ -8,6 +8,8 @@ import tempfile
 import unittest
 from unittest.mock import DEFAULT, patch

+import pytest
+
 from swh.model import hashutil
 from swh.objstorage import exc
 from swh.objstorage.constants import ID_DIGEST_LENGTH
@@ -144,13 +146,16 @@ class TestPathSlicingObjStorageGzip(TestPathSlicingObjStorage):
     compression = "gzip"

+@pytest.mark.all_compression_methods
 class TestPathSlicingObjStorageZlib(TestPathSlicingObjStorage):
     compression = "zlib"

+@pytest.mark.all_compression_methods
 class TestPathSlicingObjStorageBz2(TestPathSlicingObjStorage):
     compression = "bz2"

+@pytest.mark.all_compression_methods
 class TestPathSlicingObjStorageLzma(TestPathSlicingObjStorage):
     compression = "lzma"

Since migration to PEP 517, executing "make test" or "pytest" in the root
directory of the objstorage package triggers the following error:

ImportError: cannot import name 'add_filters' from 'swh.objstorage.multiplexer.filter'

Explicitly setting the testpaths option in pytest.ini and TEST_DIRS
variable in Makefile.local to swh/objstorage/tests fix the issue.