Skip to content

writer/inmemory: Disable shared memory use by default

Antoine Lambert requested to merge anlambert/swh-journal:in-memory-backend-optional-shared-memory into master

Turn shared memory use optional in journal writer memory backend and disable its use by default.

Such backend is typically created in tests of swh packages but only a single instance is used so enabling shared memory is not required.

This brings a great speedup when executing tests using a journal writer with memory backend but also prevent flaky tests.

Before this change when executing swh-storage tests:

(swh) ✔ ~/swh/swh-environment/swh-storage [master|⚑ 8] 
11:09 $ make test
python3 -m pytest --hypothesis-profile=fast ./swh/storage/tests
...
========================================================================================================= 1884 passed, 39 skipped, 5 xfailed, 64 warnings in 714.62s (0:11:54) ==========================================================================================================

After this change:

(swh) ✔ ~/swh/swh-environment/swh-storage [master|⚑ 8] 
12:26 $ make test
python3 -m pytest --hypothesis-profile=fast ./swh/storage/tests
...
========================================================================================================= 1884 passed, 39 skipped, 5 xfailed, 64 warnings in 508.55s (0:08:28) ==========================================================================================================

Also it enables to remove some flaky swh-storage tests on Jenkins like the one below:

10:52:35  ==================================== ERRORS ====================================
10:52:35  _ ERROR at setup of TestTenaciousStorage.test_snapshot_branch_get_by_name_dangling_branch _
10:52:35  
10:52:35  swh_storage_backend_config = {'cls': 'memory', 'journal_writer': {'cls': 'memory'}}
10:52:35  
10:52:35      @pytest.fixture
10:52:35      def swh_storage_backend(swh_storage_backend_config):
10:52:35          """
10:52:35          By default, this fixture aliases ``swh_storage``. However, when ``swh_storage``
10:52:35          is overridden to be a proxy storage, this fixture returns the storage instance
10:52:35          behind all proxies.
10:52:35      
10:52:35          This is useful to introspect the state of backends from proxy tests"""
10:52:35  >       return get_storage(**swh_storage_backend_config)
10:52:35  
10:52:35  .tox/py37/lib/python3.7/site-packages/swh/storage/pytest_plugin.py:332: 
10:52:35  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
10:52:35  .tox/py37/lib/python3.7/site-packages/swh/storage/__init__.py:81: in get_storage
10:52:35      storage = Storage(**kwargs)
10:52:35  .tox/py37/lib/python3.7/site-packages/swh/storage/in_memory.py:890: in __init__
10:52:35      self.journal_writer = JournalWriter(journal_writer)
10:52:35  .tox/py37/lib/python3.7/site-packages/swh/storage/writer.py:56: in __init__
10:52:35      value_sanitizer=model_object_dict_sanitizer, **journal_writer
10:52:35  .tox/py37/lib/python3.7/site-packages/swh/journal/writer/__init__.py:66: in get_journal_writer
10:52:35      return JournalWriter(**kwargs)
10:52:35  .tox/py37/lib/python3.7/site-packages/swh/journal/writer/inmemory.py:25: in __init__
10:52:35      self.manager = Manager()
10:52:35  /home/jenkins/.pyenv/versions/3.7.16/lib/python3.7/multiprocessing/context.py:56: in Manager
10:52:35      m.start()
10:52:35  /home/jenkins/.pyenv/versions/3.7.16/lib/python3.7/multiprocessing/managers.py:567: in start
10:52:35      self._address = reader.recv()
10:52:35  /home/jenkins/.pyenv/versions/3.7.16/lib/python3.7/multiprocessing/connection.py:250: in recv
10:52:35      buf = self._recv_bytes()
10:52:35  /home/jenkins/.pyenv/versions/3.7.16/lib/python3.7/multiprocessing/connection.py:407: in _recv_bytes
10:52:35      buf = self._recv(4)
10:52:35  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
10:52:35  
10:52:35  self = <multiprocessing.connection.Connection object at 0x7f338c1ff290>
10:52:35  size = 4, read = <built-in function read>
10:52:35  
10:52:35      def _recv(self, size, read=_read):
10:52:35          buf = io.BytesIO()
10:52:35          handle = self._handle
10:52:35          remaining = size
10:52:35          while remaining > 0:
10:52:35              chunk = read(handle, remaining)
10:52:35              n = len(chunk)
10:52:35              if n == 0:
10:52:35                  if remaining == size:
10:52:35  >                   raise EOFError
10:52:35  E                   EOFError
10:52:35  
10:52:35  /home/jenkins/.pyenv/versions/3.7.16/lib/python3.7/multiprocessing/connection.py:383: EOFError

Merge request reports