Limit the number of entries in the cache

The implementation test after each revision if there are more than 100 000 entries of any kind in the cache. If yes, it flush the content. It's quite naive but the memory seems to stay around 5Go max on big repositories like linux.

This can be still an issue with the current zeromq client because the origin are sorted by repositories a lot of snapshots of big repositories can be ingested in parallel. With the journal implementation, it should be more distributed on repositories of different sizes

Related to swh/infra/sysadm-environment#4313 (closed)

---

Migrated from D8040 (view on Phabricator)

swh/provenance/interface.py

-# Copyright (C) 2021  The Software Heritage developers
+# Copyright (C) 2021-2022  The Software Heritage developers
 # See the AUTHORS file at the top-level directory of this distribution
 # License: GNU General Public License version 3, or any later version
 # See top-level LICENSE file for more information
@@ -249,6 +249,13 @@ class ProvenanceInterface(Protocol):
         """Flush internal cache to the underlying `storage`."""
         ...
 
+    def flush_if_necessary(self) -> bool:
+        """Flush internal cache to the underlying `storage`, if the cache reached
+        a threshold (MAX_CACHE_ELEMENTS).
+        Return True if the cache is flushed, false otherwise.
+        """
+        ...
+
     def content_add_to_directory(
         self, directory: DirectoryEntry, blob: FileEntry, prefix: bytes
     ) -> None: