Get skipped content that are missing data

In reference to #1633 (closed)

Similar to db call skipped_content_missing, the function checks for content in memory storage of skipped content

---

Migrated from D1693 (view on Phabricator)

Build is green See https://jenkins.softwareheritage.org/job/DSTO/job/tox/543/ for more details.

This logic is similar to one being used in db.skipped_content_missing. However the current implementation in_memory._content_add does not populate _skipped_contents and _skipped_content_indexes.

I think there are two ways to do it:

• I can modify _content_add however I am not sure how to check decide whether content should be skipped. Is it by checking the algorithm used for hash or the length of content, if so what is the limit?
• I can add a new function that explicitly only adds skipped contents.

It should be implemented to behave like in the postgresql storage (storage.py and db.py), so you should change _content_add, and use the same algorithm.

I have a concern here, storage.py line 120. The function self.content_missing can throw an exception in case of a hash collision. Shouldn't line 120 be in a try except block to catch that error and ignore that particular content?

Secondly, I don't fully understand what it means for content to be hidden or absent when can this happen?

In db.py line 128, the query does not compare blake2s256 despite content_hash_keys = ['sha1', 'sha1_git', 'sha256', 'blake2s256']. Does this mean that skipped content will never be hashed with blake2s256?

I did not find any mechanism in db.py that is actually storing skipped_content. db.py line 51, is passed without implementation.

For in memory content_add, I am adding skipped_content, similar to how regular content is being added.

• Modify in_memory content_add to add skipped_content

• Remove dependency

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/556/ See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/556/console

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/557/ See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/557/console

• Use all hashes in a content

Build is green See https://jenkins.softwareheritage.org/job/DSTO/job/tox/559/ for more details.

• Add break to prevent multiple yields

Build is green See https://jenkins.softwareheritage.org/job/DSTO/job/tox/560/ for more details.

Thanks! As I mentioned on IRC, you should un-skip test_skipped_content_add in swh/storage/tests/test_in_memory.py, otherwise all this new code does not get tested at all.

Merge request was returned for changes

• Change index storage mechanism

Build is green See https://jenkins.softwareheritage.org/job/DSTO/job/tox/568/ for more details.

Rebase and update

Build is green See https://jenkins.softwareheritage.org/job/DSTO/job/tox/569/ for more details.