Adapt content indexer to allow journal objects processing

This only adds the content mimetype indexer for now. Some extra mocking work is needed to test the fossology license one so it will go in another diff.

Note that it also refactors the tests dataset to stop hard-coding wrong ids and use proper hash from our model.

In another extra diff after that, we'll drop obsolete parts and refactor to simplify existing base code.

Related to #4273 (closed)

---

Migrated from D8147 (view on Phabricator)

Build has FAILED

Patch application report for D8147 (id=29415)

Rebasing onto fa67b73d...

Current branch diff-target is up to date.

Changes applied before test

commit 43ab37f7ad38ed4d07f5ab74643a8ba142be34b9
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 20 19:16:12 2022 +0200

    wip: Make content indexer process data out of the journal
    
    Related to #4273

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/364/ See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/364/console

fix type

Build has FAILED

Patch application report for D8147 (id=29416)

Rebasing onto fa67b73d...

Current branch diff-target is up to date.

Changes applied before test

commit 86162040212ba8193eb537b9cfa714815f189afe
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 20 19:16:12 2022 +0200

    wip: Make content indexer process data out of the journal
    
    Related to #4273

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/365/ See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/365/console

Adapt according to remarks

Build is green

Patch application report for D8147 (id=29423)

Rebasing onto d0d9346b...

First, rewinding head to replay your work on top of it...
Applying: Adapt content indexer to allow journal objects processing

Changes applied before test

commit 1cf9b6a4836cecca882c52093a38ffedbe05c5b2
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 20 19:16:12 2022 +0200

    Adapt content indexer to allow journal objects processing
    
    Related to #4273

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/367/ for more details.

Rebase

Build has FAILED

Patch application report for D8147 (id=29429)

Rebasing onto 7cf9cf59...

Current branch diff-target is up to date.

Changes applied before test

commit 410c2188eb981f6114c1d14b63d8250a29b5784a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 20 19:16:12 2022 +0200

    Adapt content indexer to allow journal objects processing
    
    Related to #4273

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/370/ See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/370/console

Fix messed up rebase

Build is green

Patch application report for D8147 (id=29430)

Rebasing onto 7cf9cf59...

Current branch diff-target is up to date.

Changes applied before test

commit 9c16a32833f72a5baac4fd06ecf6bc38eddb92f2
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 20 19:16:12 2022 +0200

    Adapt content indexer to allow journal objects processing
    
    Related to #4273

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/371/ for more details.

you'll need to update that

yes, thx, it's done locally. ongoing tests showed it to me ;)

Add tests

TODO: Unstuck regresssions in tests (for test reason, not for runtime change reasons)

Build has FAILED

Patch application report for D8147 (id=29444)

Could not rebase; Attempt merge onto 7cf9cf59...

Updating 7cf9cf5..0ac477b
Fast-forward
 swh/indexer/cli.py                          |  26 +-
 swh/indexer/indexer.py                      |  58 +++-
 swh/indexer/tests/conftest.py               |  15 +-
 swh/indexer/tests/test_cli.py               |  91 +++++-
 swh/indexer/tests/test_ctags.py             |  17 +-
 swh/indexer/tests/test_fossology_license.py |  20 +-
 swh/indexer/tests/test_metadata.py          |   6 +-
 swh/indexer/tests/test_mimetype.py          |  50 ++--
 swh/indexer/tests/utils.py                  | 427 ++++++++++++++--------------
 9 files changed, 440 insertions(+), 270 deletions(-)

Changes applied before test

commit 0ac477be06e8ec659a5ac501385b7cf933598e6a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jul 21 19:22:41 2022 +0200

    Add tests around new content-mimetype journal client indexer

commit 9c16a32833f72a5baac4fd06ecf6bc38eddb92f2
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 20 19:16:12 2022 +0200

    Adapt content indexer to allow journal objects processing
    
    Related to #4273

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/372/ See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/372/console

Add missing requirements which fails some tests on objstorage

Build has FAILED

Patch application report for D8147 (id=29447)

Rebasing onto 7cf9cf59...

Current branch diff-target is up to date.

Changes applied before test

commit 6736fcab0aad2b87b900fa3188f9c4bedd6857cb
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jul 21 19:22:41 2022 +0200

    Add tests around new content-mimetype journal client indexer

commit c7bdb5b4ec17ad0cdf242119c9c5d84bbc20b4ea
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 20 19:16:12 2022 +0200

    Adapt content indexer to allow journal objects processing
    
    Related to #4273

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/373/ See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/373/console

Fix tests (i've kept some setup in the fixture 'cause that made the tests fail and i don't want to dig in just right now)

Build has FAILED

Patch application report for D8147 (id=29448)

Rebasing onto 7cf9cf59...

Current branch diff-target is up to date.

Changes applied before test

commit 58c257a6c936e8b61b224838ab24610f5acb3caf
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jul 21 19:22:41 2022 +0200

    Add tests around new content-mimetype journal client indexer

commit f069be81d76a0bd39684d6d06c415ddf4a9047c4
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 20 19:16:12 2022 +0200

    Adapt content indexer to allow journal objects processing
    
    Related to #4273

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/374/ See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/374/console

Fix inconsistency in test dataset

Some references in the commit message have been migrated:

• T4273 is now #4273 (closed)

Keep fossology license indexer out of the diff for now. It will go in another diff.

just to clarify the call below (target is sha1_git but the actual content id is the sha1, see below).