Start journal client to populate the origin_visit_stats model

This is only a function not plugged to anything yet. Remains the cli part to actually be able to use it.

Questions inline in the code as comment.

Related to #2967 (closed)

Test Plan

tox (failing until swh.model > 0.9 [1] is released)

• [1] swh-model!151 (closed)

---

Migrated from D4852 (view on Phabricator)

Build has FAILED

Patch application report for D4852 (id=17178)

Rebasing onto a6200339...

Current branch diff-target is up to date.

Changes applied before test

commit 52031853850394482c697fe2f9f634562cace31b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jan 13 13:03:28 2021 +0100

    Start journal client to populate the origin_visit_stats model
    
    Related to #2967

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/100/ See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/100/console

This needs more than introspecting the visit status information to determine the date field to update in origin_visit_stats.

Build has FAILED

Patch application report for D4852 (id=17179)

Rebasing onto a6200339...

Current branch diff-target is up to date.

Changes applied before test

commit ab9871d3f14ccc4510ef0912af45716259630878
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jan 13 13:03:28 2021 +0100

    Populate origin_visit_stats table out of the origin_visit_status topic
    
    Related to #2967

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/101/ See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/101/console

You should make origin_visit_stats_upsert take a list as arguments so that process_journal_objects can send them in batches.

Right now, it's doing one query + one commit for each stats, and it's the bottleneck of the simulator

Build is green

Patch application report for D4852 (id=17179)

Rebasing onto ca45d40f...

First, rewinding head to replay your work on top of it...
Applying: Populate origin_visit_stats table out of the origin_visit_status topic

Changes applied before test

commit a2a046e2bc67b455707888bca01c99af7cbdfc13
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jan 13 13:03:28 2021 +0100

    Populate origin_visit_stats table out of the origin_visit_status topic
    
    Related to #2967

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/122/ for more details.

Merge request was returned for changes

superseded by !318 (closed)

superseded by !318 (closed)

Merge request was abandoned

closed

mentioned in merge request !171 (closed)

mentioned in merge request !318 (closed)