All the current swh.provenance changes that are running in production... (!143) · Merge requests · Platform / Development / swh-provenance

Nicolas Dandrimont requested to merge generated-differential-D8243-source into generated-differential-D8243-target Aug 12, 2022
this is the accumulation of changes that have been tested in production on mmca and met over the past weeks. A lot of this has been pair-programmed, and the tests still pass, so we're probably in good shape.
git log origin/master.. says:
commit 8f476d494b4aeab6e0cd6a7adb5f2bce095e8c60
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Aug 12 13:37:12 2022 +0200

    swhgraph: handle empty responses

    When the visit_edges response is empty, swh.graph.client generates an
    empty tuple, which can't be unpacked. Work around the issue.

 swh/provenance/swhgraph/archive.py | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

commit edf00f88894fb9cf407017944dc5cd751b012357
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Aug 12 13:36:39 2022 +0200

    Use proper signatures in journal_client

    We're always passing the provenance-internal object types, not those of
    swh.storage.

 swh/provenance/journal_client.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

commit 08de80b680bdf008f9a1f45805f2d54a7a397549
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Aug 12 11:26:01 2022 +0200

    origin layer: retrieve multiple levels of revision history at once

    Replace `revision_get_parents` with `revision_get_some_outbound_edges`,
    which can optionally retrieve more levels of history than just a single
    one. This allows us to do way fewer queries on the swh.graph or
    swh.storage backend if the revision exists there.

    The swh.storage backend does limited recursion, so we still process the
    origin in multiple steps to fetch the whole history.

 swh/provenance/archive.py                      | 15 +++++----
 swh/provenance/graph.py                        | 43 +++++++++++++-------------
 swh/provenance/interface.py                    | 10 +++---
 swh/provenance/journal_client.py               |  1 -
 swh/provenance/model.py                        | 20 +-----------
 swh/provenance/multiplexer/archive.py          | 28 ++++++++++-------
 swh/provenance/origin.py                       | 26 ++++++----------
 swh/provenance/postgresql/archive.py           | 27 +++++++++-------
 swh/provenance/provenance.py                   | 28 ++++++++---------
 swh/provenance/storage/archive.py              | 12 ++++---
 swh/provenance/swhgraph/archive.py             | 23 ++++++++------
 swh/provenance/tests/test_archive_interface.py | 29 ++++++++++-------
 12 files changed, 130 insertions(+), 132 deletions(-)

commit 68e1907e7f37863d732edcb6211be893df94b9c7
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Aug 12 11:03:12 2022 +0200

    Appease pyright by ensuring target_type is bound

 swh/provenance/tests/test_archive_interface.py | 2 ++
 1 file changed, 2 insertions(+)

commit d935abf431df5105fec8422e87eb5ee47d3c177a
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Aug 12 11:01:37 2022 +0200

    Rename origin.proceed_origin to origin.process_origin

 swh/provenance/origin.py | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

commit 2ac46f58346f7c3763f1263109885fea6797e155
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Aug 3 18:21:32 2022 +0200

    multiplexer: add endpoint counts per backend

 swh/provenance/__init__.py                     |  6 ++-
 swh/provenance/multiplexer/archive.py          | 61 +++++++++++++++++++-------
 swh/provenance/tests/test_archive_interface.py |  4 +-
 swh/provenance/tests/test_init.py              |  6 ++-
 4 files changed, 57 insertions(+), 20 deletions(-)

commit 8d323c322df2bf9a429a1329de6c87636927df19
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Aug 12 17:46:20 2022 +0200

    journal client: only use the provenance context manager once

    The context manager for the provenance storage rabbitmq client doesn't
    like being used multiple times over the lifetime of a process. Only use
    it once in the cli of the journal client.

 swh/provenance/cli.py            | 6 ++++--
 swh/provenance/journal_client.py | 6 ++----
 2 files changed, 6 insertions(+), 6 deletions(-)

commit f5f8555f8e3d8c72a5d51f4a10d0b761e74c97fe
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Aug 12 17:45:27 2022 +0200

    provenance: lower the cache thresholds

    Instead of flushing if any entry is over the threshold, flush when the
    cumulative count goes over.

 swh/provenance/provenance.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

commit 4b3de6177b4f2c5b45dede931004c719fdfb0f7d
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Aug 12 17:44:47 2022 +0200

    revision: only trigger partial flushes when necessary

 swh/provenance/revision.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

commit 9c936c39779cdb42b0f8f1a40df23d2de3032dfb
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Aug 12 17:43:13 2022 +0200

    revision: sort batches by date, improve logging, add incremental flushing

 swh/provenance/revision.py | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

commit 5b66b98e62c50c5958936adcc3b0ab651fb2d279
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Aug 12 17:40:59 2022 +0200

    revision: capture datetime exceptions with sentry

 swh/provenance/journal_client.py | 3 +++
 1 file changed, 3 insertions(+)

commit af09058f0a80aac79a4e477fb2f7bd9800e3603f
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Aug 12 17:40:26 2022 +0200

    revision: don't process revisions before the epoch

 swh/provenance/journal_client.py | 7 +++++++
 1 file changed, 7 insertions(+)

commit 3473d4af62d85255845aafc1def6c591090062e7
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Aug 12 17:39:30 2022 +0200

    revision: don't process revisions with unknown dates

 swh/provenance/journal_client.py | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)

commit d7d0c3d876059abe6a1d60a6c38ed4245e1b58c9
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Aug 12 17:34:21 2022 +0200

    postgresql archive: add support for partially copied databases

    The incremental copy of the archive to mmca is not atomic: the directory
    table needs to be copied first, then the directory_entry_* tables need
    to be updated. This means that the client can view inconsistent entries,
    where the directory has been synced but not all the entry rows.

    We return an empty list when one of these bogus entries is detected.
    This allows smooth fallback to the main database through the
    multiplexer.

 swh/provenance/postgresql/archive.py | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

commit 95eb9622a00ce99d089bb9accdaed0bdbf1bdc37
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Aug 12 17:33:28 2022 +0200

    postgresql archive: don't use custom types

    The partial copy of the archive on mmca doesn't have them anyway.

 swh/provenance/postgresql/archive.py | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

commit 34a9a1ac220bfabdda26b243c79742bdab090d76
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Aug 12 17:32:09 2022 +0200

    Remove sneaky caches in the postgresql archive implementation

 mypy.ini                             | 3 ---
 requirements.txt                     | 1 -
 swh/provenance/postgresql/archive.py | 3 ---
 3 files changed, 7 deletions(-)

commit bae8f4afda455ca28e64e54f1c9c37c6af2214b6
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Aug 12 17:29:45 2022 +0200

    rabbitmq: Extend timeouts for reception of acks

    The retry logic is not very refined, extending the timeouts makes more
    sense.

 swh/provenance/api/client.py | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

commit 1efc40c7917feaedfa1204b6e4e395d41530d14c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Aug 12 17:28:31 2022 +0200

    rabbitmq: close the consumer only after all acks are received

    This is not quite working but it seems to reduce issues on worker
    termination a bit.

 swh/provenance/api/client.py | 63 ++++++++++++++++++++++++++++----------------
 1 file changed, 41 insertions(+), 22 deletions(-)

commit ef7cd991712e47a14d7877f726f427a9de22e545
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Aug 12 17:14:58 2022 +0200

    Improve logging in the API client and the revision layer

 swh/provenance/api/client.py | 39 +++++++++++++++++++++++----------------
 swh/provenance/provenance.py |  2 +-
 swh/provenance/revision.py   | 12 ++++++++++++
 3 files changed, 36 insertions(+), 17 deletions(-)

commit 3edf3690258b9e61de5452967c6ee178120276e7
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Aug 12 16:53:11 2022 +0200

    Add systemd notification support

 mypy.ini                         |  3 +++
 swh/provenance/cli.py            | 15 +++++++++++++++
 swh/provenance/journal_client.py |  9 +++++++++
 3 files changed, 27 insertions(+)

commit 5cadb13de9eb27b309d2ada3df54dc86452785b3
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Aug 12 16:54:27 2022 +0200

    Try to avoid some circular imports

 swh/provenance/__init__.py   | 2 +-
 swh/provenance/api/server.py | 3 ++-
 2 files changed, 3 insertions(+), 2 deletions(-)

commit 98254d2e930f639c7b1fdb3c27f5eb2a668b857d
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Aug 12 17:17:11 2022 +0200

    blacken swhgraph/archive.py

 swh/provenance/swhgraph/archive.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
Test Plan

tests pass, and the journal clients seem happy enough...
Migrated from D8243 (view on Phabricator)
All the current swh.provenance changes that are running in production...

Test Plan

Merge request reports
