Skip to content
Snippets Groups Projects

Add a Python script to migrate extrinsic metadata from revision metadata.

Closed Add a Python script to migrate extrinsic metadata from revision metadata.
generated-differential-D3820-source into generated-differential-D3820-target
Closed vlorentz requested to merge generated-differential-D3820-source into generated-differential-D3820-target

Migrated from D3820 (view on Phabricator)

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build has FAILED

    Patch application report for D3820 (id=13444)

    Rebasing onto 40739076...

    Current branch diff-target is up to date.
    Changes applied before test
    commit 00ab89f26786b1b1b5c3711105f06e28f8f9c62a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 20 15:24:28 2020 +0200

    [WIP] Write a Python script to migrate extrinsic metadata from revision metadata.

    Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/833/ See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/833/console

  • vlorentz
    vlorentz @vlorentz ·
    Author Maintainer
    • rebase
    • add clause WHERE metadata IS NOT NULL (faster)
  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build is green

    Patch application report for D3820 (id=13449)

    Could not rebase; Attempt merge onto 3ac332ec...

    Updating 3ac332ec..8f7aab7e
Fast-forward
 bin/migrate-extrinsic-metadata.py  | 213 +++++++++++++++++++++++++++++++
 swh/storage/tests/storage_tests.py | 249 +------------------------------------
 2 files changed, 215 insertions(+), 247 deletions(-)
 create mode 100644 bin/migrate-extrinsic-metadata.py
    Changes applied before test
    commit 8f7aab7e23cd3a8c8f8c51f88d6ff3c7264cb964
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 20 15:24:28 2020 +0200

    [WIP] Write a Python script to migrate extrinsic metadata from revision metadata.

commit 5390a4cc0aad9182f62e8555db1efecba1f2dbc5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 20 16:04:30 2020 +0200

    storage_tests: Remove duplicated postgresql-specific tests.
    
    They got copied to test_postgresql.py instead of moved.

    See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/837/ for more details.

  • vlorentz
    vlorentz @vlorentz ·
    Author Maintainer

    Allow list as xmlns:codemeta

  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build is green

    Patch application report for D3820 (id=13456)

    Rebasing onto 5390a4cc...

    Current branch diff-target is up to date.
    Changes applied before test
    commit be641cf49fb2dfa6fb6fd78128797e7850d8df29
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 20 15:24:28 2020 +0200

    [WIP] Write a Python script to migrate extrinsic metadata from revision metadata.

    See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/840/ for more details.

  • vlorentz
    vlorentz @vlorentz ·
    Author Maintainer

    Connect to the deposit DB, and parse revision messages to get deposit ids.

  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build is green

    Patch application report for D3820 (id=13467)

    Rebasing onto d1f19e97...

    First, rewinding head to replay your work on top of it...
Applying: [WIP] Add a Python script to migrate extrinsic metadata from revision metadata.
    Changes applied before test
    commit 67d830d0c33059f92c618838214bfcb0f357349e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 20 15:24:28 2020 +0200

    [WIP] Add a Python script to migrate extrinsic metadata from revision metadata.

    See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/843/ for more details.

  • vlorentz
    vlorentz @vlorentz ·
    Author Maintainer

    add support for deposits with multiple metadata requests + allow missing 'committer' key.

  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build is green

    Patch application report for D3820 (id=13468)

    Rebasing onto d1f19e97...

    First, rewinding head to replay your work on top of it...
Applying: [WIP] Add a Python script to migrate extrinsic metadata from revision metadata.
    Changes applied before test
    commit 822f5f6088cfde7905162d08d03a44a71ff4e22c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 20 15:24:28 2020 +0200

    [WIP] Add a Python script to migrate extrinsic metadata from revision metadata.

    See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/844/ for more details.

  • vlorentz
    vlorentz @vlorentz ·
    Author Maintainer

    Go up to creating RawExtrinsicMetadata objects.

  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build is green

    Patch application report for D3820 (id=13478)

    Rebasing onto d1f19e97...

    First, rewinding head to replay your work on top of it...
Applying: [WIP] Add a Python script to migrate extrinsic metadata from revision metadata.
    Changes applied before test
    commit 0d9beac75a682ea153bb3118a874674781de0274
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 20 15:24:28 2020 +0200

    [WIP] Add a Python script to migrate extrinsic metadata from revision metadata.

    See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/845/ for more details.

  • vlorentz
    vlorentz @vlorentz ·
    Author Maintainer
    • properly deal with deposit dates
    • load original_artifact
  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build is green

    Patch application report for D3820 (id=13484)

    Could not rebase; Attempt merge onto d1f19e97...

    Merge made by the 'recursive' strategy.
 swh/storage/algos/snapshot.py             |  38 ++-
 swh/storage/migrate_extrinsic_metadata.py | 489 ++++++++++++++++++++++++++++++
 swh/storage/tests/algos/test_snapshot.py  |  61 +++-
 3 files changed, 581 insertions(+), 7 deletions(-)
 create mode 100644 swh/storage/migrate_extrinsic_metadata.py
    Changes applied before test
    commit 4a76950962845d1d394d7ef8b57952af707fde33
Merge: d1f19e97 3f843967
Author: Jenkins user <jenkins@localhost>
Date:   Mon Aug 24 12:37:29 2020 +0000

    Merge branch 'diff-target' into HEAD

commit 3f843967cf822782f7a2251632ba8551046ccb6a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 20 15:24:28 2020 +0200

    [WIP] Add a Python script to migrate extrinsic metadata from revision metadata.

commit a9c68fe060159c8a9667360e4e77956c0247337a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 24 12:49:16 2020 +0200

    algos.snapshot: Add visits_and_snapshots_get_from_revision
    
    Its code is moved from snapshot_id_get_from_revision so it's a rather small change;
    and the revision metadata migration script (bin/migrate-extrinsic-metadata.py)
    will need it.

    See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/847/ for more details.

  • vlorentz
    vlorentz @vlorentz ·
    Author Maintainer
    • Start adding the origin context
    • PyPI: add tests
  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build is green

    Patch application report for D3820 (id=13502)

    Could not rebase; Attempt merge onto cc33dd35...

    Merge made by the 'recursive' strategy.
 mypy.ini                                           |   3 +
 requirements.txt                                   |   1 +
 swh/storage/migrate_extrinsic_metadata.py          | 545 +++++++++++++++++++++
 .../tests/migrate_extrinsic_metadata/test_cran.py  | 108 ++++
 .../migrate_extrinsic_metadata/test_debian.py      | 234 +++++++++
 .../tests/migrate_extrinsic_metadata/test_pypi.py  | 150 ++++++
 6 files changed, 1041 insertions(+)
 create mode 100644 swh/storage/migrate_extrinsic_metadata.py
 create mode 100644 swh/storage/tests/migrate_extrinsic_metadata/test_cran.py
 create mode 100644 swh/storage/tests/migrate_extrinsic_metadata/test_debian.py
 create mode 100644 swh/storage/tests/migrate_extrinsic_metadata/test_pypi.py
    Changes applied before test
    commit 97db3d37dd7ef5da2eed7b3a53b8d4b85bae6fa4
Merge: cc33dd35 1eebe547
Author: Jenkins user <jenkins@localhost>
Date:   Mon Aug 24 15:34:54 2020 +0000

    Merge branch 'diff-target' into HEAD

commit 1eebe54758b8b6d924c409b1de111ef61b8cbf33
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 24 17:05:09 2020 +0200

    pypi: add test

commit 3c27604f2952a9959ce92831798a34bd22969148
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 24 17:04:53 2020 +0200

    Start adding the origin context

commit bd58b38651fb5155b8b4f13cd2692712b40b99a1
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 24 16:39:52 2020 +0200

    cran: add test

commit b5a42302fc39f2eaf07cfa771d7e631b0101fff7
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 24 16:39:34 2020 +0200

    cran: handle date

commit 0188692ab17fa50cbba58b4e1f98310fe8af5727
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 24 15:57:44 2020 +0200

    add tests for revisions generated by the debian loader.

commit 4bffbf19082c7c15068b66a0997b443fd4f13f97
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 20 15:24:28 2020 +0200

    [WIP] Add a Python script to migrate extrinsic metadata from revision metadata.

    See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/863/ for more details.

  • Antoine R. Dumont
    Antoine R. Dumont @ardumont ·
    Maintainer

    Heads up, the tested code is no longer included ;)

  • vlorentz
    vlorentz @vlorentz ·
    Author Maintainer
    • re-add commits
  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build is green

    Patch application report for D3820 (id=13516)

    Rebasing onto e8b1b216...

    Current branch diff-target is up to date.
    Changes applied before test
    commit 2a2f914d697fe93ae9b3cd9058a2b1f8b6b13714
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 24 17:05:09 2020 +0200

    pypi: add test

commit 1ab3a8d8c05adcdc5b1ff1791c66eb6df83aa09b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 24 17:04:53 2020 +0200

    Start adding the origin context

commit de7cd7e97aa951f786465e9add178a104a841422
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 24 16:39:52 2020 +0200

    cran: add test

commit bb43b24cba109ae8b2262d68c6ad56b3dde8c857
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 24 16:39:34 2020 +0200

    cran: handle date

commit 7c8ce23fd029d2f408ae27fb20ea4e19fccfdc77
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 24 15:57:44 2020 +0200

    add tests for revisions generated by the debian loader.

commit 688f664b4c5a74dad49241d05605e7278be77b41
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 20 15:24:28 2020 +0200

    [WIP] Add a Python script to migrate extrinsic metadata from revision metadata.

    See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/869/ for more details.

  • vlorentz
    vlorentz @vlorentz ·
    Author Maintainer
    • Rename original-artifact-json to original-artifacts-json.
    • npm format 2: build origins urls.
    • npm format 2: fix format of original_artifact.
    • npm: add tests
  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build is green

    Patch application report for D3820 (id=13523)

    Rebasing onto 4532a4dc...

    First, rewinding head to replay your work on top of it...
Applying: [WIP] Add a Python script to migrate extrinsic metadata from revision metadata.
Applying: add tests for revisions generated by the debian loader.
Applying: cran: handle date
Applying: cran: add test
Applying: Start adding the origin context
Applying: pypi: add test
Applying: Rename original-artifact-json to original-artifacts-json.
Applying: npm format 2: build origins urls.
Applying: npm format 2: fix format of original_artifact.
Applying: npm: add tests
    Changes applied before test
    commit 8c5162f0e7d56327e0ec2e944f3fa59aee8d0f50
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Aug 25 12:19:43 2020 +0200

    npm: add tests

commit bf141d27f23d7ef2af40ab2b5f4a7db50b812311
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Aug 25 12:19:31 2020 +0200

    npm format 2: fix format of original_artifact.

commit 0f603fe2dbf95ca934d56006a42605192ba28324
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Aug 25 12:18:56 2020 +0200

    npm format 2: build origins urls.

commit bd98d8ce5272958e4166c0efaad0a7521e9cc73e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Aug 25 12:17:47 2020 +0200

    Rename original-artifact-json to original-artifacts-json.
    
    As in the core loader.

commit fe7dedca82a520b7c03a8531b1eb9d437f7fcd48
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 24 17:05:09 2020 +0200

    pypi: add test

commit a63dc054bb827dcae4dc05d589f3c7b0d191abb5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 24 17:04:53 2020 +0200

    Start adding the origin context

commit b5af7bc04cbe9a609b276c79274ed61a5701638b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 24 16:39:52 2020 +0200

    cran: add test

commit 672cf25a0af8ddb51039cb3becf15958dbc744e6
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 24 16:39:34 2020 +0200

    cran: handle date

commit 306f7acdc29b5351c9e9b9b9c9581c8fee34ae4c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Mon Aug 24 15:57:44 2020 +0200

    add tests for revisions generated by the debian loader.

commit b83fd09bbad8d3f1c9f359871080add924a490bd
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Aug 20 15:24:28 2020 +0200

    [WIP] Add a Python script to migrate extrinsic metadata from revision metadata.

    See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/870/ for more details.

  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
Please register or sign in to reply