Skip to content
Snippets Groups Projects

Replace 'normalize' parameter of _translate_dict() with a hook method

Closed Replace 'normalize' parameter of _translate_dict() with a hook method
generated-differential-D8275-source into generated-differential-D8275-target
2 unresolved threads
Closed vlorentz requested to merge generated-differential-D8275-source into generated-differential-D8275-target
2 unresolved threads

This parameter was only used to execute extra code before normalize_translation is called. This caused some duplication, and will not work when switching to a non-JSON-LD internal representation.

Removing it also makes the code of mappings more consistent, by removing specific field handling from their implementation of the translate method itself.

Depends on !367 (closed).

Migrated from D8275 (view on Phabricator)

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build was aborted

    Patch application report for D8275 (id=29889)

    Could not rebase; Attempt merge onto 7d7f29fb...

    Updating 7d7f29f..c8a4571
Fast-forward
 swh/indexer/codemeta.py                     | 23 +++++++++----------
 swh/indexer/metadata_dictionary/base.py     | 28 +++++++++++++-----------
 swh/indexer/metadata_dictionary/cff.py      | 17 ++++++++-------
 swh/indexer/metadata_dictionary/composer.py |  9 ++++----
 swh/indexer/metadata_dictionary/dart.py     | 13 ++++++-----
 swh/indexer/metadata_dictionary/github.py   | 34 +++++++++++++----------------
 swh/indexer/metadata_dictionary/maven.py    | 25 +++++++++++++--------
 swh/indexer/metadata_dictionary/npm.py      | 11 +++++-----
 swh/indexer/metadata_dictionary/nuget.py    |  6 ++---
 swh/indexer/metadata_dictionary/python.py   | 22 ++++++++++---------
 swh/indexer/metadata_dictionary/ruby.py     |  7 +++---
 swh/indexer/namespaces.py                   | 28 ++++++++++++++++++++++++
 12 files changed, 130 insertions(+), 93 deletions(-)
 create mode 100644 swh/indexer/namespaces.py
    Changes applied before test
    commit c8a4571c8763c84c064e832f598e949407e0e429
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Aug 19 12:18:30 2022 +0200

    Replace 'normalize' parameter of _translate_dict() with a hook method
    
    This parameter was only used to execute extra code before
    `normalize_translation` is called. This caused some duplication, and
    will not work when switching to a non-JSON-LD internal representation.
    
    Removing it also makes the code of mappings more consistent, by removing
    specific field handling from their implementation of the `translate`
    method itself.

commit 466108c1667c88be7ff272e565ffe076e16064d8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Aug 19 12:13:33 2022 +0200

    python: Simplify translation logic of author metadata

commit 92b53419f6f9d699451609cb23a946978ecb6b07
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Aug 19 11:37:06 2022 +0200

    metadata_dictionary: Simplify code using rdflib-style namespace classes

    Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/436/ See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/436/console

  • Software Heritage Continuous Integration for public repositories
    Software Heritage Continuous Integration for public repositories @swh-public-ci ·

    Build is green

    Patch application report for D8275 (id=29889)

    Could not rebase; Attempt merge onto 7d7f29fb...

    Updating 7d7f29f..c8a4571
Fast-forward
 swh/indexer/codemeta.py                     | 23 +++++++++----------
 swh/indexer/metadata_dictionary/base.py     | 28 +++++++++++++-----------
 swh/indexer/metadata_dictionary/cff.py      | 17 ++++++++-------
 swh/indexer/metadata_dictionary/composer.py |  9 ++++----
 swh/indexer/metadata_dictionary/dart.py     | 13 ++++++-----
 swh/indexer/metadata_dictionary/github.py   | 34 +++++++++++++----------------
 swh/indexer/metadata_dictionary/maven.py    | 25 +++++++++++++--------
 swh/indexer/metadata_dictionary/npm.py      | 11 +++++-----
 swh/indexer/metadata_dictionary/nuget.py    |  6 ++---
 swh/indexer/metadata_dictionary/python.py   | 22 ++++++++++---------
 swh/indexer/metadata_dictionary/ruby.py     |  7 +++---
 swh/indexer/namespaces.py                   | 28 ++++++++++++++++++++++++
 12 files changed, 130 insertions(+), 93 deletions(-)
 create mode 100644 swh/indexer/namespaces.py
    Changes applied before test
    commit c8a4571c8763c84c064e832f598e949407e0e429
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Aug 19 12:18:30 2022 +0200

    Replace 'normalize' parameter of _translate_dict() with a hook method
    
    This parameter was only used to execute extra code before
    `normalize_translation` is called. This caused some duplication, and
    will not work when switching to a non-JSON-LD internal representation.
    
    Removing it also makes the code of mappings more consistent, by removing
    specific field handling from their implementation of the `translate`
    method itself.

commit 466108c1667c88be7ff272e565ffe076e16064d8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Aug 19 12:13:33 2022 +0200

    python: Simplify translation logic of author metadata

commit 92b53419f6f9d699451609cb23a946978ecb6b07
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Aug 19 11:37:06 2022 +0200

    metadata_dictionary: Simplify code using rdflib-style namespace classes

    See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/441/ for more details.

  • Antoine R. Dumont
    Antoine R. Dumont @ardumont started a thread on the diff
swh/indexer/metadata_dictionary/maven.py
41 41 if not isinstance(d, dict):
42 42 self.log.warning("Skipping ill-formed XML content: %s", content)
43 43 return None
44 metadata = self._translate_dict(d, normalize=False)
45 metadata[SCHEMA.codeRepository] = self.parse_repositories(d)
46 metadata[SCHEMA.license] = self.parse_licenses(d)
47 return self.normalize_translation(metadata)
44 return self._translate_dict(d)
48 45
49 46 _default_repository = {"url": "https://repo.maven.apache.org/maven2/"}
50 47
48 def extra_translation(self, translated_metadata, d):
  • Antoine R. Dumont
    Antoine R. Dumont @ardumont started a thread on the diff
    • swh/indexer/metadata_dictionary/maven.py
    102 104 if isinstance(id_, str):
    103 105 return {"@id": id_}
    104 106
    105 def parse_licenses(self, d):
    107 def translate_licenses(self, translated_metadata, d):
  • Antoine R. Dumont mentioned in merge request !399 (closed)

    mentioned in merge request !399 (closed)

  • Antoine R. Dumont
    Antoine R. Dumont @ardumont ·
    Maintainer

    couple of remarks inline.

  • Antoine R. Dumont
    Antoine R. Dumont @ardumont ·
    Maintainer

    Merge request was accepted

  • Antoine R. Dumont approved this merge request

    approved this merge request

  • vlorentz
    vlorentz @vlorentz ·
    Author Maintainer

    meh, it's Dict[str, Any] and Any everywhere, so not really worth it

  • vlorentz
    vlorentz @vlorentz ·
    Author Maintainer

    Merge request was merged

  • vlorentz closed

    closed

    • Please register or sign in to reply