Skip to content

metadata-indexer: Configuration tool creating multiple different tools even though the same

At the moment, there is something off in the tool configuration for the metadata indexer. The tool referenced for the metadata indexer associates a 'dynamic' context. Thus, adding unnecessary new tools even though they are the same.

See below for an extract [1]

A priori, the solution would be to remove the context key from the tool_configuration column (seen with @vlorentz).

In any case, the impacts i foresee are:

  • fix the code (according to the desired solution)

  • fix the associated puppet manifest tool configuration

  • sql scripts to migrate the data from the swh-indexer db (the indexer_configuration below should be merged where it makes sense and then the revision_metadata entries should be updated to link to the right indexer_configuration_id).

  • [1]

    id     |        tool_name        | tool_version |                                                    tool_configuration
-----------+-------------------------+--------------+---------------------------------------------------------------------------------------------------------------------------
  74460503 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["NpmMapping", "CodemetaMapping"]}
  74460516 | swh-metadata-translator | 0.0.2        | {"type": "local", "context": "NpmMapping"}
  74496667 | swh-metadata-translator | 0.0.2        | {"type": "local", "context": "MavenMapping"}
  74505359 | swh-metadata-translator | 0.0.2        | {"type": "local", "context": "PythonPkginfoMapping"}
  74608577 | swh-metadata-translator | 0.0.2        | {"type": "local", "context": "CodemetaMapping"}
  79181228 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["NpmMapping", "MavenMapping", "CodemetaMapping", "GemspecMapping", "PythonPkginfoMapping"]}
  79181505 | swh-metadata-translator | 0.0.2        | {"type": "local", "context": "GemspecMapping"}
  79182115 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["NpmMapping", "PythonPkginfoMapping", "MavenMapping", "CodemetaMapping", "GemspecMapping"]}
  79183761 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["GemspecMapping", "NpmMapping", "MavenMapping", "PythonPkginfoMapping", "CodemetaMapping"]}
  79183783 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["PythonPkginfoMapping", "MavenMapping", "NpmMapping", "GemspecMapping", "CodemetaMapping"]}
  79187154 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["MavenMapping", "PythonPkginfoMapping", "GemspecMapping", "NpmMapping", "CodemetaMapping"]}
  79187160 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["GemspecMapping", "MavenMapping", "NpmMapping", "CodemetaMapping", "PythonPkginfoMapping"]}
  79187161 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["PythonPkginfoMapping", "GemspecMapping", "MavenMapping", "CodemetaMapping", "NpmMapping"]}
  79187163 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["NpmMapping", "CodemetaMapping", "PythonPkginfoMapping", "MavenMapping", "GemspecMapping"]}
  79187164 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["MavenMapping", "CodemetaMapping", "GemspecMapping", "NpmMapping", "PythonPkginfoMapping"]}
  79187165 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["NpmMapping", "MavenMapping", "GemspecMapping", "PythonPkginfoMapping", "CodemetaMapping"]}
  79187166 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["NpmMapping", "CodemetaMapping", "PythonPkginfoMapping", "GemspecMapping", "MavenMapping"]}
  79187167 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["PythonPkginfoMapping", "CodemetaMapping", "MavenMapping", "GemspecMapping", "NpmMapping"]}
  79187631 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["NpmMapping", "MavenMapping", "PythonPkginfoMapping", "GemspecMapping", "CodemetaMapping"]}
  79194567 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["CodemetaMapping", "PythonPkginfoMapping", "NpmMapping", "GemspecMapping", "MavenMapping"]}
  79202057 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["GemspecMapping", "NpmMapping", "CodemetaMapping", "MavenMapping", "PythonPkginfoMapping"]}
  79211856 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["CodemetaMapping", "GemspecMapping", "PythonPkginfoMapping", "NpmMapping", "MavenMapping"]}
  79211865 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["CodemetaMapping", "MavenMapping", "NpmMapping", "GemspecMapping", "PythonPkginfoMapping"]}
  79211870 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["PythonPkginfoMapping", "NpmMapping", "CodemetaMapping", "MavenMapping", "GemspecMapping"]}
  79211877 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["CodemetaMapping", "PythonPkginfoMapping", "MavenMapping", "GemspecMapping", "NpmMapping"]}
  79211883 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["PythonPkginfoMapping", "NpmMapping", "MavenMapping", "GemspecMapping", "CodemetaMapping"]}
  79211894 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["PythonPkginfoMapping", "NpmMapping", "GemspecMapping", "MavenMapping", "CodemetaMapping"]}
  79211895 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["NpmMapping", "CodemetaMapping", "GemspecMapping", "PythonPkginfoMapping", "MavenMapping"]}
  79211902 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["MavenMapping", "NpmMapping", "GemspecMapping", "CodemetaMapping", "PythonPkginfoMapping"]}
  79211940 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["MavenMapping", "PythonPkginfoMapping", "GemspecMapping", "CodemetaMapping", "NpmMapping"]}
  79211954 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["CodemetaMapping", "MavenMapping", "PythonPkginfoMapping", "NpmMapping", "GemspecMapping"]}
  79268197 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["CodemetaMapping", "PythonPkginfoMapping", "GemspecMapping", "NpmMapping", "MavenMapping"]}
  79275070 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["NpmMapping", "PythonPkginfoMapping", "CodemetaMapping", "GemspecMapping", "MavenMapping"]}
  79276366 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["GemspecMapping", "MavenMapping", "PythonPkginfoMapping", "CodemetaMapping", "NpmMapping"]}
  79280294 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["PythonPkginfoMapping", "GemspecMapping", "NpmMapping", "MavenMapping", "CodemetaMapping"]}
  79286793 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["CodemetaMapping", "NpmMapping", "MavenMapping", "PythonPkginfoMapping", "GemspecMapping"]}
  79287831 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["GemspecMapping", "MavenMapping", "CodemetaMapping", "PythonPkginfoMapping", "NpmMapping"]}
  79289245 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["PythonPkginfoMapping", "GemspecMapping", "CodemetaMapping", "NpmMapping", "MavenMapping"]}
  79346055 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["MavenMapping", "GemspecMapping", "CodemetaMapping", "PythonPkginfoMapping", "NpmMapping"]}
  79346957 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["NpmMapping", "PythonPkginfoMapping", "CodemetaMapping", "MavenMapping", "GemspecMapping"]}
  79347770 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["MavenMapping", "CodemetaMapping", "PythonPkginfoMapping", "GemspecMapping", "NpmMapping"]}
  79348677 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["CodemetaMapping", "PythonPkginfoMapping", "NpmMapping", "MavenMapping", "GemspecMapping"]}
  79349895 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["CodemetaMapping", "GemspecMapping", "MavenMapping", "NpmMapping", "PythonPkginfoMapping"]}
  79350848 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["CodemetaMapping", "NpmMapping", "PythonPkginfoMapping", "MavenMapping", "GemspecMapping"]}
  79351864 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["CodemetaMapping", "PythonPkginfoMapping", "GemspecMapping", "MavenMapping", "NpmMapping"]}
  79352739 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["PythonPkginfoMapping", "CodemetaMapping", "GemspecMapping", "MavenMapping", "NpmMapping"]}
  80838260 | swh-metadata-detector   | 0.0.2        | {"type": "local", "context": ["GemspecMapping", "NpmMapping", "MavenMapping", "CodemetaMapping", "PythonPkginfoMapping"]}
...

Migrated from T1540 (view on Phabricator)

Edited by Phabricator Migration user