Skip to content
Snippets Groups Projects
Forked from Platform / Development / swh-indexer
231 commits behind the upstream repository.
Valentin Lorentz's avatar
vlorentz authored
Simpler code and less error-prone.
cd42c667
History

swh-indexer

Tools to compute multiple indexes on SWH's raw contents:

  • content:
    • mimetype
    • ctags
    • language
    • fossology-license
    • metadata
  • revision:
    • metadata

An indexer is in charge of:

  • looking up objects
  • extracting information from those objects
  • store those information in the swh-indexer db

There are multiple indexers working on different object types:

  • content indexer: works with content sha1 hashes
  • revision indexer: works with revision sha1 hashes
  • origin indexer: works with origin identifiers

Indexation procedure:

  • receive batch of ids
  • retrieve the associated data depending on object type
  • compute for that object some index
  • store the result to swh's storage

Current content indexers:

  • mimetype (queue swh_indexer_content_mimetype): detect the encoding and mimetype

  • language (queue swh_indexer_content_language): detect the programming language

  • ctags (queue swh_indexer_content_ctags): compute tags information

  • fossology-license (queue swh_indexer_fossology_license): compute the license

  • metadata: translate file into translated_metadata dict

Current revision indexers:

  • metadata: detects files containing metadata and retrieves translated_metadata in content_metadata table in storage or run content indexer to translate files.