Skip to content

deposit: Adapt loader to send extrinsic raw metadata to the metadata storage

Antoine R. Dumont requested to merge generated-differential-D4105-source into generated-differential-D4105-target

This keeps the metadata written in the revision in the same format as before though (json dict).

Questions:

  • (storage metadata) Do we keep the raw metadata attached to the revision as it's currently done or do we attach it to the directory instead or both?

-> currently the deposit server writes the raw metadata to the directory (during an update swh-deposit!377 (closed)), not on the revision as it cannot.

-> if we need to change it (attach the raw extrinsic metadata to the directory in the metadata storage), that's doable. It's a more involved refactoring though as the package loader does not allow it yet. And this can be done as a next diff.

  • (storage) Do we keep writing the metadata to the revision at all (we could stop now that the loaders are writing to the metadata storage)?

Remark: Implementation wise, this sends one raw extrinsic metadata column per raw xml received (a deposit can be done in multiple requests, which translates into multiple raw xml files). The first implementation of this was to join all xml into one concatenated xml. As this meant more handling down the line, when reading, this was changed into this instead.

Related to swh-deposit#2649 (closed)

Related to swh-deposit!388 (closed) Related to swh-deposit!389 (closed)

Test Plan

tox

Migrated from D4105 (view on Phabricator)

Merge request reports