We now depend on swh.objstorage.interface.objid_from_dict

vlorentz authored 1 month ago and vlorentz committed 1 month ago

https://gitlab.softwareheritage.org/swh/devel/swh-loader-metadata/-/blob/c2cdfd5c86d2783ce28c80f84df4930c2eba3cfc/swh/loader/metadata/gitea.py#L121-138

corresponding spec update: swh-storage!1163

The previous bulk migration left some `sha1` variables containing
composite object identifiers, making following the logic a bit
precarious.

Bump development tools: mypy, codespell, isort, ...

Move all tools configuration in pyproject.toml.

Remove no longer needed mypy overrides.

Those were spotted thanks to the sphinx linkcheck builder.

This reverts commit 2a1c8db8.

The new rdflib 7.1.3 release reverted the typing changes they
introduced in 7.1.2 to keep compatibility with Python 3.8, so
revert the commit that adapted typings to rdflib 7.1.2.

Previously, a generated bibtex citation was using REPLACEME as placeholder
for identifier.

Prefer to generate a default citation identifier from SWHID if provided.

This should be enough to fully support Codemeta v3, to the extent of what properties
we can actually export to BibTeX

This does not mean we fully support CodeMeta 3.0 at the time of writing
but it should fix most of the bibtex citation generations when that URL
is used as context in codemeta.json file.

Fixes #4822.

Previously, content indexers were only using content sha1 as object
identifier but this could lead to errors in production when attempting
to fetch content data from an object storage where key is the sha256
checksum of the content (winery for instance).

So ensure to use a CompositeObjId dictionary as content object identifier
to avoid these kind of issues when indexing content objects.

- remove the `self.result` attribute and make this an actual output of
  the `run()` method
- refactor the `process_journal_objects` to use `run()` to prevent code
  duplication.

scheduling indexer tasks is not supported any more for a while now.

Antoine Lambert authored 4 months ago and vlorentz committed 4 months ago

This works around citation.cff files with extra spaces in quoted strings

Previous implementation could lead to errors when an invalid date
is present in source codemeta.

The new 'backend' handling mechanism in swh.core 3.6 expect the
registered package to be a simple name (without '.').

So this partially revert changes from b681d147, reverting to using
'indexer_storage' as config section name.

This avoids accidentally interpreting user-provided data as a macro

When a citation.cff file contains a preferred-citation field, ensure
citation metadata are extracted from it.

Resolves #4743.

It's now expected the sql/ directory to live in the module implementing
the swh.core.db based backend.

But do not fail if no config file is given by default; required to be able
to run 'swh indexer mapping list-terms' for example (which is used to build the
documentation).

Normalize the indexer storage db for swh.core 3.6 with improved `swh db` handling
capabilities.

Remove storage/test_init.py, it's now outdated.

Warning: this rename the config entry for indexer storages from
'indexer_storage' to 'indexer.storage'. Bw compat core handle this when
loading the configuration for indexer cli tools and the api server only;
generic loading mechanism will not handle the old name properly
(i.e. when managing the storage db using `swh db` commands).

rdflib implicitly sets xsd:integer as the type of Literal(42); while the
ActivityStream specification requires a xsd:nonNegativeInteger here.

This was highlighted by a bugfix in rdflib 7.1.0, as previous versions
omitted serializing the "default" datatype corresponding to the Python
type of the literal value when writing in JSON-LD.

This is only a workaround until these tests are properly fixed.
They should probably be converted as actual (py)tests; it would make it
easier to support these types of breakage due to formatting changes in
rdflib...

Catch parsing error exceptions when attempting to parse a codemeta
document in XML or JSON then print error log and return None.

Fixes #4738
for existing metadata objects