Look into triple-stores suitable as swh-search backends

Currently we rely on ElasticSearch, by embedding JSON-LD documents and treating them as regular JSON when searching.

This causes issues like T2876 and #4396 (closed) because ES expects some sort of strict schema, which goes against JSON-LD's design.

Plus, it does not support any sort of relations between documents, which we would need to run queries against related projects (eg. dependency graphs, fork graphs, "related software")

Therefore, I would like to try using a proper triple-store. Virtuoso in particular looks promising, as it support both SPARQL and full-text search.

---

Migrated from T4438 (view on Phabricator)

assigned to @vlorentz

added Archive search priority:Normal labels

changed the description

changed title from Look into Virtuoso as a new backend for swh-search to Look into triple-stores suitable as swh-search backends

changed the description

Useful benchmark tool: https://github.com/oxigraph/oxigraph/tree/main/bench

Another option I am considering if other options do not scale: repurposing swh-graph (especially now that it supports edge labels) as a triple store; but we would need to either reimplement SPARQL on top or use https://github.com/SoftwareHeritage/swh-graph-tinkerpop + https://tinkerpop.apache.org/docs/current/reference/#sparql-gremlin

Actually the open-source version of Virtuoso is unmaintained, and I couldn't figure out how to use it. Blazegraph and Apache Jena look promising too, though.

@KShivendu pointed out that Neo4j may be a strong option too. It doesn't natively support SPARQL, but we could have it as a layer on top of the Tinkerpop API

added activity::Epic label

changed milestone to %Evaluate the storage of indexed metadata in a triple-store [Roadmap - Share]

https://thomas.pellissier-tanon.fr/blog/2023-01-15-wdqs.html

Did you looked at nebula graph DB?

It doesn't support sparql but an equivalent language and the benchmarks look promising (and it's backed by cassandra)

Hum, I made a mistake in my notes, it looks like nebula is not using cassandra as a storage backend but a proprietary storage implementation (but I'm not saying it can't be interesting to look at it).

Apparently a spark connector is available: https://docs.nebula-graph.io/3.4.0/nebula-spark-connector/

JanusGraph can use cassandra as a storage backend and also have a Spark connector.

A tool to create a Rest API on top of a sparql endpoint with queries: https://github.com/CLARIAH/grlc/

unassigned @vlorentz