swh-graph in production

We need to have an in production swh-graph service, including fully automated periodic exports from storage. This is a meta-task to track all the related activities to achieve this goal.

---

Migrated from T2220 (view on Phabricator)

mentioned in merge request !182 (closed)

mentioned in issue swh-vault#887

marked this issue as related to swh-vault#887

marked this issue as related to swh/meta#2204

marked this issue as related to swh/meta#2217

added Compressed graph service Roadmap 2021 Roadmap 2022 meta-task labels

added priority:Normal label

marked this issue as related to swh-vault#3096 (closed)

changed the description

marked this issue as related to #3161 (closed)

assigned to @zack

marked this issue as related to swh/meta#3550

added priority:High label and removed priority:Normal label

Graph status meeting

• Interrogation du graph

  • Rocquencourt : Granet 700GO de ram (max atteint)

• Compression du graph : 1.7TO minimum

  • Telecom : machine 4TO

Compression

• Dataset: https://docs.softwareheritage.org/devel/swh-dataset/graph/dataset.html?highlight=dataset
• pipeline de compression : python
• implementation des étapes : org.softwareheritage.grap.compress

TODO

• Finish GRPC migration (seirl)

  • Forge issues to cleanup once GRPC is merged (seirl)

• Automate deployment (sysadm) >> prepare the command

  • swh/infra/puppet/puppet-swh-site#3168 (closed)

• Native hadoop libraries (?)

  • #4250 (closed)
  • Benchmark perfs with and without the hadoop librairies

• Luigi ETL[1] pipeline for compression / deployment

  • #3832 (closed)

• Integration of the generated javadoc in swh docs (vlorentz)

  • T1971

• Integration of Java code coverage in the forge

• Unit test the compression pipeline

• [1] https://luigi.readthedocs.io/en/stable/

assigned to @vlorentz and unassigned @zack

mentioned in issue swh/meta#3550

changed milestone to %Graph export and graph compression in production [Roadmap - Tooling and infrastructure]