Skip to content
GitLab
Open
Issue created by Nicolas Dandrimont@olasdMaintainer

loader git: enable global deduplication of head branches before fetching them

This task tracks the efforts to (re-)enable global deduplication of revisions in the git loader, to reduce the amount of data downloaded from upstreams (and converted uselessly by workers).

  • first enabling partial global deduplication through extid mappings for snapshot heads (for which we know that we have done a complete load of the history): #3635
  • then surveying the opportunity of "just" doing a global lookup for any object types: swh/devel/experiments/swh-db-audit#3656 (moved), and #3654 to avoid creating new "history holes"

Migrated from T3655 (view on Phabricator)

Edited by Phabricator Migration user