Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Register
  • Sign in
  • S swh-loader-git
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 28
    • Issues 28
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 4
    • Merge requests 4
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Platform
  • Development
  • swh-loader-git
  • Issues
  • #3653
Closed
Open
Issue created Oct 14, 2021 by Antoine R. Dumont@ardumontMaintainer1 of 4 checklist items completed1/4 checklist items

Stabilize loader git

Recent runs actually showed that the loader git:

  • #3625 (closed): uses too much memory
  • #3635: does too much work (regarding forks, it does not deduplicate computations for in the end noops)
  • swh/infra/sysadm-environment#3640: is too slow to finish which renders maintenance a pain
  • #3652: Does not properly handle some origins due to a packfile too large

So for example, given too much concurrency, the first point actually make it unstable. Another example, too many forks (at the same time or not) actually make the loader git overall slow.

This task is actually a parent task to track the work done to actually fix those points, make the loader git:

  • uses less memory
  • do less unneeded work
  • stops fast when required to
  • actually able to ingest "larger" origins

The perimeter of this task is frozen to this 4 points (it's not a neverending task ;)


Migrated from T3653 (view on Phabricator)

Edited Jan 07, 2023 by Phabricator Migration user
Assignee
Assign to
Time tracking