Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Register
  • Sign in
  • S swh-loader-git
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 26
    • Issues 26
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 5
    • Merge requests 5
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Platform
  • Development
  • swh-loader-git
  • Issues
  • #3625
Closed
Open
Issue created Sep 30, 2021 by Antoine R. Dumont@ardumontMaintainer

Reduce git loader memory footprint

It's currently using a huge amount of memory especially on large repositories. Which is currently a blocking point so we need to decrease their concurrency so not everything fails on oom.

Checking a bit the code, currently the loader retrieves the full packfile and then parse it multiple times to ingest the dag objects. We believe it's possible to retrieve packfiles incrementally.

This increases the communication with the server but that should be reducing the memory usage.

Related to swh/infra/sysadm-environment#3025


Migrated from T3625 (view on Phabricator)

Edited Jan 07, 2023 by Phabricator Migration user
Assignee
Assign to
Time tracking