Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Register
  • Sign in
  • S swh-loader-core
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 28
    • Issues 28
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 2
    • Merge requests 2
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • Platform
  • Development
  • swh-loader-core
  • Merge requests
  • !417

Allow partial snapshot creation during ingestion

  • Review changes

  • Download
  • Patches
  • Plain diff
Merged Antoine R. Dumont requested to merge generated-differential-D6380-source into master Sep 30, 2021
  • Overview 22
  • Commits 1
  • Pipelines 1
  • Changes 2

This introduces a create_partial_snapshot parameter to the base loader constructor. When activated, during each call of the store_data method, if there are more data to fetch, this will create a partial snapshot.

The final loop behaves as before, create the last visit with status 'full' targeting the snapshot.

The main difference between the 2 behavior is that an ingestion with that parameter on is more verbose in terms of origin_visit_status. This, in turn, allows to be incremental in subsequent visits for the same origin. This may especially be interesting for cases when loading fail due to out of hand resources issues (e.g. large svn or git repositories).

This is required to allow performance improvments on the loader git [1].

  • [1] D6386

Related to swh-loader-git#3625 (closed)

Test Plan

tox


Migrated from D6380 (view on Phabricator)

Assignee
Assign to
Reviewers
Request review from
Time tracking
Source branch: generated-differential-D6380-source