Skip to content
Snippets Groups Projects
Verified Commit fc1adf07 authored by Antoine R. Dumont's avatar Antoine R. Dumont
Browse files

Allow partial snapshot creation during ingestion

This introduces a `create_partial_snapshot` parameter to the base loader constructor.
When activated, during each call of the `store_data` method, if there are more data to
fetch, this will create a partial snapshot (and an associated visit status).

The final loop behaves as before, create the last visit with status 'full' targeting the
final snapshot.

The main difference between the 2 behaviors is that an ingestion with that parameter on
is more verbose in terms of origin_visit_status. This, in turn, allows to be incremental
in subsequent visits for the same origin. This may especially be interesting for cases
when loading fail due to out of hand resources issues (e.g. large svn or git
repositories).

Related to T3625
parent a63b39e5
No related branches found
No related tags found
No related merge requests found
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment