Snippets Groups Projects

Forked from Platform / Development / swh-loader-git

106 commits behind the upstream repository.

Use all snapshots to compute base_snapshot_reverse_branches

vlorentz authored 2 years ago

base_snapshot_reverse_branches needs to contain all objects that
may be a snapshot target that the remote did not send to us.

Because we now use all snapshots to build the "have" list, such
targets include all targets of snapshots of "parent" origins,
not just the previous snapshot of the current origin.

This typically happens when a forge-fork pull branches from its
parent.

Resolves Sentry issue [[ https://sentry.softwareheritage.org/share/issue/1c08f5d764e7494e83ba254dc47f17af/ | SWH-LOADER-GIT-102 ]]

cf674f7c

cf674f7c 2 years ago

Name	Last commit	Last update
bin
docs
resources
swh
.git-blame-ignore-revs
.gitignore
.pre-commit-config.yaml
AUTHORS
CODE_OF_CONDUCT.md
CONTRIBUTORS
LICENSE
MANIFEST.in
Makefile
README.md
conftest.py
mypy.ini
pyproject.toml
pytest.ini
requirements-swh.txt
requirements-test.txt
requirements.txt
setup.cfg
setup.py
tox.ini

swh-loader-git

The Software Heritage Git Loader is a tool and a library to walk a local Git repository and inject into the SWH dataset all contained files that weren't known before.

The main entry points are:

:class:swh.loader.git.loader.GitLoader for the main loader which can ingest either local or remote git repository's contents. This is the main implementation deployed in production.
:class:swh.loader.git.from_disk.GitLoaderFromDisk which ingests only local git clone repository.
:class:swh.loader.git.loader.GitLoaderFromArchive which ingests a git repository wrapped in an archive.

License

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

See top-level LICENSE file for the full text of the GNU General Public License along with this program.

Dependencies

Runtime

python3
python3-dulwich
python3-retrying
python3-swh.core
python3-swh.model
python3-swh.storage
python3-swh.scheduler

Test

python3-nose

Requirements

implementation language, Python3
coding guidelines: conform to PEP8
Git access: via dulwich

CLI Run

You can run the loader from a remote origin (loader) or from an origin on disk (from_disk) directly by calling:

swh loader -C <config-file> run git <git-repository-url>

or "git_disk".

Configuration sample

/tmp/git.yml:

storage:
  cls: remote
  args:
    url: http://localhost:5002/