Skip to content
Snippets Groups Projects
Forked from Platform / Development / swh-loader-core
86 commits behind the upstream repository.
Antoine R. Dumont (@ardumont)'s avatar
Antoine R. Dumont authored
As it's already running in production, the 'tar' visit type is now immutable. So we
cannot change anything related to it.

So, instead rename it as before and adapt the check utility functions to allow those
checks to be bypassed for some specific cases (like the ArchiveLoader). New loaders
should fix their assertion failure immediately though.

Refs. swh/infra/sysadm-environment#4906
93c8791c
History

Software Heritage - Loader foundations

The Software Heritage Loader Core is a low-level loading utilities and helpers used by :term:`loaders <loader>`.

The main entry points are classes: - :class:`swh.loader.core.loader.BaseLoader` for VCS loaders (e.g. svn) - :class:`swh.loader.core.loader.DVCSLoader` for DVCS loaders (e.g. git, ...) - :class:`swh.loader.core.loader.ContentLoader` for Content loader - :class:`swh.loader.core.loader.DirectoryLoader` for Directory loader - :class:`swh.loader.package.loader.PackageLoader` for Package loaders (e.g. PyPI, Npm, ...) - ...

Package loaders

This package also implements many package loaders directly, out of convenience, as they usually are quite similar and each fits in a single file.

They all roughly follow these steps, explained in the :py:meth:`swh.loader.package.loader.PackageLoader.load` documentation. See the :ref:`package-loader-tutorial` for details.

VCS loaders

Unlike package loaders, VCS loaders remain in separate packages, as they often need more advanced conversions and very VCS-specific operations.

This usually involves getting the branches of a repository and recursively loading revisions in the history (and directory trees in these revisions), until a known revision is found