Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • lunar/swh-loader-mercurial
  • anlambert/swh-loader-mercurial
  • ardumont/swh-loader-mercurial
  • swh/devel/swh-loader-mercurial
  • douardda/swh-loader-mercurial
  • marmoute/swh-loader-mercurial
6 results
Show changes
Commits on Source (137)
# Changes here will be overwritten by Copier
_commit: v0.3.3
_src_path: https://gitlab.softwareheritage.org/swh/devel/swh-py-template.git
description: Software Heritage Mercurial Loader
distribution_name: swh-loader-mercurial
have_cli: true
have_workers: false
package_root: swh/loader/mercurial
project_name: swh.loader.mercurial
python_minimal_version: '3.7'
readme_format: rst
# Enable black
4c5f78830dda4380b279af8dfc3399168fd1eb15
# python: Reformat code with black 22.3.0
bdf1d9634d4719fdcc791125fdc63ce6a6c3e1e3
*.egg-info/
*.pyc
*.sw?
*~
/.coverage
/.coverage.*
.coverage
.eggs/
.hypothesis
.mypy_cache
.tox
__pycache__
*.egg-info/
version.txt
build/
dist/
.tox/
.mypy_cache/
# these are symlinks created by a hook in swh-docs' main sphinx conf.py
docs/README.rst
docs/README.md
# this should be a symlink for people who want to build the sphinx doc
# without using tox, generally created by the swh-env/bin/update script
docs/Makefile.sphinx
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.4.0
hooks:
- id: trailing-whitespace
- id: flake8
- id: check-json
- id: check-yaml
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
hooks:
- id: trailing-whitespace
- id: check-json
- id: check-yaml
- repo: https://github.com/codespell-project/codespell
rev: v1.16.0
hooks:
- id: codespell
- repo: https://github.com/python/black
rev: 25.1.0
hooks:
- id: black
- repo: local
hooks:
- id: mypy
name: mypy
entry: mypy
args: [swh]
pass_filenames: false
language: system
types: [python]
- repo: https://github.com/PyCQA/isort
rev: 6.0.0
hooks:
- id: isort
- repo: https://github.com/python/black
rev: 19.10b0
hooks:
- id: black
- repo: https://github.com/pycqa/flake8
rev: 7.1.1
hooks:
- id: flake8
additional_dependencies: [flake8-bugbear==24.12.12, flake8-pyproject]
# unfortunately, we are far from being able to enable this...
# - repo: https://github.com/PyCQA/pydocstyle.git
# rev: 4.0.0
# hooks:
# - id: pydocstyle
# name: pydocstyle
# description: pydocstyle is a static analysis tool for checking compliance with Python docstring conventions.
# entry: pydocstyle --convention=google
# language: python
# types: [python]
- repo: https://github.com/codespell-project/codespell
rev: v2.4.1
hooks:
- id: codespell
name: Check source code spelling
stages: [pre-commit]
- id: codespell
name: Check commit message spelling
stages: [commit-msg]
- repo: local
hooks:
- id: mypy
name: mypy
entry: mypy
args: [swh]
pass_filenames: false
language: system
types: [python]
- id: twine-check
name: twine check
description: call twine check when pushing an annotated release tag
entry: bash -c "ref=$(git describe) &&
[[ $ref =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]] &&
(python3 -m build --sdist && twine check $(ls -t dist/* | head -1)) || true"
pass_filenames: false
stages: [pre-push]
language: python
additional_dependencies: [twine, build]
......@@ -6,7 +6,7 @@ In the interest of fostering an open and welcoming environment, we as Software
Heritage contributors and maintainers pledge to making participation in our
project and our community a harassment-free experience for everyone, regardless
of age, body size, disability, ethnicity, sex characteristics, gender identity
and expression, level of experience, education, socio-economic status,
and expression, level of experience, education, socioeconomic status,
nationality, personal appearance, race, religion, or sexual identity and
orientation.
......
include Makefile
include requirements*.txt
include version.txt
recursive-include swh/loader/mercurial/tests/data *
recursive-include swh py.typed
swh-loader-mercurial
=========================
# Configuration file
In usual location for a loader, *{/etc/softwareheritage/ | ~/.swh/ |
~/.config/swh/}loader/hg.yml*:
``` YAML
storage:
cls: remote
args:
url: http://localhost:5002/
```
# Basic use
The main entry point to import a Mercurial repository is the `main` function
defined in the `swh.loader.mercurial.cli` module:
``` bash
python3 -m swh.loader.mercurial.cli
```
If the Python package has been installed via `pip`, you should be able
to type:
``` bash
user@host:~$ swh-loader-hg --help
Usage: swh-loader-hg [OPTIONS] ORIGIN_URL
Options:
-d, --hg-directory TEXT Path to the hg (local) directory to load
from. If unset, the hg repo will ben cloned
from the given (origin) url
-a, --hg-archive TEXT Path to the hg (local) archive file to load
from.
-D, --visit-date TEXT Visit date (defaults to now)
-l, --log-level [NOTSET|DEBUG|INFO|WARNING|ERROR|CRITICAL]
Log level
--help Show this message and exit.
```
For example:
``` bash
user@host:~$ swh-loader-hg https://www.mercurial-scm.org/repo/hello
[...]
```
# From Python
From python3's toplevel:
## Remote
``` Python
project = 'hello'
# remote repository
origin_url = 'https://www.mercurial-scm.org/repo/%s' % project
# local clone
directory = '/home/storage/hg/repo/%s' % project
import logging
logging.basicConfig(level=logging.DEBUG)
from swh.loader.mercurial.tasks import LoadMercurial
t = LoadMercurial()
t.run(origin_url=origin_url, directory=directory, visit_date='2016-05-03T15:16:32+00:00')
```
## local directory
Only origin, contents, and directories are filled so far.
Remaining objects are empty (revision, release, occurrence).
``` Python
project = '756015-ipv6'
directory = '/home/storage/hg/repo/%s' % project
origin_url = 'https://%s.googlecode.com' % project
import logging
logging.basicConfig(level=logging.DEBUG)
from swh.loader.mercurial.tasks import LoadMercurial
t = LoadMercurial()
t.run(origin_url=origin_url, directory=directory, visit_date='2016-05-03T15:16:32+00:00')
```
## local archive
``` Python
project = '756015-ipv6-source-archive.zip'
archive_path = '/home/storage/hg/repo/%s' % project
origin_url = 'https://%s-archive.googlecode.com' % project
import logging
logging.basicConfig(level=logging.DEBUG)
from swh.loader.mercurial.tasks import LoadArchiveMercurial
t = LoadArchiveMercurial()
t.run(origin_url=origin_url, archive_path=archive_path, visit_date='2016-05-03T15:16:32+00:00')
```
swh-loader-mercurial
====================
The Software Heritage Mercurial Loader is a tool and a library to walk a local
mercurial repository and inject into the SWH dataset all contained files that
weren't known before.
The main entry points are:
- ``swh.loader.mercurial.loader.HgLoader`` which reads and loads a local
repository into an SWH archive.
- ``swh.loader.mercurial.loader.HgArchiveLoader`` which reads and loads
a local repository wrapped within a tarball
- ``swh.loader.mercurial.directory.HgCheckoutLoader`` which ingests the hg
tree at a specific changeset or tag.
CLI run
-------
Configuration file
++++++++++++++++++
/tmp/mercurial.yml:
.. code-block:: yaml
storage:
cls: remote
args:
url: http://localhost:5002/
Basic use
+++++++++
.. code-block:: shell
swh loader --C /tmp/mercurial.yml run mercurial https://www.mercurial-scm.org/repo/hello
# Copyright (C) 2020 The Software Heritage developers
# Copyright (C) 2020-2021 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
import pytest
pytest_plugins = ["swh.scheduler.pytest_plugin", "swh.storage.pytest_plugin"]
pytest_plugins = [
"swh.scheduler.pytest_plugin",
"swh.storage.pytest_plugin",
"swh.loader.pytest_plugin",
]
@pytest.fixture(scope="session")
......
include ../../swh-docs/Makefile.sphinx
include Makefile.sphinx
.. _swh-loader-mercurial:
Software Heritage - Mercurial loader
====================================
Loader for `Mercurial <https://www.mercurial-scm.org/>`_ repositories.
.. include:: README.rst
Reference Documentation
-----------------------
......@@ -12,4 +8,11 @@ Reference Documentation
.. toctree::
:maxdepth: 2
/apidoc/swh.loader.mercurial
.. only:: standalone_package_doc
Indices and tables
------------------
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`
[mypy]
namespace_packages = True
warn_unused_ignores = True
# 3rd party libraries without stubs (yet)
[mypy-billiard.*]
ignore_missing_imports = True
[mypy-celery.*]
ignore_missing_imports = True
[mypy-hglib.*]
ignore_missing_imports = True
[mypy-pkg_resources.*]
ignore_missing_imports = True
[mypy-patoolib.*]
ignore_missing_imports = True
[mypy-pytest.*]
ignore_missing_imports = True
[mypy-sqlitedict.*]
ignore_missing_imports = True
[mypy-swh.loader.*]
ignore_missing_imports = True
[project]
name = "swh.loader.mercurial"
authors = [
{name="Software Heritage developers", email="swh-devel@inria.fr"},
]
description = "Software Heritage Mercurial Loader"
readme = {file = "README.rst", content-type = "text/x-rst"}
requires-python = ">=3.7"
classifiers = [
"Programming Language :: Python :: 3",
"Intended Audience :: Developers",
"License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
"Operating System :: OS Independent",
"Development Status :: 4 - Beta",
]
dynamic = ["version", "dependencies", "optional-dependencies"]
[tool.setuptools.packages.find]
include = ["swh.*"]
[tool.setuptools.dynamic]
dependencies = {file = ["requirements.txt", "requirements-swh.txt"]}
[tool.setuptools.dynamic.optional-dependencies]
testing = {file = ["requirements.txt", "requirements-swh.txt", "requirements-test.txt"]}
[project.entry-points."console_scripts"]
"swh-hg-identify" = "swh.loader.mercurial.identify:main"
[project.entry-points."swh.workers"]
"loader.mercurial" = "swh.loader.mercurial:register"
"loader.mercurial-checkout" = "swh.loader.mercurial:register_checkout"
[project.urls]
"Homepage" = "https://gitlab.softwareheritage.org/swh/devel/swh-loader-mercurial"
"Bug Reports" = "https://gitlab.softwareheritage.org/swh/devel/swh-loader-mercurial/-/issues"
"Funding" = "https://www.softwareheritage.org/donate"
"Documentation" = "https://docs.softwareheritage.org/devel/swh-loader-mercurial/"
"Source" = "https://gitlab.softwareheritage.org/swh/devel/swh-loader-mercurial.git"
[build-system]
requires = ["setuptools", "setuptools-scm"]
build-backend = "setuptools.build_meta"
[tool.setuptools_scm]
fallback_version = "0.0.1"
[tool.black]
target-version = ['py37']
target-version = ['py39', 'py310', 'py311', 'py312']
[tool.isort]
multi_line_output = 3
include_trailing_comma = true
force_grid_wrap = 0
use_parentheses = true
ensure_newline_before_comments = true
line_length = 88
force_sort_within_sections = true
known_first_party = ['swh']
[tool.mypy]
namespace_packages = true
warn_unused_ignores = true
explicit_package_bases = true
# ^ Needed for mypy to detect py.typed from swh packages installed
# in editable mode
plugins = []
# 3rd party libraries without stubs (yet)
[[tool.mypy.overrides]]
module = [
"mercurial.*",
"patoolib.*",
]
ignore_missing_imports = true
[tool.flake8]
select = ["C", "E", "F", "W", "B950"]
ignore = [
"E203", # whitespaces before ':' <https://github.com/psf/black/issues/315>
"E231", # missing whitespace after ','
"E501", # line too long, use B950 warning from flake8-bugbear instead
"W503" # line break before binary operator <https://github.com/psf/black/issues/52>
]
max-line-length = 88
[tool.pytest.ini_options]
norecursedirs = "build docs .*"
asyncio_mode = "strict"
consider_namespace_packages = true
[pytest]
# Drop this when these fixtures aren't imported automatically anymore
addopts = -p no:pytest_swh_scheduler -p no:pytest_swh_storage
norecursedirs = docs
swh.model >= 0.4.0
swh.storage >= 0.10.0
swh.model >= 6.13.0
swh.storage >= 2.4.1
swh.scheduler >= 0.0.39
swh.loader.core >= 0.5.5
swh.loader.core >= 5.18.1
celery-types
pytest
pytest-mock
swh.core[http] >= 0.0.61
swh.loader.core[testing] >= 5.18.1
swh.scheduler[testing] >= 0.5.0
swh.storage[testing]
types-click
types-deprecated
types-python-dateutil
# Add here external Python modules dependencies, one per line. Module names
# should match https://pypi.python.org/pypi names. For the full spec or
# dependency lines, see https://pip.readthedocs.org/en/1.1/requirements.html
billiard
click
patool
python-dateutil
python-hglib
retrying
sqlitedict
vcversioner
mercurial
[flake8]
# E203: whitespaces before ':' <https://github.com/psf/black/issues/315>
# E231: missing whitespace after ','
# W503: line break before binary operator <https://github.com/psf/black/issues/52>
ignore = E203,E231,W503
max-line-length = 88
#!/usr/bin/env python3
# Copyright (C) 2015-2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
from setuptools import setup, find_packages
from os import path
from io import open
here = path.abspath(path.dirname(__file__))
# Get the long description from the README file
with open(path.join(here, "README.md"), encoding="utf-8") as f:
long_description = f.read()
def parse_requirements(name=None):
if name:
reqf = "requirements-%s.txt" % name
else:
reqf = "requirements.txt"
requirements = []
if not path.exists(reqf):
return requirements
with open(reqf) as f:
for line in f.readlines():
line = line.strip()
if not line or line.startswith("#"):
continue
requirements.append(line)
return requirements
setup(
name="swh.loader.mercurial",
description="Software Heritage Mercurial Loader",
long_description=long_description,
long_description_content_type="text/markdown",
python_requires=">=3.7",
author="Software Heritage developers",
author_email="swh-devel@inria.fr",
url="https://forge.softwareheritage.org/diffusion/DLDHG/",
packages=find_packages(),
scripts=[],
install_requires=parse_requirements() + parse_requirements("swh"),
setup_requires=["vcversioner"],
extras_require={"testing": parse_requirements("test")},
vcversioner={},
include_package_data=True,
entry_points="""
[swh.workers]
loader.mercurial=swh.loader.mercurial:register
""",
classifiers=[
"Programming Language :: Python :: 3",
"Intended Audience :: Developers",
"License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
"Operating System :: OS Independent",
"Development Status :: 4 - Beta",
],
project_urls={
"Bug Reports": "https://forge.softwareheritage.org/maniphest",
"Funding": "https://www.softwareheritage.org/donate",
"Source": ("https://forge.softwareheritage.org/source/swh-loader-mercurial"),
"Documentation": "https://docs.softwareheritage.org/devel/swh-loader-mercurial/", # NoQA: E501
},
)
from pkgutil import extend_path
from typing import Iterable
__path__ = extend_path(__path__, __name__) # type: Iterable[str]