From fc533432bfcc29f5d193e0a85a1b00d1113ebfba Mon Sep 17 00:00:00 2001
From: David Douard <david.douard@sdfa3.org>
Date: Thu, 16 Nov 2023 18:35:08 +0100
Subject: [PATCH] docs: include the README file in the main index page
Convert README from markdown to ReST to make it embeddable in
docs/index.rst
---
README.md => README.rst | 224 +++++++++++++++++++++-------------------
docs/index.rst | 28 +++--
setup.py | 4 +-
3 files changed, 131 insertions(+), 125 deletions(-)
rename README.md => README.rst (52%)
diff --git a/README.md b/README.rst
similarity index 52%
rename from README.md
rename to README.rst
index b18b1cdbf..31df2e299 100644
--- a/README.md
+++ b/README.rst
@@ -1,13 +1,9 @@
-swh-storage
-===========
+Software Heritage - Storage
+===========================
Abstraction layer over the archive, allowing to access all stored source code
artifacts as well as their metadata.
-See the
-[documentation](https://docs.softwareheritage.org/devel/swh-storage/index.html)
-for more details.
-
## Quick start
### Dependencies
@@ -19,9 +15,10 @@ cassandra server.
#### Debian-like host
-```
-$ sudo apt install libpq-dev postgresql-11 cassandra
-```
+.. code-block:: shell
+
+ $ sudo apt install libpq-dev postgresql-11 cassandra
+
#### Non Debian-like host
@@ -31,73 +28,79 @@ variable `SWH_CASSANDRA_BIN`.
Optionally, you can avoid running the cassandra tests.
-```
-(swh) :~/swh-storage$ tox -- -m 'not cassandra'
-```
+.. code-block:: shell
+
+ (swh) :~/swh-storage$ tox -- -m 'not cassandra'
+
### Installation
It is strongly recommended to use a virtualenv. In the following, we
consider you work in a virtualenv named `swh`. See the
-[developer setup guide](https://docs.softwareheritage.org/devel/developer-setup.html#developer-setup)
+`developer setup guide <https://docs.softwareheritage.org/devel/developer-setup.html#developer-setup>`_
for a more details on how to setup a working environment.
You can install the package directly from
-[pypi](https://pypi.org/p/swh.storage):
+`pypi <https://pypi.org/p/swh.storage>`_:
+
+.. code-block:: shell
+
+ (swh) :~$ pip install swh.storage
+ [...]
-```
-(swh) :~$ pip install swh.storage
-[...]
-```
Or from sources:
-```
-(swh) :~$ git clone https://forge.softwareheritage.org/source/swh-storage.git
-[...]
-(swh) :~$ cd swh-storage
-(swh) :~/swh-storage$ pip install .
-[...]
-```
+.. code-block:: shell
+
+ (swh) :~$ git clone https://forge.softwareheritage.org/source/swh-storage.git
+ [...]
+ (swh) :~$ cd swh-storage
+ (swh) :~/swh-storage$ pip install .
+ [...]
+
Then you can check it's properly installed:
-```
-(swh) :~$ swh storage --help
-Usage: swh storage [OPTIONS] COMMAND [ARGS]...
- Software Heritage Storage tools.
+.. code-block:: shell
+
+ (swh) :~$ swh storage --help
+ Usage: swh storage [OPTIONS] COMMAND [ARGS]...
-Options:
- -h, --help Show this message and exit.
+ Software Heritage Storage tools.
-Commands:
- rpc-serve Software Heritage Storage RPC server.
-```
+ Options:
+ -h, --help Show this message and exit.
+
+ Commands:
+ rpc-serve Software Heritage Storage RPC server.
## Tests
The best way of running Python tests for this module is to use
-[tox](https://tox.readthedocs.io/).
+`tox <https://tox.readthedocs.io>`_.
+
+.. code-block:: shell
+
+ (swh) :~$ pip install tox
-```
-(swh) :~$ pip install tox
-```
### tox
From the sources directory, simply use tox:
-```
-(swh) :~/swh-storage$ tox
-[...]
-========= 315 passed, 6 skipped, 15 warnings in 40.86 seconds ==========
-_______________________________ summary ________________________________
- flake8: commands succeeded
- py3: commands succeeded
- congratulations :)
-```
+.. code-block:: shell
+
+ (swh) :~/swh-storage$ tox
+ [...]
+ ========= 315 passed, 6 skipped, 15 warnings in 40.86 seconds ==========
+ _______________________________ summary ________________________________
+ flake8: commands succeeded
+ py3: commands succeeded
+ congratulations :)
+
Note: it is possible to set the `JAVA_HOME` environment variable to specify the
version of the JVM to be used by Cassandra. For example, at the time of writing
@@ -105,11 +108,12 @@ this, Cassandra is meant to be run with Java 11. On Debian bookworm, one needs
to manually install openjdk-11-jre-headless from bullseye or unstable and
set the appropriate environment variable:
-```
-(swh) :~/swh-storage$ export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
-(swh) :~/swh-storage$ tox
-[...]
-```
+.. code-block:: shell
+
+ (swh) :~/swh-storage$ export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
+ (swh) :~/swh-storage$ tox
+ [...]
+
## Development
@@ -120,15 +124,16 @@ a running Postgresql database.
A typical configuration `storage.yml` file is:
-```
-storage:
- cls: postgresql
- db: "dbname=softwareheritage-dev user=<user> password=<pwd>"
- objstorage:
- cls: pathslicing
- root: /tmp/swh-storage/
- slicing: 0:2/2:4/4:6
-```
+.. code-block:: yaml
+
+ storage:
+ cls: postgresql
+ db: "dbname=softwareheritage-dev user=<user> password=<pwd>"
+ objstorage:
+ cls: pathslicing
+ root: /tmp/swh-storage/
+ slicing: 0:2/2:4/4:6
+
which means, this uses:
@@ -155,71 +160,76 @@ Note that the `root` path should exist on disk before starting the server.
If the python package has been properly installed (e.g. in a virtual env), you
should be able to use the command:
-```
-(swh) :~/swh-storage$ swh storage -C storage.yml rpc-serve
-```
+.. code-block:: shell
+
+ (swh) :~/swh-storage$ swh storage -C storage.yml rpc-serve
+
This runs a local swh-storage api at 5002 port.
-```
-(swh) :~/swh-storage$ curl http://127.0.0.1:5002
-<html>
-<head><title>Software Heritage storage server</title></head>
-<body>
-<p>You have reached the
-<a href="https://www.softwareheritage.org/">Software Heritage</a>
-storage server.<br />
-See its
-<a href="https://docs.softwareheritage.org/devel/swh-storage/">documentation
-and API</a> for more information</p>
-```
+.. code-block:: shell
+
+ (swh) :~/swh-storage$ curl http://127.0.0.1:5002
+ <html>
+ <head><title>Software Heritage storage server</title></head>
+ <body>
+ <p>You have reached the
+ <a href="https://www.softwareheritage.org/">Software Heritage</a>
+ storage server.<br />
+ See its
+ <a href="https://docs.softwareheritage.org/devel/swh-storage/">documentation
+ and API</a> for more information</p>
+
### And then what?
In your upper layer
-([loader-git](https://forge.softwareheritage.org/source/swh-loader-git/),
-[loader-svn](https://forge.softwareheritage.org/source/swh-loader-svn/),
+(`loader-git <https://forge.softwareheritage.org/source/swh-loader-git>`_,
+`loader-svn <https://forge.softwareheritage.org/source/swh-loader-svn>`_,
etc...), you can define a remote storage with this snippet of yaml
configuration.
-```
-storage:
- cls: remote
- url: http://localhost:5002/
-```
+.. code-block:: yaml
+
+ storage:
+ cls: remote
+ url: http://localhost:5002/
+
You could directly define a postgresql storage with the following snippet:
-```
-storage:
- cls: postgresql
- db: service=swh-dev
- objstorage:
- cls: pathslicing
- root: /home/storage/swh-storage/
- slicing: 0:2/2:4/4:6
-```
+.. code-block:: yaml
+
+ storage:
+ cls: postgresql
+ db: service=swh-dev
+ objstorage:
+ cls: pathslicing
+ root: /home/storage/swh-storage/
+ slicing: 0:2/2:4/4:6
+
## Cassandra
-As an alternative to PostgreSQL, swh-storage can use Cassandra as a database backend.
-It can be used like this:
-
-```
-storage:
- cls: cassandra
- hosts:
- - localhost
- keyspace: swh
- objstorage:
- cls: pathslicing
- root: /home/storage/swh-storage/
- slicing: 0:2/2:4/4:6
-```
+As an alternative to PostgreSQL, swh-storage can use Cassandra as a database
+backend. It can be used like this:
+
+.. code-block:: yaml
+
+ storage:
+ cls: cassandra
+ hosts:
+ - localhost
+ keyspace: swh
+ objstorage:
+ cls: pathslicing
+ root: /home/storage/swh-storage/
+ slicing: 0:2/2:4/4:6
+
The Cassandra swh-storage implementation supports both Cassandra >= 4.0-alpha2
and ScyllaDB >= 4.4 (and possibly earlier versions, but this is untested).
While the main code supports both transparently, running tests
or configuring the schema requires specific code when using ScyllaDB,
-enabled by setting the `SWH_USE_SCYLLADB=1` environment variable.
+enabled by setting the ``SWH_USE_SCYLLADB=1`` environment variable.
diff --git a/docs/index.rst b/docs/index.rst
index 790a7c32f..3ce5c6350 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -1,11 +1,6 @@
.. _swh-storage:
-Software Heritage - Storage
-===========================
-
-Abstraction layer over the archive, allowing to access all stored source code
-artifacts as well as their metadata
-
+.. include:: README.rst
The Software Heritage storage consist of a high-level storage layer
(:mod:`swh.storage`) that exposes a client/server API
@@ -29,20 +24,21 @@ If you want to access the Software Heritage archive without running your own,
you should use the :swh_web:`Web API <api/>` instead.
As ``swh-storage`` has multiple backends, it is instantiated via the
-:py:func:`swh.storage.get_storage` function, which takes as argument the backend type
-(usually ``remote``, if you already have access to a running swh-storage).
+:py:func:`swh.storage.get_storage` function, which takes as argument the
+backend type (usually ``remote``, if you already have access to a running
+swh-storage).
It returns an instance of a class implementing
-:py:class:`swh.storage.interface.StorageInterface`; which is mostly a set of key-value
-stores, one for each object type.
+:py:class:`swh.storage.interface.StorageInterface`; which is mostly a set of
+key-value stores, one for each object type.
-Many of the arguments and return types are "model objects", ie. immutable objects
-that are instances of the classes defined in :py:mod:`swh.model.model`.
+Many of the arguments and return types are "model objects", ie. immutable
+objects that are instances of the classes defined in :py:mod:`swh.model.model`.
-Methods returning long lists of arguments are paginated; by returning both a list
-of results and an opaque token to get the next page of results.
-For example, to list all the visits of an origin using ``origin_visit_get``
-ten visits at a time, you can do:
+Methods returning long lists of arguments are paginated; by returning both a
+list of results and an opaque token to get the next page of results. For
+example, to list all the visits of an origin using ``origin_visit_get`` ten
+visits at a time, you can do:
.. code-block::
diff --git a/setup.py b/setup.py
index 76257341f..bc0522b4b 100755
--- a/setup.py
+++ b/setup.py
@@ -12,7 +12,7 @@ from setuptools import find_namespace_packages, setup
here = path.abspath(path.dirname(__file__))
# Get the long description from the README file
-with open(path.join(here, "README.md"), encoding="utf-8") as f:
+with open(path.join(here, "README.rst"), encoding="utf-8") as f:
long_description = f.read()
@@ -39,7 +39,7 @@ setup(
name="swh.storage",
description="Software Heritage storage manager",
long_description=long_description,
- long_description_content_type="text/markdown",
+ long_description_content_type="text/x-rst",
python_requires=">=3.7",
author="Software Heritage developers",
author_email="swh-devel@inria.fr",
--
GitLab