From ee51a133bf9ebbc18fad83410d22a07e79aa4876 Mon Sep 17 00:00:00 2001 From: David Douard <david.douard@sdfa3.org> Date: Thu, 28 Mar 2024 18:28:43 +0100 Subject: [PATCH 1/4] Remove symlink introduced by mistake --- docs/devel/README.rst | 1 - 1 file changed, 1 deletion(-) delete mode 120000 docs/devel/README.rst diff --git a/docs/devel/README.rst b/docs/devel/README.rst deleted file mode 120000 index f90a62a2..00000000 --- a/docs/devel/README.rst +++ /dev/null @@ -1 +0,0 @@ -/home/anlambert/swh/swh-environment/swh-docs/docs/devel/../README.rst \ No newline at end of file -- GitLab From b441a54f4ce0fb303d9be7a423f1f058d0dfd054 Mon Sep 17 00:00:00 2001 From: David Douard <david.douard@sdfa3.org> Date: Thu, 28 Mar 2024 11:58:43 +0100 Subject: [PATCH 2/4] devel/getting-started: use code-block:: console for cli snippets --- docs/devel/getting-started.rst | 40 +++++++++++++++++++++++++--------- 1 file changed, 30 insertions(+), 10 deletions(-) diff --git a/docs/devel/getting-started.rst b/docs/devel/getting-started.rst index 1a58ffc3..8b73bfa0 100644 --- a/docs/devel/getting-started.rst +++ b/docs/devel/getting-started.rst @@ -29,7 +29,9 @@ Quick start ----------- First, retrieve Software Heritage development environment to get the -Docker configuration:: +Docker configuration: + +.. code-block:: console ~$ git clone https://gitlab.softwareheritage.org/swh/devel/docker.git swh-docker ~$ cd swh-docker @@ -40,7 +42,9 @@ Docker configuration:: you should rather follow the instructions in section :ref:`checkout-source-code` to install the full Software Heritage development environment that includes Docker configuration. -Then, start containers:: +Then, start containers: + +.. code-block:: console ~/swh-docker$ docker compose up -d [...] @@ -52,7 +56,9 @@ Then, start containers:: [...] This will build Docker images and run them. Check everything is running -fine with:: +fine with: + +.. code-block:: console ~/swh-docker$ docker compose ps Name Command State Ports @@ -69,7 +75,9 @@ dependency-related problems. If some containers failed to start, just run the ``docker compose up -d`` command again. If a container really refuses to start properly, you can check why using -the ``docker compose logs`` command. For example:: +the ``docker compose logs`` command. For example: + +.. code-block:: console ~/swh-docker$ docker compose logs swh-lister Attaching to docker_swh-lister_1 @@ -93,7 +101,9 @@ content. The simplest way to start loading software is to use the http://localhost:5080/browse/origin/save/ You can also use the command line interface to inject code. For -example to retrieve projects hossted on the https://0xacab.org GitLab forge:: +example to retrieve projects hossted on the https://0xacab.org GitLab forge: + +.. code-block:: console ~/swh-docker$ docker compose exec swh-scheduler \ swh scheduler task add list-gitlab-full \ @@ -114,7 +124,9 @@ This task will scrape the forge’s project list and register origins to the sch This takes at most a couple of minutes. Then, you must tell the scheduler to create loading tasks for these origins. -For example, to create tasks for 100 of these origins:: +For example, to create tasks for 100 of these origins: + +.. code-block:: console ~/swh-docker$ docker compose exec swh-scheduler \ swh scheduler origin schedule-next git 100 @@ -122,7 +134,9 @@ For example, to create tasks for 100 of these origins:: This will take a bit of time to complete. To increase the speed at which git repositories are imported, you can -spawn more ``swh-loader-git`` workers:: +spawn more ``swh-loader-git`` workers: + +.. code-block:: console ~/swh-docker$ docker compose exec swh-scheduler \ celery status @@ -141,7 +155,9 @@ spawn more ``swh-loader-git`` workers:: "prefetch_count": 4 Now there are 4 workers ingesting git repositories. You can also -increase the number of ``swh-loader-git`` containers:: +increase the number of ``swh-loader-git`` containers: + +.. code-block:: console ~/swh-docker$ docker compose up -d --scale swh-loader=4 [...] @@ -159,7 +175,9 @@ Heritage. When new versions of these components are released, the docker image will not be automatically updated. In order to update all Software Heritage components to their latest version, the docker image needs to be explicitly rebuilt by issuing the following command from within the -``docker`` directory:: +``docker`` directory: + +.. code-block:: console ~/swh-docker$ docker build --no-cache -t swh/stack . @@ -174,7 +192,9 @@ You can monitor your local installation by looking at: Shut down your local installation --------------------------------- -To shut down your SoftWare Heritage, just run:: +To shut down your SoftWare Heritage, just run: + +.. code-block:: console ~/swh-docker$ docker compose down -- GitLab From 466c234b4a78b8e21a1ff17bfce2fee031fa4753 Mon Sep 17 00:00:00 2001 From: David Douard <david.douard@sdfa3.org> Date: Thu, 28 Mar 2024 11:59:38 +0100 Subject: [PATCH 3/4] devel/getting-started: Fix the doc about port used to access the archive It is not hardcoded 5080 any more. --- docs/devel/getting-started.rst | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/docs/devel/getting-started.rst b/docs/devel/getting-started.rst index 8b73bfa0..27dc2830 100644 --- a/docs/devel/getting-started.rst +++ b/docs/devel/getting-started.rst @@ -91,14 +91,30 @@ the ``docker compose logs`` command. For example: For details on the various Docker images and how to work with them, see the full :ref:`docker-environment` documentation. -Once all containers are running, you can use the web interface by -opening http://localhost:5080/ in your web browser. +Once all containers are running, you can use the web interface by opening +http://localhost:<nginx-port>/ in your web browser. ``<nginx-port>`` is the +port on which nginx is exposed to the host. By default, it is randomly +attributed by docker. Use: + +.. code-block:: console + + ~/swh-docker$ docker compose port nginx 80 + +To find which port is actually used. + +.. note:: + + Please read the "Exposed Ports" section of the README file in the + `swh-docker`_ repository for more details and options on this topic. + +.. _`swh-docker`: https://gitlab.softwareheritage.org/swh/devel/docker.git + At this point, the archive is empty and needs to be filled with some content. The simplest way to start loading software is to use the *Save Code Now* feature of the archive web interface: - http://localhost:5080/browse/origin/save/ + http://localhost:<nginx-port>/browse/origin/save/ You can also use the command line interface to inject code. For example to retrieve projects hossted on the https://0xacab.org GitLab forge: @@ -186,8 +202,8 @@ Monitor your local installation You can monitor your local installation by looking at: -- http://localhost:5080/rabbitmq to access the rabbitmq dashboard (guest/guest), -- http://localhost:5080/grafana to explore the platform's metrics (admin/admin), +- http://localhost:<nginx-port>/rabbitmq to access the rabbitmq dashboard (guest/guest), +- http://localhost:<nginx-port>/grafana to explore the platform's metrics (admin/admin), Shut down your local installation --------------------------------- -- GitLab From 1c0a615f8b245a3879855bc0a596568847c1a86c Mon Sep 17 00:00:00 2001 From: David Douard <david.douard@sdfa3.org> Date: Thu, 28 Mar 2024 12:08:35 +0100 Subject: [PATCH 4/4] devel/developer-setup: use code-block:: console for cli snippets --- docs/devel/developer-setup.rst | 60 +++++++++++++++++++++++++--------- 1 file changed, 45 insertions(+), 15 deletions(-) diff --git a/docs/devel/developer-setup.rst b/docs/devel/developer-setup.rst index c1319f4d..f490db33 100644 --- a/docs/devel/developer-setup.rst +++ b/docs/devel/developer-setup.rst @@ -22,7 +22,9 @@ Install required dependencies ----------------------------- Software Heritage requires some dependencies that are usually packaged by your -package manager. On Debian/Ubuntu-based distributions:: +package manager. On Debian/Ubuntu-based distributions: + +.. code-block:: console sudo apt install lsb-release wget apt-transport-https sudo wget https://www.postgresql.org/media/keys/ACCC4CF8.asc -O /etc/apt/trusted.gpg.d/postgresql.asc @@ -45,13 +47,17 @@ This installs basic system utilities, Python library dependencies, development t documentation tools and our main database management systems. Cassandra and PostgreSQL will be started by tests when they need it, so you -don't need them started globally (this will save you some RAM):: +don't need them started globally (this will save you some RAM): + +.. code-block:: console sudo systemctl disable --now cassandra postgresql If you intend to hack on the frontend part of |swh| Web Applications, you will also need to have ``nodejs >= 18`` in your development environment. If the version in your -Debian-based distribution is lower, you can install node 18 using these commands:: +Debian-based distribution is lower, you can install node 18 using these commands: + +.. code-block:: console sudo wget https://deb.nodesource.com/gpgkey/nodesource.gpg.key -O /etc/apt/trusted.gpg.d/nodesource.asc echo "deb https://deb.nodesource.com/node_18.x $(lsb_release -cs) main" | sudo tee -a /etc/apt/sources.list.d/nodesource.list @@ -60,7 +66,9 @@ Debian-based distribution is lower, you can install node 18 using these commands Also related to Web Applications development, |swh| uses the ``yarn`` package manager to retrieve frontend dependencies and development tools. It is recommended to install its -latest classic version using these commands:: +latest classic version using these commands: + +.. code-block:: console sudo wget https://dl.yarnpkg.com/debian/pubkey.gpg -O /etc/apt/trusted.gpg.d/yarn.asc echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list @@ -68,7 +76,9 @@ latest classic version using these commands:: sudo apt install yarn If you intend to work on |swh| archive search features, Elasticsearch must also be -present in your development environment. Proceed as follows to install it:: +present in your development environment. Proceed as follows to install it: + +.. code-block:: console sudo wget https://artifacts.elastic.co/GPG-KEY-elasticsearch -O /etc/apt/trusted.gpg.d/elasticsearch.asc echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch.list @@ -84,30 +94,40 @@ to do so. Checkout the source code ------------------------ -Clone the |swh| environment repository:: +Clone the |swh| environment repository: + +.. code-block:: console ~$ git clone https://gitlab.softwareheritage.org/swh/devel/swh-environment.git [...] ~$ cd swh-environment ~/swh-environment$ -Create a virtualenv:: +Create a virtualenv: + +.. code-block:: console ~/swh-environment$ source /usr/share/virtualenvwrapper/virtualenvwrapper.sh ~/swh-environment$ mkvirtualenv -p /usr/bin/python3 -a $PWD swh [...] (swh) ~/swh-environment$ -Checkout all the swh packages source repositories:: +Checkout all the swh packages source repositories: + +.. code-block:: console (swh) ~/swh-environment$ pip install pre-commit (swh) ~/swh-environment$ ./bin/update -Use the same mypy version our tox containers use:: +Use the same mypy version our tox containers use: + +.. code-block:: console (swh) ~/swh-environment$ pip install mypy==1.8.0 -In the future you can re-activate the created virtualenv with:: +In the future you can re-activate the created virtualenv with: + +.. code-block:: console $ workon swh (swh) ~/swh-environment$ @@ -122,7 +142,9 @@ In the future you can re-activate the created virtualenv with:: .. _pipenv: https://pipenv.readthedocs.io/ -Install all the swh packages (in development mode, with testing dependencies):: +Install all the swh packages (in development mode, with testing dependencies): + +.. code-block:: console (swh) ~/swh-environment$ bin/install @@ -131,7 +153,9 @@ Install all the swh packages (in development mode, with testing dependencies):: If you experience issues with :program:`pip` dependency resolution, try with ``bin/install --use-deprecated=legacy-resolver`` (the flag will be passed on to ``pip install``). The same flag can also be set globally in - :file:`~/.config/pip/pip.conf`:: + :file:`~/.config/pip/pip.conf`: + + .. code-block:: console [install] use-deprecated=legacy-resolver @@ -151,7 +175,9 @@ tox_. The main difference between these 2 test execution environments is: current virtualenv, installed from the git repositories: you test your modification against the HEAD of every swh package. -For example, running unit tests for the swh-loader-git_ package:: +For example, running unit tests for the swh-loader-git_ package: + +.. code-block:: console (swh) ~/swh-environment$ cd swh-loader-git (swh) ~/swh-environment/swh-loader-git$ pytest @@ -171,7 +197,9 @@ For example, running unit tests for the swh-loader-git_ package:: [...] ================== 25 passed, 12 warnings in 6.66 seconds ================== -Running the same test, plus code linting and static analysis, using tox:: +Running the same test, plus code linting and static analysis, using tox: + +.. code-block:: console (swh) ~/swh-environment/swh-loader-git$ tox GLOB sdist-make: ~/swh-environment/swh-loader-git/setup.py @@ -235,7 +263,9 @@ Running the same test, plus code linting and static analysis, using tox:: Beware that some swh packages require a postgresql server properly configured to execute the tests. In this case, you will want to use pifpaf_, which will spawn a temporary instance of postgresql, to encapsulate the call to pytest. -For example, running pytest in the swh-core package:: +For example, running pytest in the swh-core package: + +.. code-block:: console (swh) ~/swh-environment$ cd swh-core (swh) ~/swh-environment/swh-core$ pifpaf run postgresql -- pytest -- GitLab