Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • ardumont/swh-apps
  • vlorentz/swh-apps
  • swh/infra/swh-apps
  • anlambert/swh-apps
4 results
Show changes
Commits on Source (4225)
4,125 additional commits have been omitted to prevent performance issues.
Hide whitespace changes
Inline Side-by-side
Showing
with 1382 additions and 5 deletions
.pre-commit-config.yaml
View file @ fc0214bb
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.3.0
rev: v4.6.0
hooks:
- id: trailing-whitespace
- id: check-json
- id: check-yaml
- repo: https://github.com/pycqa/flake8
rev: 5.0.4
rev: 7.1.0
hooks:
- id: flake8
additional_dependencies: [flake8-bugbear==22.9.23]
- repo: https://github.com/codespell-project/codespell
rev: v2.2.2
rev: v2.3.0
hooks:
- id: codespell
name: Check source code spelling
......@@ -30,11 +30,11 @@ repos:
types: [python]
- repo: https://github.com/PyCQA/isort
rev: 5.10.1
rev: 5.13.2
hooks:
- id: isort
- repo: https://github.com/python/black
rev: 22.10.0
rev: 24.4.2
hooks:
- id: black
README.rst
View file @ fc0214bb
......@@ -38,3 +38,184 @@ file. Out of this, a virtualenv is generated and frozen into a
The frozen requirements file can also be used to deploy virtual environments
directly, e.g. on an existing VM or bare metal server.
Local development
-----------------
App-manager
~~~~~~~~~~~
It's the cli tools which allows to execute various actions:
- list dependency between swh modules
- manipulate the generation of frozen requirements for our python applications
- helm chart dependency version update
- etc...
.. code::
# Build the app-manager image
docker build -t app-manager scripts
List dependent applications
~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is the listing command to list whatever other applications is depending
on a specification application version.
.. code::
docker run --rm -v $workspace/swh-apps:/src \
app-manager list-dependent-apps \
--application swh.graph --version v6.3.0
Example:
.. code::
docker run --rm -v $workspace/swh-apps:/src \
app-manager list-dependent-apps \
--application swh.graph --version v6.3.0
swh-alter
swh-graph
swh-provenance
swh-vault-cookers
swh-web
Freeze application dependencies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is in charge of using the existing requirements.txt declared besides the
application's Dockerfile and generate an updated frozen-requirements.txt. This
file is then used within the docker image to build an identical python
environments within the image.
Note that it does update any dependencies within the environments so any new
upstream libraries will get updated if new releases happened.
.. code::
docker run --rm -v $workspace/swh-apps:/src \
app-manager generate-frozen-requirements swh-provenance
.. code::
$ docker run --rm -v $workspace/swh-apps:/src \
app-manager generate-frozen-requirements swh-provenance
Collecting uv
Downloading uv-0.5.26-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.2/16.2 MB 36.4 MB/s eta 0:00:00
Installing collected packages: uv
Successfully installed uv-0.5.26
[notice] A new release of pip is available: 23.0.1 -> 25.0
[notice] To update, run: python3 -m pip install --upgrade pip
Using Python 3.10.15 environment at: /tmp/swh-provenancep16_3kh7
Resolved 3 packages in 46ms
Prepared 3 packages in 86ms
Uninstalled 2 packages in 51ms
Installed 3 packages in 32ms
- pip==23.0.1
+ pip==25.0
- setuptools==65.5.0
+ setuptools==75.8.0
+ wheel==0.45.1
# This file was autogenerated by uv via the following command:
# uv pip compile /src/apps/swh-provenance/requirements.txt -o /src/apps/swh-provenance/tmpk6i4m4rz
Resolved 76 packages in 850ms
aiohappyeyeballs==2.4.4
# via aiohttp
# via -r /src/apps/swh-provenance/requirements.txt
python-magic==0.4.27
# via swh-core
python-mimeparse==2.0.0
# via aiohttp-utils
pyyaml==6.0.2
# via swh-core
redis==5.2.1
# via
# via -r /src/apps/swh-provenance/requirements.txt
...
swh-storage==2.9.0
# via
# swh-dataset
# swh-provenance
tenacity==9.0.0
# via
# swh-core
# swh-journal
# swh-storage
tqdm==4.67.1
# via swh-dataset
types-protobuf==5.29.1.20241207
# via mypy-protobuf
types-requests==2.32.0.20241016
# via swh-dataset
typing-extensions==4.12.2
# via
# multidict
# swh-core
# swh-model
# swh-objstorage
# swh-storage
urllib3==2.3.0
# via
# botocore
# requests
# sentry-sdk
# types-requests
werkzeug==3.1.3
# via flask
wrapt==1.17.2
# via deprecated
yarl==1.18.3
# via aiohttp
Update swh-charts' version
~~~~~~~~~~~~~~~~~~~~~~~~~~
Providing access to both the swh-apps and the swh-charts repositories, this
allows to update the values of values-swh-application-versions.yaml with the
most recent docker image versions. This also bumps the Charts.yaml's version
incrementally.
.. code::
docker run --rm \
--volume $workspace/swh-apps:/src \
--volume $workspace/swh-charts:/tmp/swh-charts \
app-manager update-versions \
--applications-filepath /tmp/s-charts/values-swh-application-versions.yaml \
--chart-filepath /tmp/swh-charts/swh/Chart.yaml
Build application image
~~~~~~~~~~~~~~~~~~~~~~~
Each application is stored in swh-apps:/apps/$application_dir/ folder.
It usually holds the same set of files:
- Dockerfile: The set of instructions to build the application's docker image
- entrypoint.sh: The last Dockerfile instruction refers to this executable to run in the
image.
- requirements.txt: The set of python dependencies the application requires to run.
- requirements-frozen.txt: The derivative frozen set of dependencies to reproduce the
environment in the docker image. It's built by the app-manager out of the
requirements.txt file.
Cli call to build the application locally:
.. code::
$ DOCKER_BUILDKIT=1 docker build -t swh-toolbox:latest apps/swh-toolbox \
--build-arg REGISTRY=
Note: The REGISTRY is empty so we only rely to local docker image. You can avoid
changing it, in which case, it will use the swh's gitlab registry.
Cli call to run the application and be dropped in a shell:
.. code::
$ docker run -it swh-toolbox:latest shell
Note: This is specific command depending on the entrypoint.sh. In that case, the
entrypoint.sh allows a shell option to be dropped in an interactive bash session.
apps/cassandra-checks/Dockerfile 0 → 100644
View file @ fc0214bb
ARG REGISTRY=container-registry.softwareheritage.org/swh/infra/swh-apps/
ARG base_image=${REGISTRY}base
ARG base_image_version=latest
FROM ${base_image}:${base_image_version}
ARG user=swh
ARG workdir=/opt/${user}
ARG configdir=/etc/${user}
USER root
RUN mkdir /volume && chown root: /volume
COPY --chmod=0644 requirements-frozen.txt ${workdir}
RUN --mount=type=cache,target=.cache,uid=1000,gid=1000,uid=1000,gid=1000 \
uv pip sync requirements-frozen.txt
COPY --chmod=0755 entrypoint.sh ${workdir}
USER ${user}
ENV SWH_CONFIG_FILENAME=${configdir}/config.yml
ENTRYPOINT ["/opt/swh/entrypoint.sh"]
apps/cassandra-checks/entrypoint.sh 0 → 100644
View file @ fc0214bb
#!/bin/bash
set -e
CLONE_DIR=/opt/swh/snippets
if [ ! -d $CLONE_DIR ]; then
git clone --depth=1 https://gitlab.softwareheritage.org/swh/devel/snippets.git $CLONE_DIR
else
pushd $CLONE_DIR
git fetch
git merge origin/master
popd
fi
case "$1" in
"shell")
shift
if (( $# == 0)); then
exec bash -i
else
"$@"
fi
;;
*)
echo Starting Cassandra objects checker.
exec $CLONE_DIR/sysadmin/cassandra_checks/get_journal_check_and_replay.py
;;
esac
apps/cassandra-checks/requirements-frozen.txt 0 → 100644
View file @ fc0214bb
# This file was autogenerated by uv via the following command:
# uv pip compile /src/apps/cassandra-checks/requirements.txt -o /src/apps/cassandra-checks/tmpw7lyjvsq
aiohappyeyeballs==2.6.1
# via aiohttp
aiohttp==3.11.14
# via
# aiohttp-utils
# swh-core
aiohttp-utils==3.2.1
# via swh-core
aiosignal==1.3.2
# via aiohttp
attrs==25.3.0
# via
# aiohttp
# attrs-strict
# hypothesis
# swh-model
attrs-strict==1.0.1
# via swh-model
backports-entry-points-selectable==1.3.0
# via
# swh-core
# swh-storage
blinker==1.9.0
# via
# flask
# swh-core
cassandra-driver==3.29.2
# via swh-storage
certifi==2025.1.31
# via
# requests
# sentry-sdk
cffi==1.17.1
# via swh-perfecthash
charset-normalizer==3.4.1
# via requests
click==8.1.8
# via
# flask
# geomet
# swh-core
# swh-objstorage
# swh-storage
confluent-kafka==2.8.2
# via swh-journal
deprecated==1.2.18
# via
# swh-core
# swh-model
# swh-objstorage
# swh-storage
flask==3.1.0
# via
# swh-core
# swh-counters
# swh-storage
frozenlist==1.5.0
# via
# aiohttp
# aiosignal
geomet==0.2.1.post1
# via cassandra-driver
gunicorn==23.0.0
# via aiohttp-utils
hypothesis==6.130.0
# via swh-model
idna==3.10
# via
# requests
# yarl
iso8601==2.1.0
# via
# swh-core
# swh-model
# swh-storage
itsdangerous==2.2.0
# via flask
jinja2==3.1.6
# via flask
markupsafe==3.0.2
# via
# jinja2
# werkzeug
msgpack==1.1.0
# via
# swh-core
# swh-journal
# swh-objstorage
multidict==6.2.0
# via
# aiohttp
# yarl
mypy-extensions==1.0.0
# via swh-storage
packaging==24.2
# via gunicorn
propcache==0.3.0
# via
# aiohttp
# yarl
psycopg==3.2.6
# via
# swh-core
# swh-storage
psycopg-pool==3.2.6
# via
# swh-core
# swh-storage
pycparser==2.22
# via cffi
python-dateutil==2.9.0.post0
# via swh-model
python-magic==0.4.27
# via swh-core
python-mimeparse==2.0.0
# via aiohttp-utils
pyyaml==6.0.2
# via swh-core
redis==5.2.1
# via
# swh-counters
# swh-storage
requests==2.32.3
# via
# swh-core
# swh-objstorage
sentry-sdk==2.24.0
# via swh-core
six==1.17.0
# via
# geomet
# python-dateutil
sortedcontainers==2.4.0
# via hypothesis
swh-core==4.0.0
# via
# swh-counters
# swh-journal
# swh-objstorage
# swh-storage
swh-counters==0.11.0
# via swh-storage
swh-journal==1.5.3
# via swh-counters
swh-model==7.1.0
# via
# -r /src/apps/cassandra-checks/requirements.txt
# swh-journal
# swh-objstorage
# swh-storage
swh-objstorage==4.0.0
# via swh-storage
swh-perfecthash==1.3.2
# via swh-objstorage
swh-storage==3.0.0
# via -r /src/apps/cassandra-checks/requirements.txt
tenacity==9.0.0
# via
# swh-core
# swh-journal
# swh-storage
typing-extensions==4.12.2
# via
# psycopg
# psycopg-pool
# swh-core
# swh-model
# swh-storage
urllib3==2.3.0
# via
# requests
# sentry-sdk
werkzeug==3.1.3
# via flask
wrapt==1.17.2
# via deprecated
yarl==1.18.3
# via aiohttp
apps/cassandra-checks/requirements.txt 0 → 100644
View file @ fc0214bb
swh-storage
swh-model
apps/ipfs-gateway/Dockerfile 0 → 100644
View file @ fc0214bb
#######
# build
#######
# TODO: use an swh repository if it becomes a permanent thing
ARG REPOSITORY=https://github.com/obsidiansystems/go-ipfs-swh-plugin.git
ARG GIT_HEAD=2380683e3309e2c5bb167dbbbcb50f71eed063bb # master at 2023-01-31
FROM debian:bullseye as build
RUN apt update -y && apt dist-upgrade -y && apt install -y git nix
RUN mkdir /src && cd /src && \
git clone --depth 1 https://github.com/obsidiansystems/go-ipfs-swh-plugin.git && \
cd go-ipfs-swh-plugin && \
git checkout $HEAD
RUN cd /src/go-ipfs-swh-plugin && \
nix-build && \
nix-store --export $(nix-store --query --requisites ./result) > ipfs.dump
# #############
# # final image
# #############
FROM debian:bullseye
RUN apt update && apt dist-upgrade -y && apt install -y nix && apt clean
COPY --from=build /src/go-ipfs-swh-plugin/ipfs.dump /tmp
# Install ipfs cli with the minimalist runtime dependencies set
RUN nix-store --import < /tmp/ipfs.dump && \
update-alternatives --install /usr/bin/ipfs ipfs /nix/store/*ipfs*/bin/ipfs 1
RUN addgroup --gid 1000 ipfs && \
useradd --gid 1000 --uid 1000 -m ipfs
USER ipfs
WORKDIR /home/ipfs
USER ipfs
WORKDIR /data/ipfs
ENV IPFS_PATH /data/ipfs
# | true to ignore the empty directory error:
# https://github.com/ipfs/kubo/issues/9155
RUN ipfs init -e -p swhbridge || true && \
ipfs config Addresses.API /ip4/0.0.0.0/tcp/5001 && \
ipfs config Addresses.Gateway /ip4/0.0.0.0/tcp/8080
ENTRYPOINT [ "ipfs" ]
CMD ["daemon"]
apps/ipfs-gateway/Readme.md 0 → 100644
View file @ fc0214bb
# IPFS swh gateway
## Usage
To be able to retrieve the swh contents, the gateway uses a private SWH api endpoint.
To do so, an SWH api token must be configured in the gateway configuration.
- Bootstrap the gateway
```
mkdir /srv/ipfs # This is where the ipfs data will be stored
docker run --rm -ti -v /tmp/test:/data/ipfs --user $UID \
container-registry.softwareheritage.org/swh/infra/swh-apps/ipfs-gateway \
initipfs init -e -p swhbridge
```
WARN: ignore the ipld error (https://github.com/ipfs/kubo/issues/9155)
- Edit the config file `/srv/ipfs/config` and search the Datastore.Spec.mounts section.
- In the swhbridge child, add an `auth-token` entry:
```json
{
Datastore {
Spec {
"mounts": [
{
"child": {
"type": "swhbridge",
"auth-token: "<replace by the very long token>"
},
]
}
}
}
```
- Start the gateway with:
```
docker run -d --name ipfs-gateway -p 5001:5001 -p 4001:4001/udp \
-v /srv/ipfs:/data/ipfs \
container-registry.softwareheritage.org/swh/infra/swh-apps/ipfs-gateway
```
The gateway detects the available interfaces and tries to guess its public ip address.
```
Swarm listening on /ip4/127.0.0.1/tcp/4001
Swarm listening on /ip4/127.0.0.1/udp/4001/quic
Swarm listening on /ip4/172.17.0.2/tcp/4001
Swarm listening on /ip4/172.17.0.2/udp/4001/quic
Swarm listening on /p2p-circuit
Swarm announcing /ip4/127.0.0.1/tcp/4001
Swarm announcing /ip4/127.0.0.1/udp/4001/quic
Swarm announcing /ip4/172.17.0.2/tcp/4001
Swarm announcing /ip4/172.17.0.2/udp/4001/quic
Swarm announcing /ip4/1.2.3.4/udp/4001/quic <------ the detected public address
API server listening on /ip4/0.0.0.0/tcp/5001
WebUI: http://0.0.0.0:5001/webui
Gateway (readonly) server listening on /ip4/0.0.0.0/tcp/8080
```
If the annouces are not correct, it can be fixed in the configuration, the section `Adresses.Announce` and `Addresses.AppendAnnounce`
**To be reachable from the internet, the port 4001 must be routed to the container**
## Testing
Once the gateway is up and publicly reachable, it can be tested like this:
```
docker run -ti --name ipfs-client --rm ipfs/kubo
```
On another terminal:
```
docker exec client ipfs dag get \
--output-codec=git-raw \
f0178111494a9ed024d3859793618152ea559a168bbcbb5e2
```
For more details on the ipfs usage, check at the ipfs gateway main repository:
https://github.com/obsidiansystems/go-ipfs-swh-plugin.git
apps/software-stories/Dockerfile 0 → 100644
View file @ fc0214bb
FROM node:19-bullseye
RUN apt-get -y update && \
apt-get -y upgrade && \
npm install -g serve && \
mkdir -p /opt/swh/swh-stories && \
chown -R node:node /opt/swh
USER node
ENV NODE_OPTIONS=--openssl-legacy-provider
WORKDIR /opt/swh/swh-stories
RUN git clone https://github.com/ScienceStories/swh-stories.git /opt/swh/swh-stories && \
npm install --no-audit && \
npm run build
COPY --chown=node:node entrypoint.sh /opt/swh
RUN chmod u+x /opt/swh/entrypoint.sh
ENTRYPOINT ["/opt/swh/entrypoint.sh"]
apps/software-stories/entrypoint.sh 0 → 100644
View file @ fc0214bb
#!/bin/bash
set -e
case "$1" in
"shell")
shift
echo "Running command $@"
exec bash -i "$@"
;;
*)
exec serve -s build
;;
esac
apps/swh-add-forge-now/Dockerfile 0 → 100644
View file @ fc0214bb
ARG REGISTRY=container-registry.softwareheritage.org/swh/infra/swh-apps/
ARG base_image=${REGISTRY}base
ARG base_image_version=latest
FROM ${base_image}:${base_image_version}
ARG user=swh
ARG workdir=/opt/${user}
ARG configdir=/etc/${user}
USER root
RUN apt-get -y update && \
apt-get install -y netcat-openbsd jq && \
apt-get clean
COPY --chmod=0644 requirements-frozen.txt ${workdir}
RUN --mount=type=cache,target=.cache,uid=1000,gid=1000 \
uv pip sync requirements-frozen.txt
USER ${user}
CMD [ "/bin/bash", "-i" ]
apps/swh-add-forge-now/requirements-frozen.txt 0 → 100644
View file @ fc0214bb
# This file was autogenerated by uv via the following command:
# uv pip compile /src/apps/swh-add-forge-now/requirements.txt -o /src/apps/swh-add-forge-now/tmp5l99kmcs
aiofiles==24.1.0
# via python-keycloak
aiohappyeyeballs==2.6.1
# via aiohttp
aiohttp==3.11.14
# via
# aiohttp-utils
# swh-core
aiohttp-utils==3.2.1
# via swh-core
aiosignal==1.3.2
# via aiohttp
amqp==5.3.1
# via kombu
anyio==4.9.0
# via httpx
async-property==0.2.2
# via python-keycloak
attrs==25.3.0
# via
# aiohttp
# attrs-strict
# hypothesis
# swh-model
# swh-scheduler
attrs-strict==1.0.1
# via
# swh-model
# swh-scheduler
backports-entry-points-selectable==1.3.0
# via
# swh-core
# swh-storage
billiard==4.2.1
# via celery
blinker==1.9.0
# via
# flask
# swh-core
cassandra-driver==3.29.2
# via swh-storage
celery==5.4.0
# via swh-scheduler
certifi==2025.1.31
# via
# httpcore
# httpx
# requests
# sentry-sdk
cffi==1.17.1
# via
# cryptography
# swh-perfecthash
charset-normalizer==3.4.1
# via requests
click==8.1.8
# via
# celery
# click-didyoumean
# click-plugins
# click-repl
# flask
# geomet
# swh-auth
# swh-core
# swh-objstorage
# swh-scheduler
# swh-storage
click-didyoumean==0.3.1
# via celery
click-plugins==1.1.1
# via celery
click-repl==0.3.0
# via celery
confluent-kafka==2.8.2
# via swh-journal
cryptography==44.0.2
# via jwcrypto
deprecated==1.2.18
# via
# swh-core
# swh-model
# swh-objstorage
# swh-storage
deprecation==2.1.0
# via python-keycloak
flask==3.1.0
# via
# swh-core
# swh-counters
# swh-scheduler
# swh-storage
frozenlist==1.5.0
# via
# aiohttp
# aiosignal
geomet==0.2.1.post1
# via cassandra-driver
gunicorn==23.0.0
# via aiohttp-utils
h11==0.14.0
# via httpcore
httpcore==1.0.7
# via httpx
httpx==0.28.1
# via python-keycloak
humanize==4.12.1
# via swh-scheduler
hypothesis==6.130.0
# via swh-model
idna==3.10
# via
# anyio
# httpx
# requests
# yarl
importlib-metadata==8.6.1
# via swh-scheduler
iso8601==2.1.0
# via
# swh-core
# swh-model
# swh-storage
itsdangerous==2.2.0
# via flask
jinja2==3.1.6
# via flask
jwcrypto==1.5.6
# via python-keycloak
kombu==5.5.0
# via celery
markupsafe==3.0.2
# via
# jinja2
# werkzeug
msgpack==1.1.0
# via
# swh-core
# swh-journal
# swh-objstorage
multidict==6.2.0
# via
# aiohttp
# yarl
mypy-extensions==1.0.0
# via swh-storage
packaging==24.2
# via
# deprecation
# gunicorn
pika==1.3.2
# via swh-scheduler
prompt-toolkit==3.0.50
# via click-repl
propcache==0.3.0
# via
# aiohttp
# yarl
psycopg==3.2.6
# via
# swh-core
# swh-scheduler
# swh-storage
psycopg-pool==3.2.6
# via
# swh-core
# swh-scheduler
# swh-storage
pycparser==2.22
# via cffi
python-dateutil==2.9.0.post0
# via
# celery
# swh-model
python-keycloak==5.3.1
# via swh-auth
python-magic==0.4.27
# via swh-core
python-mimeparse==2.0.0
# via aiohttp-utils
pyyaml==6.0.2
# via
# swh-auth
# swh-core
# swh-scheduler
redis==5.2.1
# via
# swh-counters
# swh-storage
requests==2.32.3
# via
# python-keycloak
# requests-toolbelt
# swh-core
# swh-objstorage
# swh-scheduler
requests-toolbelt==1.0.0
# via python-keycloak
sentry-sdk==2.24.0
# via
# swh-core
# swh-scheduler
setuptools==77.0.3
# via swh-scheduler
six==1.17.0
# via
# geomet
# python-dateutil
sniffio==1.3.1
# via anyio
sortedcontainers==2.4.0
# via hypothesis
swh-auth==0.10.0
# via -r /src/apps/swh-add-forge-now/requirements.txt
swh-core==4.0.0
# via
# swh-auth
# swh-counters
# swh-journal
# swh-objstorage
# swh-scheduler
# swh-storage
swh-counters==0.11.0
# via swh-storage
swh-journal==1.5.3
# via swh-counters
swh-model==7.1.0
# via
# swh-journal
# swh-objstorage
# swh-storage
swh-objstorage==4.0.0
# via swh-storage
swh-perfecthash==1.3.2
# via swh-objstorage
swh-scheduler==3.0.0
# via -r /src/apps/swh-add-forge-now/requirements.txt
swh-storage==3.0.0
# via swh-scheduler
tabulate==0.9.0
# via swh-scheduler
tenacity==9.0.0
# via
# swh-core
# swh-journal
# swh-storage
typing-extensions==4.12.2
# via
# anyio
# jwcrypto
# psycopg
# psycopg-pool
# swh-core
# swh-model
# swh-scheduler
# swh-storage
tzdata==2025.1
# via
# celery
# kombu
urllib3==2.3.0
# via
# requests
# sentry-sdk
vine==5.1.0
# via
# amqp
# celery
# kombu
wcwidth==0.2.13
# via prompt-toolkit
werkzeug==3.1.3
# via flask
wrapt==1.17.2
# via deprecated
yarl==1.18.3
# via aiohttp
zipp==3.21.0
# via importlib-metadata
apps/swh-add-forge-now/requirements.txt 0 → 100644
View file @ fc0214bb
swh-auth
swh-scheduler
apps/swh-alter/Dockerfile 0 → 100644
View file @ fc0214bb
ARG REGISTRY=container-registry.softwareheritage.org/swh/infra/swh-apps/
ARG base_image=${REGISTRY}base
ARG base_image_version=latest
FROM ${base_image}:${base_image_version}
ARG user=swh
ARG workdir=/opt/${user}
USER root
RUN apt-get update && \
apt-get -y upgrade && \
apt-get install -y \
curl \
dnsutils \
emacs-nox \
host \
jq \
libcmph-dev \
librdkafka-dev \
libsystemd-dev \
netcat-openbsd \
postgresql-client \
sed \
strace \
telnet \
tmux \
vim && \
apt-get clean
COPY --chmod=0644 requirements-frozen.txt ${workdir}
RUN --mount=type=cache,target=.cache,uid=1000,gid=1000 \
uv pip sync requirements-frozen.txt
USER ${user}
ENV CARGO_HOME="${workdir}/.cargo"
ENV RUSTUP_HOME="${workdir}/.rustup"
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs -o /tmp/rustup-init.sh && \
chmod +x /tmp/rustup-init.sh && \
/tmp/rustup-init.sh -y && \
rm /tmp/rustup-init.sh
ENV PATH="${CARGO_HOME}/bin:${PATH}"
# install rage (for swh-alter)
# Use the latest available version so we can detect if rage behavior has changed when
# testing swh-alter. Otherwise, this might only be detected when processing a
# takedown notice or restoring a bundle… which would not be a nice experience.
RUN --mount=type=cache,target=.cache,uid=1000,gid=1000 \
cargo install rage
COPY --chmod=0755 entrypoint.sh ${workdir}
ENTRYPOINT ["/opt/swh/entrypoint.sh"]
apps/swh-alter/entrypoint.sh 0 → 100644
View file @ fc0214bb
#!/bin/bash
set -e
case "$1" in
"shell")
shift
if (( $# == 0)); then
exec bash -i
else
"$@"
fi
exit 0
;;
*)
echo "swh-alter pod: running a tmux server"
tmux set -g exit-empty off \; start
trap "tmux kill-server" INT TERM EXIT
while tmux list-sessions 2>/dev/null; do sleep 5; done
echo "tmux server has exited, bailing"
;;
esac
apps/swh-alter/requirements-frozen.txt 0 → 100644
View file @ fc0214bb
# This file was autogenerated by uv via the following command:
# uv pip compile /src/apps/swh-alter/requirements.txt -o /src/apps/swh-alter/tmpwgbqe86q
aiohappyeyeballs==2.6.1
# via aiohttp
aiohttp==3.11.14
# via
# aiohttp-utils
# swh-core
# swh-graph
aiohttp-utils==3.2.1
# via swh-core
aiosignal==1.3.2
# via aiohttp
apache-libcloud==3.8.0
# via swh-objstorage
attrs==25.3.0
# via
# aiohttp
# attrs-strict
# hypothesis
# swh-alter
# swh-model
attrs-strict==1.0.1
# via swh-model
azure-core==1.32.0
# via azure-storage-blob
azure-storage-blob==12.25.0
# via swh-objstorage
backports-entry-points-selectable==1.3.0
# via
# swh-core
# swh-storage
blinker==1.9.0
# via
# flask
# swh-core
boto3==1.37.17
# via swh-graph
botocore==1.37.17
# via
# boto3
# s3transfer
cachetools==5.5.2
# via pyld
cassandra-driver==3.29.2
# via swh-storage
certifi==2025.1.31
# via
# elasticsearch
# requests
# sentry-sdk
cffi==1.17.1
# via
# cryptography
# swh-perfecthash
charset-normalizer==3.4.1
# via requests
click==8.1.8
# via
# flask
# geomet
# swh-alter
# swh-core
# swh-graph
# swh-indexer
# swh-objstorage
# swh-search
# swh-storage
confluent-kafka==2.8.2
# via swh-journal
cryptography==44.0.2
# via azure-storage-blob
deprecated==1.2.18
# via
# swh-core
# swh-model
# swh-objstorage
# swh-storage
elasticsearch==7.17.12
# via swh-search
flask==3.1.0
# via
# swh-core
# swh-counters
# swh-storage
frozendict==2.4.6
# via
# pyld
# swh-indexer
frozenlist==1.5.0
# via
# aiohttp
# aiosignal
geomet==0.2.1.post1
# via cassandra-driver
grpcio==1.71.0
# via grpcio-tools
grpcio-tools==1.71.0
# via swh-graph
gunicorn==23.0.0
# via aiohttp-utils
humanize==4.12.1
# via swh-alter
hypothesis==6.130.0
# via swh-model
idna==3.10
# via
# requests
# yarl
igraph==0.11.8
# via swh-alter
iso8601==2.1.0
# via
# swh-core
# swh-indexer
# swh-model
# swh-search
# swh-storage
isodate==0.7.2
# via azure-storage-blob
itsdangerous==2.2.0
# via flask
jinja2==3.1.6
# via flask
jmespath==1.0.1
# via
# boto3
# botocore
latexcodec==3.0.0
# via pybtex
lxml==5.3.1
# via pyld
markupsafe==3.0.2
# via
# jinja2
# werkzeug
msgpack==1.1.0
# via
# swh-core
# swh-journal
# swh-objstorage
multidict==6.2.0
# via
# aiohttp
# yarl
mypy-extensions==1.0.0
# via swh-storage
mypy-protobuf==3.6.0
# via swh-graph
packaging==24.2
# via gunicorn
propcache==0.3.0
# via
# aiohttp
# yarl
protobuf==5.29.4
# via
# grpcio-tools
# mypy-protobuf
# swh-graph
psutil==7.0.0
# via swh-graph
psycopg==3.2.6
# via
# swh-core
# swh-storage
psycopg-pool==3.2.6
# via
# swh-core
# swh-storage
py4j==0.10.9.9
# via swh-graph
pybtex==0.24.0
# via swh-indexer
pycparser==2.22
# via cffi
pyld==2.0.4
# via swh-indexer
pyparsing==3.2.1
# via rdflib
python-dateutil==2.9.0.post0
# via
# botocore
# swh-model
python-magic==0.4.27
# via
# swh-core
# swh-indexer
python-mimeparse==2.0.0
# via aiohttp-utils
pyyaml==6.0.2
# via
# pybtex
# swh-alter
# swh-core
rdflib==7.1.3
# via swh-indexer
redis==5.2.1
# via
# swh-counters
# swh-storage
requests==2.32.3
# via
# apache-libcloud
# azure-core
# swh-core
# swh-objstorage
s3transfer==0.11.4
# via boto3
sentry-sdk==2.24.0
# via
# swh-core
# swh-indexer
setuptools==77.0.3
# via grpcio-tools
shamir-mnemonic==0.3.0
# via swh-alter
six==1.17.0
# via
# azure-core
# geomet
# pybtex
# python-dateutil
sortedcontainers==2.4.0
# via hypothesis
swh-alter==1.3.0
# via -r /src/apps/swh-alter/requirements.txt
swh-core==4.0.0
# via
# swh-alter
# swh-counters
# swh-graph
# swh-indexer
# swh-journal
# swh-objstorage
# swh-search
# swh-storage
swh-counters==0.11.0
# via swh-storage
swh-graph==6.7.1
# via swh-alter
swh-indexer==4.0.0
# via swh-search
swh-journal==1.5.3
# via
# swh-counters
# swh-indexer
# swh-search
swh-model==7.1.0
# via
# swh-alter
# swh-graph
# swh-indexer
# swh-journal
# swh-objstorage
# swh-search
# swh-storage
swh-objstorage==4.0.0
# via
# -r /src/apps/swh-alter/requirements.txt
# swh-alter
# swh-indexer
# swh-storage
swh-perfecthash==1.3.2
# via swh-objstorage
swh-search==0.22.1
# via swh-alter
swh-storage==3.0.0
# via
# swh-alter
# swh-indexer
tabulate==0.9.0
# via swh-alter
tenacity==9.0.0
# via
# swh-core
# swh-journal
# swh-storage
texttable==1.7.0
# via igraph
tree-sitter==0.21.3
# via swh-search
types-protobuf==5.29.1.20250315
# via mypy-protobuf
typing-extensions==4.12.2
# via
# azure-core
# azure-storage-blob
# psycopg
# psycopg-pool
# swh-alter
# swh-core
# swh-indexer
# swh-model
# swh-search
# swh-storage
urllib3==1.26.20
# via
# botocore
# elasticsearch
# requests
# sentry-sdk
werkzeug==3.1.3
# via flask
wrapt==1.17.2
# via deprecated
xmltodict==0.14.2
# via swh-indexer
yarl==1.18.3
# via aiohttp
apps/swh-alter/requirements.txt 0 → 100644
View file @ fc0214bb
swh.alter
swh.objstorage[azure,libcloud]
apps/swh-base/Dockerfile 0 → 100644
View file @ fc0214bb
FROM python:3.11-bookworm
ARG userid=1000
ARG groupid=1000
ARG user=swh
ARG workdir=/opt/${user}
ARG venv=${workdir}/venv
ARG configdir=/etc/${user}
RUN apt-get update && \
apt-get upgrade -y && \
apt-get install -y git sed \
gzip lzip zstd cpio \
less && \
apt-get clean && \
addgroup --gid ${userid} ${user} && \
useradd --gid ${groupid} --uid ${userid} -m -d ${workdir} ${user} && \
mkdir ${configdir}
USER ${user}
RUN python -m venv ${venv}
WORKDIR ${workdir}
ENV PYTHONPATH=${venv}
ENV PATH=${venv}/bin:$PATH
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
apps/swh-counters/Dockerfile 0 → 100644
View file @ fc0214bb
ARG REGISTRY=container-registry.softwareheritage.org/swh/infra/swh-apps/
ARG base_image=${REGISTRY}base
ARG base_image_version=latest
FROM ${base_image}:${base_image_version}
ARG user=swh
ARG workdir=/opt/${user}
ARG configdir=/etc/${user}
USER root
# retry command is used by the refresh-history cron job
# to ensure the rpc service is well up
RUN apt-get -y update && \
apt-get -y upgrade && \
apt-get install retry && \
apt-get clean
COPY --chmod=0644 requirements-frozen.txt ${workdir}
RUN --mount=type=cache,target=.cache,uid=1000,gid=1000 \
uv pip sync requirements-frozen.txt
COPY --chmod=0755 entrypoint.sh ${workdir}
USER ${user}
ENV SWH_CONFIG_FILENAME=${configdir}/config.yml
ENV PORT 5011
EXPOSE $PORT
ENV THREADS 2
ENV WORKERS 2
ENV TIMEOUT 3600
ENTRYPOINT ["/opt/swh/entrypoint.sh"]
apps/swh-counters/entrypoint.sh 0 → 100644
View file @ fc0214bb
#!/bin/bash
set -e
case "$1" in
"shell")
shift
if (( $# == 0)); then
exec bash -i
else
"$@"
fi
;;
"swh")
shift
echo "Running swh command $@"
exec swh $@
;;
*)
EXTRA_CLI_FLAGS=()
if [ -n "${SWH_LOG_CONFIG_JSON}" ]; then
EXTRA_CLI_FLAGS+=('--log-config-json' "${SWH_LOG_CONFIG_JSON}")
fi
if [ -n "${STATSD_HOST}" -a -n "${STATSD_PORT}" ]; then
EXTRA_CLI_FLAGS+=('--statsd-host' "${STATSD_HOST}:${STATSD_PORT}")
fi
if [ -n "${STATSD_SERVICE_TYPE}" ]; then
EXTRA_CLI_FLAGS+=('--statsd-prefix' "${STATSD_SERVICE_TYPE}")
fi
echo 'Starting the swh-counters API server'
exec gunicorn --bind "0.0.0.0:${PORT}" \
--log-level "${SWH_LOG_LEVEL:-INFO}" \
"${EXTRA_CLI_FLAGS[@]}" \
--threads "${THREADS}" \
--workers "${WORKERS}" \
--timeout "${TIMEOUT}" \
--reload \
--config 'python:swh.core.api.gunicorn_config' \
'swh.counters.api.server:make_app_from_configfile()'
esac