Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • ardumont/swh-apps
  • vlorentz/swh-apps
  • swh/infra/swh-apps
  • anlambert/swh-apps
4 results
Show changes
Commits on Source (1334)
1,234 additional commits have been omitted to prevent performance issues.
Hide whitespace changes
Inline Side-by-side
Showing
with 1404 additions and 453 deletions
README.rst
View file @ c65886ce
...@@ -38,3 +38,184 @@ file. Out of this, a virtualenv is generated and frozen into a ...@@ -38,3 +38,184 @@ file. Out of this, a virtualenv is generated and frozen into a
The frozen requirements file can also be used to deploy virtual environments The frozen requirements file can also be used to deploy virtual environments
directly, e.g. on an existing VM or bare metal server. directly, e.g. on an existing VM or bare metal server.
Local development
-----------------
App-manager
~~~~~~~~~~~
It's the cli tools which allows to execute various actions:
- list dependency between swh modules
- manipulate the generation of frozen requirements for our python applications
- helm chart dependency version update
- etc...
.. code::
# Build the app-manager image
docker build -t app-manager scripts
List dependent applications
~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is the listing command to list whatever other applications is depending
on a specification application version.
.. code::
docker run --rm -v $workspace/swh-apps:/src \
app-manager list-dependent-apps \
--application swh.graph --version v6.3.0
Example:
.. code::
docker run --rm -v $workspace/swh-apps:/src \
app-manager list-dependent-apps \
--application swh.graph --version v6.3.0
swh-alter
swh-graph
swh-provenance
swh-vault-cookers
swh-web
Freeze application dependencies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is in charge of using the existing requirements.txt declared besides the
application's Dockerfile and generate an updated frozen-requirements.txt. This
file is then used within the docker image to build an identical python
environments within the image.
Note that it does update any dependencies within the environments so any new
upstream libraries will get updated if new releases happened.
.. code::
docker run --rm -v $workspace/swh-apps:/src \
app-manager generate-frozen-requirements swh-provenance
.. code::
$ docker run --rm -v $workspace/swh-apps:/src \
app-manager generate-frozen-requirements swh-provenance
Collecting uv
Downloading uv-0.5.26-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.2/16.2 MB 36.4 MB/s eta 0:00:00
Installing collected packages: uv
Successfully installed uv-0.5.26
[notice] A new release of pip is available: 23.0.1 -> 25.0
[notice] To update, run: python3 -m pip install --upgrade pip
Using Python 3.10.15 environment at: /tmp/swh-provenancep16_3kh7
Resolved 3 packages in 46ms
Prepared 3 packages in 86ms
Uninstalled 2 packages in 51ms
Installed 3 packages in 32ms
- pip==23.0.1
+ pip==25.0
- setuptools==65.5.0
+ setuptools==75.8.0
+ wheel==0.45.1
# This file was autogenerated by uv via the following command:
# uv pip compile /src/apps/swh-provenance/requirements.txt -o /src/apps/swh-provenance/tmpk6i4m4rz
Resolved 76 packages in 850ms
aiohappyeyeballs==2.4.4
# via aiohttp
# via -r /src/apps/swh-provenance/requirements.txt
python-magic==0.4.27
# via swh-core
python-mimeparse==2.0.0
# via aiohttp-utils
pyyaml==6.0.2
# via swh-core
redis==5.2.1
# via
# via -r /src/apps/swh-provenance/requirements.txt
...
swh-storage==2.9.0
# via
# swh-dataset
# swh-provenance
tenacity==9.0.0
# via
# swh-core
# swh-journal
# swh-storage
tqdm==4.67.1
# via swh-dataset
types-protobuf==5.29.1.20241207
# via mypy-protobuf
types-requests==2.32.0.20241016
# via swh-dataset
typing-extensions==4.12.2
# via
# multidict
# swh-core
# swh-model
# swh-objstorage
# swh-storage
urllib3==2.3.0
# via
# botocore
# requests
# sentry-sdk
# types-requests
werkzeug==3.1.3
# via flask
wrapt==1.17.2
# via deprecated
yarl==1.18.3
# via aiohttp
Update swh-charts' version
~~~~~~~~~~~~~~~~~~~~~~~~~~
Providing access to both the swh-apps and the swh-charts repositories, this
allows to update the values of values-swh-application-versions.yaml with the
most recent docker image versions. This also bumps the Charts.yaml's version
incrementally.
.. code::
docker run --rm \
--volume $workspace/swh-apps:/src \
--volume $workspace/swh-charts:/tmp/swh-charts \
app-manager update-versions \
--applications-filepath /tmp/s-charts/values-swh-application-versions.yaml \
--chart-filepath /tmp/swh-charts/swh/Chart.yaml
Build application image
~~~~~~~~~~~~~~~~~~~~~~~
Each application is stored in swh-apps:/apps/$application_dir/ folder.
It usually holds the same set of files:
- Dockerfile: The set of instructions to build the application's docker image
- entrypoint.sh: The last Dockerfile instruction refers to this executable to run in the
image.
- requirements.txt: The set of python dependencies the application requires to run.
- requirements-frozen.txt: The derivative frozen set of dependencies to reproduce the
environment in the docker image. It's built by the app-manager out of the
requirements.txt file.
Cli call to build the application locally:
.. code::
$ DOCKER_BUILDKIT=1 docker build -t swh-toolbox:latest apps/swh-toolbox \
--build-arg REGISTRY=
Note: The REGISTRY is empty so we only rely to local docker image. You can avoid
changing it, in which case, it will use the swh's gitlab registry.
Cli call to run the application and be dropped in a shell:
.. code::
$ docker run -it swh-toolbox:latest shell
Note: This is specific command depending on the entrypoint.sh. In that case, the
entrypoint.sh allows a shell option to be dropped in an interactive bash session.
apps/cassandra-checks/Dockerfile
View file @ c65886ce
FROM python:3.10-bullseye ARG REGISTRY=container-registry.softwareheritage.org/swh/infra/swh-apps/
ARG base_image=${REGISTRY}base
ARG base_image_version=latest
RUN apt-get -y update && \ FROM ${base_image}:${base_image_version}
apt-get -y upgrade && \
apt-get install -y libpq-dev git && \
apt clean && \
addgroup --gid 1000 swh && \
useradd --gid 1000 --uid 1000 -m -d /opt/swh swh && \
mkdir /etc/swh
RUN mkdir /volume && chown root: /volume ARG user=swh
ARG workdir=/opt/${user}
ARG configdir=/etc/${user}
USER swh USER root
WORKDIR /opt/swh RUN mkdir /volume && chown root: /volume
COPY --chown=swh:swh requirements-frozen.txt /opt/swh COPY --chmod=0644 requirements-frozen.txt ${workdir}
COPY --chown=swh:swh entrypoint.sh /opt/swh RUN --mount=type=cache,target=.cache,uid=1000,gid=1000,uid=1000,gid=1000 \
uv pip sync requirements-frozen.txt
ENV PYTHONPATH=/opt/swh COPY --chmod=0755 entrypoint.sh ${workdir}
ENV PATH=/opt/swh/.local/bin:$PATH
RUN chmod u+x /opt/swh/entrypoint.sh && \ USER ${user}
/usr/local/bin/python -m pip install --upgrade pip && \
pip install --no-cache-dir -r requirements-frozen.txt
ENV SWH_CONFIG_FILENAME=/etc/swh/config.yml ENV SWH_CONFIG_FILENAME=${configdir}/config.yml
ENTRYPOINT ["/opt/swh/entrypoint.sh"] ENTRYPOINT ["/opt/swh/entrypoint.sh"]
apps/cassandra-checks/entrypoint.sh
View file @ c65886ce
...@@ -14,13 +14,16 @@ else ...@@ -14,13 +14,16 @@ else
fi fi
case "$1" in case "$1" in
"shell") "shell")
shift shift
echo "Running command $@" if (( $# == 0)); then
exec bash -i "$@" exec bash -i
;; else
*) "$@"
echo Starting Cassandra objects checker. fi
exec $CLONE_DIR/sysadmin/cassandra_checks/get_journal_check_and_replay.py ;;
;; *)
echo Starting Cassandra objects checker.
exec $CLONE_DIR/sysadmin/cassandra_checks/get_journal_check_and_replay.py
;;
esac esac
apps/cassandra-checks/requirements-frozen.txt
View file @ c65886ce
aiohappyeyeballs==2.3.5 # This file was autogenerated by uv via the following command:
aiohttp==3.10.3 # uv pip compile /src/apps/cassandra-checks/requirements.txt -o /src/apps/cassandra-checks/tmps_g2ik4c
aiohappyeyeballs==2.6.1
# via aiohttp
aiohttp==3.11.15
# via
# aiohttp-utils
# swh-core
aiohttp-utils==3.2.1 aiohttp-utils==3.2.1
aiosignal==1.3.1 # via swh-core
async-timeout==4.0.3 aiosignal==1.3.2
attrs==24.2.0 # via aiohttp
attrs==25.3.0
# via
# aiohttp
# attrs-strict
# hypothesis
# swh-model
attrs-strict==1.0.1 attrs-strict==1.0.1
blinker==1.8.2 # via swh-model
cassandra-driver==3.29.1 backports-entry-points-selectable==1.3.0
certifi==2024.7.4 # via
cffi==1.17.0 # swh-core
charset-normalizer==3.3.2 # swh-storage
click==8.1.7 blinker==1.9.0
confluent-kafka==2.5.0 # via
Deprecated==1.2.14 # flask
exceptiongroup==1.2.2 # swh-core
Flask==3.0.3 cassandra-driver==3.29.2
frozenlist==1.4.1 # via swh-storage
certifi==2025.1.31
# via
# requests
# sentry-sdk
cffi==1.17.1
# via swh-perfecthash
charset-normalizer==3.4.1
# via requests
click==8.1.8
# via
# flask
# geomet
# swh-core
# swh-objstorage
# swh-storage
deprecated==1.2.18
# via
# swh-core
# swh-model
# swh-objstorage
# swh-storage
flask==3.1.0
# via
# swh-core
# swh-storage
frozenlist==1.5.0
# via
# aiohttp
# aiosignal
geomet==0.2.1.post1 geomet==0.2.1.post1
# via cassandra-driver
gunicorn==23.0.0 gunicorn==23.0.0
hypothesis==6.111.0 # via aiohttp-utils
idna==3.7 hypothesis==6.130.6
# via swh-model
idna==3.10
# via
# requests
# yarl
iso8601==2.1.0 iso8601==2.1.0
# via
# swh-core
# swh-model
# swh-storage
itsdangerous==2.2.0 itsdangerous==2.2.0
Jinja2==3.1.4 # via flask
MarkupSafe==2.1.5 jinja2==3.1.6
msgpack==1.0.8 # via flask
multidict==6.0.5 markupsafe==3.0.2
# via
# jinja2
# werkzeug
msgpack==1.1.0
# via
# swh-core
# swh-objstorage
multidict==6.3.0
# via
# aiohttp
# yarl
mypy-extensions==1.0.0 mypy-extensions==1.0.0
packaging==24.1 # via swh-storage
psycopg2==2.9.9 packaging==24.2
# via gunicorn
propcache==0.3.1
# via
# aiohttp
# yarl
psycopg==3.2.6
# via
# swh-core
# swh-storage
psycopg-pool==3.2.6
# via
# swh-core
# swh-storage
pycparser==2.22 pycparser==2.22
# via cffi
python-dateutil==2.9.0.post0 python-dateutil==2.9.0.post0
# via swh-model
python-magic==0.4.27 python-magic==0.4.27
python-mimeparse==1.6.0 # via swh-core
PyYAML==6.0.2 python-mimeparse==2.0.0
redis==5.0.8 # via aiohttp-utils
pyyaml==6.0.2
# via swh-core
redis==5.2.1
# via swh-storage
requests==2.32.3 requests==2.32.3
sentry-sdk==2.12.0 # via
six==1.16.0 # swh-core
# swh-objstorage
sentry-sdk==2.25.0
# via swh-core
six==1.17.0
# via
# geomet
# python-dateutil
sortedcontainers==2.4.0 sortedcontainers==2.4.0
swh.core==3.4.0 # via hypothesis
swh.counters==0.11.0 swh-core==4.1.0
swh.journal==1.5.2 # via
swh.model==6.14.0 # swh-objstorage
swh.objstorage==3.3.0 # swh-storage
swh.perfecthash==1.3.2 swh-model==7.1.0
swh.storage==2.7.0 # via
# -r /src/apps/cassandra-checks/requirements.txt
# swh-objstorage
# swh-storage
swh-objstorage==4.0.0
# via swh-storage
swh-perfecthash==1.3.2
# via swh-objstorage
swh-storage==3.1.0
# via -r /src/apps/cassandra-checks/requirements.txt
tenacity==9.0.0 tenacity==9.0.0
typing_extensions==4.12.2 # via
urllib3==2.2.2 # swh-core
Werkzeug==3.0.3 # swh-storage
wrapt==1.16.0 typing-extensions==4.13.0
yarl==1.9.4 # via
# psycopg
# psycopg-pool
# swh-core
# swh-model
# swh-storage
urllib3==2.3.0
# via
# requests
# sentry-sdk
werkzeug==3.1.3
# via flask
wrapt==1.17.2
# via deprecated
yarl==1.18.3
# via aiohttp
apps/ipfs-gateway/Dockerfile
View file @ c65886ce
...@@ -40,7 +40,7 @@ WORKDIR /home/ipfs ...@@ -40,7 +40,7 @@ WORKDIR /home/ipfs
USER ipfs USER ipfs
WORKDIR /data/ipfs WORKDIR /data/ipfs
ENV IPFS_PATH /data/ipfs ENV IPFS_PATH=/data/ipfs
# | true to ignore the empty directory error: # | true to ignore the empty directory error:
# https://github.com/ipfs/kubo/issues/9155 # https://github.com/ipfs/kubo/issues/9155
......
apps/software-stories/Dockerfile
View file @ c65886ce
...@@ -10,13 +10,11 @@ USER node ...@@ -10,13 +10,11 @@ USER node
ENV NODE_OPTIONS=--openssl-legacy-provider ENV NODE_OPTIONS=--openssl-legacy-provider
WORKDIR /opt/swh/swh-stories WORKDIR /opt/swh/swh-stories
COPY --chown=node:node entrypoint.sh /opt/swh RUN git clone https://github.com/ScienceStories/swh-stories.git /opt/swh/swh-stories && \
RUN chmod u+x /opt/swh/entrypoint.sh && \
git clone https://github.com/ScienceStories/swh-stories.git /opt/swh/swh-stories && \
git checkout 7dcac1ac6edf376afc0a011465094b11c27683c5 && \
rm package-lock.json && \
npm install --no-audit && \ npm install --no-audit && \
npm run build npm run build
ENTRYPOINT "/opt/swh/entrypoint.sh" COPY --chown=node:node entrypoint.sh /opt/swh
RUN chmod u+x /opt/swh/entrypoint.sh
ENTRYPOINT ["/opt/swh/entrypoint.sh"]
apps/swh-add-forge-now/Dockerfile
View file @ c65886ce
FROM python:3.10-bookworm ARG REGISTRY=container-registry.softwareheritage.org/swh/infra/swh-apps/
ARG base_image=${REGISTRY}base
ARG base_image_version=latest
RUN apt-get -y update && \ FROM ${base_image}:${base_image_version}
apt-get -y upgrade && \
apt-get install -y libcmph-dev netcat-openbsd jq && \
apt clean && \
addgroup --gid 1000 swh && \
useradd --gid 1000 --uid 1000 -m -d /opt/swh swh
USER swh ARG user=swh
WORKDIR /opt/swh ARG workdir=/opt/${user}
ARG configdir=/etc/${user}
COPY --chown=swh:swh requirements-frozen.txt /opt/swh USER root
RUN apt-get -y update && \
apt-get install -y netcat-openbsd jq && \
apt-get clean
ENV PATH=/opt/swh/.local/bin:$PATH COPY --chmod=0644 requirements-frozen.txt ${workdir}
RUN --mount=type=cache,target=.cache,uid=1000,gid=1000 \
uv pip sync requirements-frozen.txt
RUN /usr/local/bin/python -m pip install --upgrade pip && \ USER ${user}
pip install --no-cache-dir -r requirements-frozen.txt
CMD [ "/bin/bash", "-i" ] CMD [ "/bin/bash", "-i" ]
apps/swh-add-forge-now/requirements-frozen.txt
View file @ c65886ce
aiohappyeyeballs==2.3.5 # This file was autogenerated by uv via the following command:
aiohttp==3.10.3 # uv pip compile /src/apps/swh-add-forge-now/requirements.txt -o /src/apps/swh-add-forge-now/tmp3md5j_f0
aiofiles==24.1.0
# via python-keycloak
aiohappyeyeballs==2.6.1
# via aiohttp
aiohttp==3.11.15
# via
# aiohttp-utils
# swh-core
aiohttp-utils==3.2.1 aiohttp-utils==3.2.1
aiosignal==1.3.1 # via swh-core
amqp==5.2.0 aiosignal==1.3.2
asn1crypto==1.5.1 # via aiohttp
async-timeout==4.0.3 amqp==5.3.1
attrs==24.2.0 # via kombu
anyio==4.9.0
# via httpx
async-property==0.2.2
# via python-keycloak
attrs==25.3.0
# via
# aiohttp
# attrs-strict
# hypothesis
# swh-model
# swh-scheduler
attrs-strict==1.0.1 attrs-strict==1.0.1
billiard==4.2.0 # via
blinker==1.8.2 # swh-model
cassandra-driver==3.29.1 # swh-scheduler
celery==5.4.0 backports-entry-points-selectable==1.3.0
certifi==2024.7.4 # via
cffi==1.17.0 # swh-core
charset-normalizer==3.3.2 # swh-storage
click==8.1.7 billiard==4.2.1
# via celery
blinker==1.9.0
# via
# flask
# swh-core
cassandra-driver==3.29.2
# via swh-storage
celery==5.5.0
# via swh-scheduler
certifi==2025.1.31
# via
# httpcore
# httpx
# requests
# sentry-sdk
cffi==1.17.1
# via
# cryptography
# swh-perfecthash
charset-normalizer==3.4.1
# via requests
click==8.1.8
# via
# celery
# click-didyoumean
# click-plugins
# click-repl
# flask
# geomet
# swh-auth
# swh-core
# swh-objstorage
# swh-scheduler
# swh-storage
click-didyoumean==0.3.1 click-didyoumean==0.3.1
# via celery
click-plugins==1.1.1 click-plugins==1.1.1
# via celery
click-repl==0.3.0 click-repl==0.3.0
confluent-kafka==2.5.0 # via celery
Deprecated==1.2.14 cryptography==44.0.2
# via jwcrypto
deprecated==1.2.18
# via
# swh-core
# swh-model
# swh-objstorage
# swh-storage
deprecation==2.1.0 deprecation==2.1.0
ecdsa==0.19.0 # via python-keycloak
exceptiongroup==1.2.2 flask==3.1.0
Flask==3.0.3 # via
frozenlist==1.4.1 # swh-core
# swh-scheduler
# swh-storage
frozenlist==1.5.0
# via
# aiohttp
# aiosignal
geomet==0.2.1.post1 geomet==0.2.1.post1
# via cassandra-driver
gunicorn==23.0.0 gunicorn==23.0.0
humanize==4.10.0 # via aiohttp-utils
hypothesis==6.111.0 h11==0.14.0
idna==3.7 # via httpcore
importlib_metadata==8.2.0 httpcore==1.0.7
# via httpx
httpx==0.28.1
# via python-keycloak
humanize==4.12.2
# via swh-scheduler
hypothesis==6.130.6
# via swh-model
idna==3.10
# via
# anyio
# httpx
# requests
# yarl
importlib-metadata==8.6.1
# via swh-scheduler
iso8601==2.1.0 iso8601==2.1.0
# via
# swh-core
# swh-model
# swh-storage
itsdangerous==2.2.0 itsdangerous==2.2.0
Jinja2==3.1.4 # via flask
kombu==5.4.0 jinja2==3.1.6
MarkupSafe==2.1.5 # via flask
msgpack==1.0.8 jwcrypto==1.5.6
multidict==6.0.5 # via python-keycloak
kombu==5.5.2
# via celery
markupsafe==3.0.2
# via
# jinja2
# werkzeug
msgpack==1.1.0
# via
# swh-core
# swh-objstorage
multidict==6.3.0
# via
# aiohttp
# yarl
mypy-extensions==1.0.0 mypy-extensions==1.0.0
packaging==24.1 # via swh-storage
pg8000==1.31.2 packaging==24.2
# via
# deprecation
# gunicorn
pika==1.3.2 pika==1.3.2
prompt_toolkit==3.0.47 # via swh-scheduler
psycopg2==2.9.9 prompt-toolkit==3.0.50
pyasn1==0.6.0 # via click-repl
propcache==0.3.1
# via
# aiohttp
# yarl
psycopg==3.2.6
# via
# swh-core
# swh-scheduler
# swh-storage
psycopg-pool==3.2.6
# via
# swh-core
# swh-scheduler
# swh-storage
pycparser==2.22 pycparser==2.22
# via cffi
python-dateutil==2.9.0.post0 python-dateutil==2.9.0.post0
python-jose==3.3.0 # via
python-keycloak==3.8.4 # celery
# swh-model
python-keycloak==5.3.1
# via swh-auth
python-magic==0.4.27 python-magic==0.4.27
python-mimeparse==1.6.0 # via swh-core
PyYAML==6.0.2 python-mimeparse==2.0.0
redis==5.0.8 # via aiohttp-utils
pyyaml==6.0.2
# via
# swh-auth
# swh-core
# swh-scheduler
redis==5.2.1
# via swh-storage
requests==2.32.3 requests==2.32.3
# via
# python-keycloak
# requests-toolbelt
# swh-core
# swh-objstorage
# swh-scheduler
requests-toolbelt==1.0.0 requests-toolbelt==1.0.0
rsa==4.9 # via python-keycloak
scramp==1.4.5 sentry-sdk==2.25.0
sentry-sdk==2.12.0 # via
six==1.16.0 # swh-core
# swh-scheduler
setuptools==78.1.0
# via swh-scheduler
six==1.17.0
# via
# geomet
# python-dateutil
sniffio==1.3.1
# via anyio
sortedcontainers==2.4.0 sortedcontainers==2.4.0
swh.auth==0.9.1 # via hypothesis
swh.core==3.4.0 swh-auth==0.10.0
swh.counters==0.11.0 # via -r /src/apps/swh-add-forge-now/requirements.txt
swh.journal==1.5.2 swh-core==4.1.0
swh.model==6.14.0 # via
swh.objstorage==3.3.0 # swh-auth
swh.perfecthash==1.3.2 # swh-objstorage
swh.scheduler==2.4.0 # swh-scheduler
swh.storage==2.7.0 # swh-storage
swh-model==7.1.0
# via
# swh-objstorage
# swh-storage
swh-objstorage==4.0.0
# via swh-storage
swh-perfecthash==1.3.2
# via swh-objstorage
swh-scheduler==3.1.0
# via -r /src/apps/swh-add-forge-now/requirements.txt
swh-storage==3.1.0
# via swh-scheduler
tabulate==0.9.0 tabulate==0.9.0
# via swh-scheduler
tenacity==9.0.0 tenacity==9.0.0
testing.common.database==2.0.3 # via
testing.postgresql==1.3.0 # swh-core
typing_extensions==4.12.2 # swh-storage
tzdata==2024.1 typing-extensions==4.13.0
urllib3==2.2.2 # via
# anyio
# jwcrypto
# psycopg
# psycopg-pool
# swh-core
# swh-model
# swh-scheduler
# swh-storage
tzdata==2025.2
# via kombu
urllib3==2.3.0
# via
# requests
# sentry-sdk
vine==5.1.0 vine==5.1.0
# via
# amqp
# celery
# kombu
wcwidth==0.2.13 wcwidth==0.2.13
Werkzeug==3.0.3 # via prompt-toolkit
wrapt==1.16.0 werkzeug==3.1.3
yarl==1.9.4 # via flask
zipp==3.20.0 wrapt==1.17.2
# via deprecated
yarl==1.18.3
# via aiohttp
zipp==3.21.0
# via importlib-metadata
apps/swh-alter/Dockerfile
View file @ c65886ce
FROM python:3.11-bullseye ARG REGISTRY=container-registry.softwareheritage.org/swh/infra/swh-apps/
ARG base_image=${REGISTRY}base
ARG base_image_version=latest
FROM ${base_image}:${base_image_version}
ARG user=swh ARG user=swh
ARG workdir=/opt/${user} ARG workdir=/opt/${user}
RUN apt-get -y update && \ USER root
RUN apt-get update && \
apt-get -y upgrade && \ apt-get -y upgrade && \
apt-get install -y \ apt-get install -y \
curl \ curl \
...@@ -20,15 +25,14 @@ RUN apt-get -y update && \ ...@@ -20,15 +25,14 @@ RUN apt-get -y update && \
strace \ strace \
telnet \ telnet \
tmux \ tmux \
vim \ vim && \
&& apt clean && \ apt-get clean
addgroup --gid 1000 ${user} && \
useradd --gid 1000 --uid 1000 -m -d ${workdir} ${user} && \
mkdir /etc/${user}
USER ${user} COPY --chmod=0644 requirements-frozen.txt ${workdir}
WORKDIR ${workdir} RUN --mount=type=cache,target=.cache,uid=1000,gid=1000 \
uv pip sync requirements-frozen.txt
USER ${user}
ENV CARGO_HOME="${workdir}/.cargo" ENV CARGO_HOME="${workdir}/.cargo"
ENV RUSTUP_HOME="${workdir}/.rustup" ENV RUSTUP_HOME="${workdir}/.rustup"
...@@ -38,23 +42,14 @@ RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs -o /tmp/rustup-ini ...@@ -38,23 +42,14 @@ RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs -o /tmp/rustup-ini
rm /tmp/rustup-init.sh rm /tmp/rustup-init.sh
ENV PATH="${CARGO_HOME}/bin:${PATH}" ENV PATH="${CARGO_HOME}/bin:${PATH}"
# install rage (for swh-alter) # install rage (for swh-alter)
# Use the latest available version so we can detect if rage behavior has changed when # Use the latest available version so we can detect if rage behavior has changed when
# testing swh-alter. Otherwise, this might only be detected when processing a # testing swh-alter. Otherwise, this might only be detected when processing a
# takedown notice or restoring a bundle… which would not be a nice experience. # takedown notice or restoring a bundle… which would not be a nice experience.
RUN cargo install rage RUN --mount=type=cache,target=.cache,uid=1000,gid=1000 \
cargo install rage
COPY --chown=${user}:${user} requirements-frozen.txt ${workdir}
ENV PYTHONPATH=${workdir}
ENV PATH=${workdir}/.local/bin:$PATH
RUN /usr/local/bin/python -m pip install --upgrade pip && \
pip install --no-cache-dir -r requirements-frozen.txt
COPY --chown=${user}:${user} entrypoint.sh ${workdir} COPY --chmod=0755 entrypoint.sh ${workdir}
RUN chmod u+x ${workdir}/entrypoint.sh
ENTRYPOINT "/opt/swh/entrypoint.sh" ENTRYPOINT ["/opt/swh/entrypoint.sh"]
apps/swh-alter/entrypoint.sh
View file @ c65886ce
...@@ -2,10 +2,23 @@ ...@@ -2,10 +2,23 @@
set -e set -e
echo "swh-alter pod: running a tmux server" case "$1" in
tmux set -g exit-empty off \; start "shell")
shift
if (( $# == 0)); then
exec bash -i
else
"$@"
fi
exit 0
;;
*)
echo "swh-alter pod: running a tmux server"
tmux set -g exit-empty off \; start
trap "tmux kill-server" INT TERM EXIT trap "tmux kill-server" INT TERM EXIT
while tmux list-sessions 2>/dev/null; do sleep 5; done while tmux list-sessions 2>/dev/null; do sleep 5; done
echo "tmux server has exited, bailing" echo "tmux server has exited, bailing"
;;
esac
apps/swh-alter/requirements-frozen.txt
View file @ c65886ce
aiohappyeyeballs==2.3.5 # This file was autogenerated by uv via the following command:
aiohttp==3.10.3 # uv pip compile /src/apps/swh-alter/requirements.txt -o /src/apps/swh-alter/tmp26clg_wc
aiohappyeyeballs==2.6.1
# via aiohttp
aiohttp==3.11.15
# via
# aiohttp-utils
# swh-core
# swh-graph
aiohttp-utils==3.2.1 aiohttp-utils==3.2.1
aiosignal==1.3.1 # via swh-core
aiosignal==1.3.2
# via aiohttp
apache-libcloud==3.8.0 apache-libcloud==3.8.0
async-timeout==4.0.3 # via swh-objstorage
attrs==24.2.0 attrs==25.3.0
# via
# aiohttp
# attrs-strict
# hypothesis
# swh-alter
# swh-model
attrs-strict==1.0.1 attrs-strict==1.0.1
azure-core==1.30.2 # via swh-model
azure-storage-blob==12.22.0 azure-core==1.32.0
blinker==1.8.2 # via azure-storage-blob
boto3==1.34.159 azure-storage-blob==12.25.1
botocore==1.34.159 # via swh-objstorage
cachetools==5.4.0 backports-entry-points-selectable==1.3.0
cassandra-driver==3.29.1 # via
certifi==2024.7.4 # swh-core
cffi==1.17.0 # swh-storage
charset-normalizer==3.3.2 blinker==1.9.0
click==8.1.7 # via
confluent-kafka==2.5.0 # flask
cryptography==43.0.0 # swh-core
Deprecated==1.2.14 boto3==1.37.24
elasticsearch==7.17.9 # via swh-graph
exceptiongroup==1.2.2 botocore==1.37.24
Flask==3.0.3 # via
frozendict==2.4.4 # boto3
frozenlist==1.4.1 # s3transfer
cachetools==5.5.2
# via pyld
cassandra-driver==3.29.2
# via swh-storage
certifi==2025.1.31
# via
# elasticsearch
# requests
# sentry-sdk
cffi==1.17.1
# via
# cryptography
# swh-perfecthash
charset-normalizer==3.4.1
# via requests
click==8.1.8
# via
# flask
# geomet
# swh-alter
# swh-core
# swh-graph
# swh-indexer
# swh-objstorage
# swh-search
# swh-storage
confluent-kafka==2.9.0
# via swh-journal
cryptography==44.0.2
# via azure-storage-blob
deprecated==1.2.18
# via
# swh-core
# swh-model
# swh-objstorage
# swh-storage
elasticsearch==7.17.12
# via swh-search
flask==3.1.0
# via
# swh-core
# swh-storage
frozendict==2.4.6
# via
# pyld
# swh-indexer
frozenlist==1.5.0
# via
# aiohttp
# aiosignal
geomet==0.2.1.post1 geomet==0.2.1.post1
grpcio==1.65.4 # via cassandra-driver
grpcio-tools==1.62.3 grpcio==1.71.0
# via grpcio-tools
grpcio-tools==1.71.0
# via swh-graph
gunicorn==23.0.0 gunicorn==23.0.0
humanize==4.10.0 # via aiohttp-utils
hypothesis==6.111.0 humanize==4.12.2
idna==3.7 # via swh-alter
igraph==0.11.6 hypothesis==6.130.6
# via swh-model
idna==3.10
# via
# requests
# yarl
igraph==0.11.8
# via swh-alter
iso8601==2.1.0 iso8601==2.1.0
isodate==0.6.1 # via
# swh-core
# swh-indexer
# swh-model
# swh-search
# swh-storage
isodate==0.7.2
# via azure-storage-blob
itsdangerous==2.2.0 itsdangerous==2.2.0
Jinja2==3.1.4 # via flask
jinja2==3.1.6
# via flask
jmespath==1.0.1 jmespath==1.0.1
# via
# boto3
# botocore
latexcodec==3.0.0 latexcodec==3.0.0
lxml==5.3.0 # via pybtex
MarkupSafe==2.1.5 lxml==5.3.1
msgpack==1.0.8 # via pyld
multidict==6.0.5 markupsafe==3.0.2
# via
# jinja2
# werkzeug
msgpack==1.1.0
# via
# swh-core
# swh-journal
# swh-objstorage
multidict==6.3.0
# via
# aiohttp
# yarl
mypy-extensions==1.0.0 mypy-extensions==1.0.0
mypy-protobuf==3.2.0 # via swh-storage
packaging==24.1 mypy-protobuf==3.6.0
plyvel==1.5.1 # via swh-graph
protobuf==4.25.4 packaging==24.2
psutil==6.0.0 # via gunicorn
psycopg2==2.9.9 propcache==0.3.1
py4j==0.10.9.7 # via
# aiohttp
# yarl
protobuf==5.29.4
# via
# grpcio-tools
# mypy-protobuf
# swh-graph
psutil==7.0.0
# via swh-graph
psycopg==3.2.6
# via
# swh-core
# swh-storage
psycopg-pool==3.2.6
# via
# swh-core
# swh-storage
py4j==0.10.9.9
# via swh-graph
pybtex==0.24.0 pybtex==0.24.0
# via swh-indexer
pycparser==2.22 pycparser==2.22
PyLD==2.0.4 # via cffi
pyorc==0.9.0 pyld==2.0.4
pyparsing==3.1.2 # via swh-indexer
pyparsing==3.2.3
# via rdflib
python-dateutil==2.9.0.post0 python-dateutil==2.9.0.post0
# via
# botocore
# swh-model
python-magic==0.4.27 python-magic==0.4.27
python-mimeparse==1.6.0 # via
PyYAML==6.0.2 # swh-core
rdflib==7.0.0 # swh-indexer
redis==5.0.8 python-mimeparse==2.0.0
# via aiohttp-utils
pyyaml==6.0.2
# via
# pybtex
# swh-alter
# swh-core
rdflib==7.1.4
# via swh-indexer
redis==5.2.1
# via swh-storage
requests==2.32.3 requests==2.32.3
s3transfer==0.10.2 # via
sentry-sdk==2.12.0 # apache-libcloud
# azure-core
# swh-core
# swh-objstorage
s3transfer==0.11.4
# via boto3
sentry-sdk==2.25.0
# via
# swh-core
# swh-indexer
setuptools==78.1.0
# via grpcio-tools
shamir-mnemonic==0.3.0 shamir-mnemonic==0.3.0
six==1.16.0 # via swh-alter
six==1.17.0
# via
# azure-core
# geomet
# pybtex
# python-dateutil
sortedcontainers==2.4.0 sortedcontainers==2.4.0
swh.alter==1.2.0 # via hypothesis
swh.core==3.4.0 swh-alter==1.4.0
swh.counters==0.11.0 # via -r /src/apps/swh-alter/requirements.txt
swh.dataset==1.6.0 swh-core==4.1.0
swh.graph==5.1.0 # via
swh.indexer==3.4.0 # swh-alter
swh.journal==1.5.2 # swh-graph
swh.model==6.14.0 # swh-indexer
swh.objstorage==3.3.0 # swh-journal
swh.perfecthash==1.3.2 # swh-objstorage
swh.search==0.21.0 # swh-search
swh.storage==2.7.0 # swh-storage
swh-graph==6.7.1
# via swh-alter
swh-indexer==4.0.0
# via swh-search
swh-journal==2.0.0
# via
# swh-indexer
# swh-search
swh-model==7.1.0
# via
# swh-alter
# swh-graph
# swh-indexer
# swh-journal
# swh-objstorage
# swh-search
# swh-storage
swh-objstorage==4.0.0
# via
# -r /src/apps/swh-alter/requirements.txt
# swh-alter
# swh-indexer
# swh-storage
swh-perfecthash==1.3.2
# via swh-objstorage
swh-search==0.22.1
# via swh-alter
swh-storage==3.1.0
# via
# swh-alter
# swh-indexer
tabulate==0.9.0 tabulate==0.9.0
# via swh-alter
tenacity==9.0.0 tenacity==9.0.0
# via
# swh-core
# swh-storage
texttable==1.7.0 texttable==1.7.0
tqdm==4.66.5 # via igraph
tree-sitter==0.21.3 tree-sitter==0.21.3
types-protobuf==5.27.0.20240626 # via swh-search
types-requests==2.31.0.6 types-protobuf==5.29.1.20250315
types-urllib3==1.26.25.14 # via mypy-protobuf
typing_extensions==4.12.2 typing-extensions==4.13.0
urllib3==1.26.19 # via
Werkzeug==3.0.3 # azure-core
wrapt==1.16.0 # azure-storage-blob
xmltodict==0.13.0 # psycopg
yarl==1.9.4 # psycopg-pool
# swh-alter
# swh-core
# swh-indexer
# swh-model
# swh-search
# swh-storage
urllib3==1.26.20
# via
# botocore
# elasticsearch
# requests
# sentry-sdk
werkzeug==3.1.3
# via flask
wrapt==1.17.2
# via deprecated
xmltodict==0.14.2
# via swh-indexer
yarl==1.18.3
# via aiohttp
apps/swh-base/Dockerfile 0 → 100644
View file @ c65886ce
FROM python:3.11-bookworm
ARG userid=1000
ARG groupid=1000
ARG user=swh
ARG workdir=/opt/${user}
ARG venv=${workdir}/venv
ARG configdir=/etc/${user}
RUN apt-get update && \
apt-get upgrade -y && \
apt-get install -y git sed \
gzip lzip zstd cpio \
less && \
apt-get clean && \
addgroup --gid ${userid} ${user} && \
useradd --gid ${groupid} --uid ${userid} -m -d ${workdir} ${user} && \
mkdir ${configdir}
USER ${user}
RUN python -m venv ${venv}
WORKDIR ${workdir}
ENV PYTHONPATH=${venv}
ENV PATH=${venv}/bin:$PATH
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
apps/swh-counters/Dockerfile
View file @ c65886ce
# Deeply inspired from the Dockerfile of the swh-storage project ARG REGISTRY=container-registry.softwareheritage.org/swh/infra/swh-apps/
FROM python:3.10-bullseye ARG base_image=${REGISTRY}base
ARG base_image_version=latest
RUN apt-get -y update && \ FROM ${base_image}:${base_image_version}
apt-get -y upgrade && \
apt-get install -y libcmph-dev librdkafka-dev && \
apt clean && \
addgroup --gid 1000 swh && \
useradd --gid 1000 --uid 1000 -m -d /opt/swh swh && \
mkdir /etc/swh
USER swh
WORKDIR /opt/swh
COPY --chown=swh:swh requirements-frozen.txt /opt/swh ARG user=swh
ARG workdir=/opt/${user}
ARG configdir=/etc/${user}
ENV PYTHONPATH=/opt/swh USER root
ENV PATH=/opt/swh/.local/bin:$PATH # retry command is used by the refresh-history cron job
# to ensure the rpc service is well up
RUN apt-get -y update && \
apt-get -y upgrade && \
apt-get install retry && \
apt-get clean
RUN /usr/local/bin/python -m pip install --upgrade pip && \ COPY --chmod=0644 requirements-frozen.txt ${workdir}
pip install --no-cache-dir -r requirements-frozen.txt && \ RUN --mount=type=cache,target=.cache,uid=1000,gid=1000 \
pip install gunicorn uv pip sync requirements-frozen.txt
COPY --chown=swh:swh entrypoint.sh /opt/swh COPY --chmod=0755 entrypoint.sh ${workdir}
RUN chmod u+x /opt/swh/entrypoint.sh
ENV SWH_CONFIG_FILENAME=/etc/swh/config.yml USER ${user}
ENV PORT 5011 ENV SWH_CONFIG_FILENAME=${configdir}/config.yml
ENV PORT=5011
EXPOSE $PORT EXPOSE $PORT
ENV THREADS 2 ENV THREADS=2
ENV WORKERS 2 ENV WORKERS=2
ENV TIMEOUT 3600 ENV TIMEOUT=3600
ENTRYPOINT ["/opt/swh/entrypoint.sh"] ENTRYPOINT ["/opt/swh/entrypoint.sh"]
apps/swh-counters/entrypoint.sh
View file @ c65886ce
...@@ -17,19 +17,24 @@ case "$1" in ...@@ -17,19 +17,24 @@ case "$1" in
exec swh $@ exec swh $@
;; ;;
*) *)
EXTRA_CLI_FLAGS="" EXTRA_CLI_FLAGS=()
if [ ! -z "${SWH_LOG_CONFIG_JSON}" ]; then if [ -n "${SWH_LOG_CONFIG_JSON}" ]; then
EXTRA_CLI_FLAGS="--log-config-json ${SWH_LOG_CONFIG_JSON}" EXTRA_CLI_FLAGS+=('--log-config-json' "${SWH_LOG_CONFIG_JSON}")
fi fi
echo Starting the swh-counters API server if [ -n "${STATSD_HOST}" -a -n "${STATSD_PORT}" ]; then
exec gunicorn --bind 0.0.0.0:${PORT} \ EXTRA_CLI_FLAGS+=('--statsd-host' "${STATSD_HOST}:${STATSD_PORT}")
--log-level ${SWH_LOG_LEVEL:-INFO} \ fi
${EXTRA_CLI_FLAGS} \ if [ -n "${STATSD_SERVICE_TYPE}" ]; then
--threads ${THREADS} \ EXTRA_CLI_FLAGS+=('--statsd-prefix' "${STATSD_SERVICE_TYPE}")
--workers ${WORKERS} \ fi
--timeout ${TIMEOUT} \
--statsd-host=${STATSD_HOST}:${STATSD_PORT} \ echo 'Starting the swh-counters API server'
--statsd-prefix=${STATSD_SERVICE_TYPE} \ exec gunicorn --bind "0.0.0.0:${PORT}" \
--log-level "${SWH_LOG_LEVEL:-INFO}" \
"${EXTRA_CLI_FLAGS[@]}" \
--threads "${THREADS}" \
--workers "${WORKERS}" \
--timeout "${TIMEOUT}" \
--reload \ --reload \
--config 'python:swh.core.api.gunicorn_config' \ --config 'python:swh.core.api.gunicorn_config' \
'swh.counters.api.server:make_app_from_configfile()' 'swh.counters.api.server:make_app_from_configfile()'
......
apps/swh-counters/requirements-frozen.txt
View file @ c65886ce
aiohappyeyeballs==2.3.5 # This file was autogenerated by uv via the following command:
aiohttp==3.10.3 # uv pip compile /src/apps/swh-counters/requirements.txt -o /src/apps/swh-counters/tmpiv8trarv
aiohappyeyeballs==2.6.1
# via aiohttp
aiohttp==3.11.14
# via
# aiohttp-utils
# swh-core
aiohttp-utils==3.2.1 aiohttp-utils==3.2.1
aiosignal==1.3.1 # via swh-core
async-timeout==4.0.3 aiosignal==1.3.2
attrs==24.2.0 # via aiohttp
attrs==25.3.0
# via
# aiohttp
# attrs-strict
# hypothesis
# swh-model
attrs-strict==1.0.1 attrs-strict==1.0.1
blinker==1.8.2 # via swh-model
certifi==2024.7.4 backports-entry-points-selectable==1.3.0
charset-normalizer==3.3.2 # via swh-core
click==8.1.7 blinker==1.9.0
confluent-kafka==2.5.0 # via
Deprecated==1.2.14 # flask
exceptiongroup==1.2.2 # swh-core
Flask==3.0.3 certifi==2025.1.31
frozenlist==1.4.1 # via
# requests
# sentry-sdk
charset-normalizer==3.4.1
# via requests
click==8.1.8
# via
# flask
# swh-core
confluent-kafka==2.8.2
# via swh-journal
deprecated==1.2.18
# via
# swh-core
# swh-model
flask==3.1.0
# via
# swh-core
# swh-counters
frozenlist==1.5.0
# via
# aiohttp
# aiosignal
gunicorn==23.0.0 gunicorn==23.0.0
hypothesis==6.111.0 # via
idna==3.7 # -r /src/apps/swh-counters/requirements.txt
# aiohttp-utils
hypothesis==6.130.4
# via swh-model
idna==3.10
# via
# requests
# yarl
iso8601==2.1.0 iso8601==2.1.0
# via
# swh-core
# swh-model
itsdangerous==2.2.0 itsdangerous==2.2.0
Jinja2==3.1.4 # via flask
MarkupSafe==2.1.5 jinja2==3.1.6
msgpack==1.0.8 # via flask
multidict==6.0.5 markupsafe==3.0.2
packaging==24.1 # via
# jinja2
# werkzeug
msgpack==1.1.0
# via
# swh-core
# swh-journal
multidict==6.2.0
# via
# aiohttp
# yarl
packaging==24.2
# via gunicorn
propcache==0.3.1
# via
# aiohttp
# yarl
python-dateutil==2.9.0.post0 python-dateutil==2.9.0.post0
python-json-logger==2.0.7 # via swh-model
python-json-logger==3.3.0
# via -r /src/apps/swh-counters/requirements.txt
python-magic==0.4.27 python-magic==0.4.27
python-mimeparse==1.6.0 # via swh-core
PyYAML==6.0.2 python-mimeparse==2.0.0
redis==5.0.8 # via aiohttp-utils
pyyaml==6.0.2
# via swh-core
redis==5.2.1
# via swh-counters
requests==2.32.3 requests==2.32.3
sentry-sdk==2.13.0 # via swh-core
six==1.16.0 sentry-sdk==2.24.1
# via swh-core
six==1.17.0
# via python-dateutil
sortedcontainers==2.4.0 sortedcontainers==2.4.0
swh.core==3.4.0 # via hypothesis
swh.counters==0.11.0 swh-core==4.1.0
swh.journal==1.5.2 # via
swh.model==6.14.0 # swh-counters
# swh-journal
swh-counters==0.11.0
# via -r /src/apps/swh-counters/requirements.txt
swh-journal==1.5.3
# via swh-counters
swh-model==7.1.0
# via swh-journal
tenacity==9.0.0 tenacity==9.0.0
typing_extensions==4.12.2 # via
urllib3==2.2.2 # swh-core
Werkzeug==3.0.3 # swh-journal
wrapt==1.16.0 typing-extensions==4.13.0
yarl==1.9.4 # via swh-model
urllib3==2.3.0
# via
# requests
# sentry-sdk
werkzeug==3.1.3
# via flask
wrapt==1.17.2
# via deprecated
yarl==1.18.3
# via aiohttp
apps/swh-counters/requirements.txt
View file @ c65886ce
swh.counters swh.counters
python-json-logger python-json-logger
gunicorn
apps/swh-deposit-checkers/Dockerfile
View file @ c65886ce
# Deeply inspired from the Dockerfile of the swh-graph project ARG REGISTRY=container-registry.softwareheritage.org/swh/infra/swh-apps/
FROM python:3.10-bullseye ARG base_image=${REGISTRY}base
ARG base_image_version=latest
RUN apt-get -y update && \ FROM ${base_image}:${base_image_version}
apt-get -y upgrade && \
apt-get install -y libcmph-dev && \
apt clean && \
addgroup --gid 1000 swh && \
useradd --gid 1000 --uid 1000 -m -d /opt/swh swh && \
mkdir /etc/swh
USER swh ARG user=swh
WORKDIR /opt/swh ARG workdir=/opt/${user}
ARG configdir=/etc/${user}
COPY --chown=swh:swh requirements-frozen.txt /opt/swh COPY --chmod=0644 requirements-frozen.txt ${workdir}
COPY --chown=swh:swh entrypoint.sh /opt/swh RUN --mount=type=cache,target=.cache,uid=1000,gid=1000 \
uv pip sync requirements-frozen.txt
ENV PYTHONPATH=/opt/swh COPY --chmod=0755 entrypoint.sh ${workdir}
ENV PATH=/opt/swh/.local/bin:$PATH
RUN chmod u+x /opt/swh/entrypoint.sh && \ USER ${user}
/usr/local/bin/python -m pip install --upgrade pip && \ ENV SWH_CONFIG_FILENAME=${configdir}/config.yml
pip install --no-cache-dir -r requirements-frozen.txt && \
pip install gunicorn
ENV SWH_CONFIG_FILENAME=/etc/swh/config.yml
ENV SWH_WORKER_INSTANCE=checker ENV SWH_WORKER_INSTANCE=checker
ENV CONCURRENCY=1 ENV CONCURRENCY=1
ENV MAX_TASKS_PER_CHILD=5 ENV MAX_TASKS_PER_CHILD=5
ENTRYPOINT "/opt/swh/entrypoint.sh" ENTRYPOINT ["/opt/swh/entrypoint.sh"]
apps/swh-deposit-checkers/entrypoint.sh
View file @ c65886ce
...@@ -3,22 +3,25 @@ ...@@ -3,22 +3,25 @@
set -e set -e
case "$1" in case "$1" in
"shell") "shell")
shift shift
echo "Running command $@" if (( $# == 0)); then
exec bash -i "$@" exec bash -i
;; else
*) "$@"
echo Starting the swh Celery worker for ${SWH_WORKER_INSTANCE} fi
exec python -m celery \ ;;
--app=swh.scheduler.celery_backend.config.app \ *)
worker \ echo Starting the swh Celery worker for ${SWH_WORKER_INSTANCE}
--pool=prefork \ exec python -m celery \
--concurrency=${CONCURRENCY} \ --app=swh.scheduler.celery_backend.config.app \
--max-tasks-per-child=${MAX_TASKS_PER_CHILD} \ worker \
-Ofair \ --pool=prefork \
--loglevel=${SWH_LOG_LEVEL:-INFO} \ --concurrency=${CONCURRENCY} \
--without-gossip --without-mingle --without-heartbeat \ --max-tasks-per-child=${MAX_TASKS_PER_CHILD} \
--hostname "${SWH_WORKER_INSTANCE}@%h" -Ofair \
;; --loglevel=${SWH_LOG_LEVEL:-INFO} \
--without-gossip --without-mingle --without-heartbeat \
--hostname "${SWH_WORKER_INSTANCE}@%h"
;;
esac esac
apps/swh-deposit-checkers/requirements-frozen.txt
View file @ c65886ce
aiohappyeyeballs==2.3.5 # This file was autogenerated by uv via the following command:
aiohttp==3.10.3 # uv pip compile /src/apps/swh-deposit-checkers/requirements.txt -o /src/apps/swh-deposit-checkers/tmpsb1ah3tf
aiohappyeyeballs==2.6.1
# via aiohttp
aiohttp==3.11.15
# via
# aiohttp-utils
# swh-core
aiohttp-utils==3.2.1 aiohttp-utils==3.2.1
aiosignal==1.3.1 # via swh-core
amqp==5.2.0 aiosignal==1.3.2
asn1crypto==1.5.1 # via aiohttp
async-timeout==4.0.3 amqp==5.3.1
attrs==24.2.0 # via kombu
attrs==25.3.0
# via
# aiohttp
# attrs-strict
# hypothesis
# swh-model
# swh-scheduler
attrs-strict==1.0.1 attrs-strict==1.0.1
billiard==4.2.0 # via
blinker==1.8.2 # swh-model
cassandra-driver==3.29.1 # swh-scheduler
celery==5.4.0 backports-entry-points-selectable==1.3.0
certifi==2024.7.4 # via
cffi==1.17.0 # swh-core
charset-normalizer==3.3.2 # swh-storage
click==8.1.7 billiard==4.2.1
# via celery
blinker==1.9.0
# via
# flask
# swh-core
cassandra-driver==3.29.2
# via swh-storage
celery==5.5.0
# via swh-scheduler
certifi==2025.1.31
# via
# requests
# sentry-sdk
cffi==1.17.1
# via swh-perfecthash
charset-normalizer==3.4.1
# via requests
click==8.1.8
# via
# celery
# click-didyoumean
# click-plugins
# click-repl
# flask
# geomet
# swh-core
# swh-deposit
# swh-objstorage
# swh-scheduler
# swh-storage
click-didyoumean==0.3.1 click-didyoumean==0.3.1
# via celery
click-plugins==1.1.1 click-plugins==1.1.1
# via celery
click-repl==0.3.0 click-repl==0.3.0
confluent-kafka==2.5.0 # via celery
Deprecated==1.2.14 deprecated==1.2.18
exceptiongroup==1.2.2 # via
Flask==3.0.3 # swh-core
frozenlist==1.4.1 # swh-model
# swh-objstorage
# swh-storage
elementpath==4.8.0
# via xmlschema
flask==3.1.0
# via
# swh-core
# swh-scheduler
# swh-storage
frozenlist==1.5.0
# via
# aiohttp
# aiosignal
geomet==0.2.1.post1 geomet==0.2.1.post1
# via cassandra-driver
gunicorn==23.0.0 gunicorn==23.0.0
humanize==4.10.0 # via aiohttp-utils
hypothesis==6.111.0 humanize==4.12.2
idna==3.7 # via swh-scheduler
importlib_metadata==8.2.0 hypothesis==6.130.6
# via swh-model
idna==3.10
# via
# requests
# yarl
importlib-metadata==8.6.1
# via swh-scheduler
iso8601==2.1.0 iso8601==2.1.0
# via
# swh-core
# swh-deposit
# swh-model
# swh-storage
itsdangerous==2.2.0 itsdangerous==2.2.0
Jinja2==3.1.4 # via flask
kombu==5.4.0 jinja2==3.1.6
MarkupSafe==2.1.5 # via flask
msgpack==1.0.8 kombu==5.5.2
multidict==6.0.5 # via celery
markupsafe==3.0.2
# via
# jinja2
# werkzeug
msgpack==1.1.0
# via
# swh-core
# swh-objstorage
multidict==6.3.0
# via
# aiohttp
# yarl
mypy-extensions==1.0.0 mypy-extensions==1.0.0
packaging==24.1 # via swh-storage
pg8000==1.31.2 packaging==24.2
# via gunicorn
pika==1.3.2 pika==1.3.2
prompt_toolkit==3.0.47 # via swh-scheduler
psycopg2==2.9.9 prompt-toolkit==3.0.50
# via click-repl
propcache==0.3.1
# via
# aiohttp
# yarl
psycopg==3.2.6
# via
# swh-core
# swh-scheduler
# swh-storage
psycopg-pool==3.2.6
# via
# swh-core
# swh-scheduler
# swh-storage
pycparser==2.22 pycparser==2.22
# via cffi
python-dateutil==2.9.0.post0 python-dateutil==2.9.0.post0
python-json-logger==2.0.7 # via
# celery
# swh-model
python-json-logger==3.3.0
# via -r /src/apps/swh-deposit-checkers/requirements.txt
python-magic==0.4.27 python-magic==0.4.27
python-mimeparse==1.6.0 # via swh-core
PyYAML==6.0.2 python-mimeparse==2.0.0
redis==5.0.8 # via aiohttp-utils
pyyaml==6.0.2
# via
# swh-core
# swh-scheduler
redis==5.2.1
# via swh-storage
requests==2.32.3 requests==2.32.3
scramp==1.4.5 # via
sentry-sdk==2.12.0 # swh-core
six==1.16.0 # swh-deposit
# swh-objstorage
# swh-scheduler
sentry-sdk==2.25.0
# via
# swh-core
# swh-deposit
# swh-scheduler
setuptools==78.1.0
# via swh-scheduler
six==1.17.0
# via
# geomet
# python-dateutil
sortedcontainers==2.4.0 sortedcontainers==2.4.0
swh.core==3.4.0 # via hypothesis
swh.counters==0.11.0 swh-core==4.1.0
swh.deposit==2.2.2 # via
swh.journal==1.5.2 # swh-deposit
swh.model==6.14.0 # swh-objstorage
swh.objstorage==3.3.0 # swh-scheduler
swh.perfecthash==1.3.2 # swh-storage
swh.scheduler==2.4.0 swh-deposit==3.0.0
swh.storage==2.7.0 # via -r /src/apps/swh-deposit-checkers/requirements.txt
swh-model==7.1.0
# via
# swh-deposit
# swh-objstorage
# swh-storage
swh-objstorage==4.0.0
# via swh-storage
swh-perfecthash==1.3.2
# via swh-objstorage
swh-scheduler==3.1.0
# via -r /src/apps/swh-deposit-checkers/requirements.txt
swh-storage==3.1.0
# via swh-scheduler
tabulate==0.9.0 tabulate==0.9.0
# via swh-scheduler
tenacity==9.0.0 tenacity==9.0.0
testing.common.database==2.0.3 # via
testing.postgresql==1.3.0 # swh-core
typing_extensions==4.12.2 # swh-storage
tzdata==2024.1 typing-extensions==4.13.0
urllib3==2.2.2 # via
# psycopg
# psycopg-pool
# swh-core
# swh-model
# swh-scheduler
# swh-storage
tzdata==2025.2
# via kombu
urllib3==2.3.0
# via
# requests
# sentry-sdk
vine==5.1.0 vine==5.1.0
# via
# amqp
# celery
# kombu
wcwidth==0.2.13 wcwidth==0.2.13
Werkzeug==3.0.3 # via prompt-toolkit
wrapt==1.16.0 werkzeug==3.1.3
yarl==1.9.4 # via flask
zipp==3.20.0 wrapt==1.17.2
# via deprecated
xmlschema==3.4.5
# via swh-deposit
yarl==1.18.3
# via aiohttp
zipp==3.21.0
# via importlib-metadata
apps/swh-deposit/Dockerfile
View file @ c65886ce
# Deeply inspired from the Dockerfile of the swh-storage project ARG REGISTRY=container-registry.softwareheritage.org/swh/infra/swh-apps/
FROM python:3.10-bullseye ARG base_image=${REGISTRY}base
ARG base_image_version=latest
RUN apt-get -y update && \ FROM ${base_image}:${base_image_version}
apt-get -y upgrade && \
apt-get install -y libcmph-dev librdkafka-dev && \
apt clean && \
addgroup --gid 1000 swh && \
useradd --gid 1000 --uid 1000 -m -d /opt/swh swh && \
mkdir /etc/swh
USER swh ARG user=swh
WORKDIR /opt/swh ARG workdir=/opt/${user}
ARG configdir=/etc/${user}
COPY --chown=swh:swh requirements-frozen.txt /opt/swh COPY --chmod=0644 requirements-frozen.txt ${workdir}
RUN --mount=type=cache,target=.cache,uid=1000,gid=1000 \
uv pip sync requirements-frozen.txt
ENV PYTHONPATH=/opt/swh COPY --chmod=0755 entrypoint.sh ${workdir}
ENV PATH=/opt/swh/.local/bin:$PATH
RUN /usr/local/bin/python -m pip install --upgrade pip && \ USER ${user}
pip install --no-cache-dir -r requirements-frozen.txt && \ ENV SWH_CONFIG_FILENAME=${configdir}/config.yml
pip install gunicorn ENV PORT=5006
COPY --chown=swh:swh entrypoint.sh /opt/swh
RUN chmod u+x /opt/swh/entrypoint.sh
ENV SWH_CONFIG_FILENAME=/etc/swh/config.yml
ENV PORT 5006
EXPOSE $PORT EXPOSE $PORT
ENV THREADS 2 ENV THREADS=2
ENV WORKERS 2 ENV WORKERS=2
ENV TIMEOUT 3600 ENV TIMEOUT=3600
ENV DJANGO_SETTINGS_MODULE swh.deposit.settings.production ENV DJANGO_SETTINGS_MODULE=swh.deposit.settings.production
ENV SWH_MAIN_PACKAGE swh.deposit ENV SWH_MAIN_PACKAGE=swh.deposit
ENTRYPOINT ["/opt/swh/entrypoint.sh"] ENTRYPOINT ["/opt/swh/entrypoint.sh"]