Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • ardumont/swh-apps
  • vlorentz/swh-apps
  • swh/infra/swh-apps
  • anlambert/swh-apps
4 results
Show changes
Commits on Source (1282)
1,182 additional commits have been omitted to prevent performance issues.
Hide whitespace changes
Inline Side-by-side
Showing
with 1453 additions and 450 deletions
README.rst
View file @ a8b3a6d3
......@@ -38,3 +38,184 @@ file. Out of this, a virtualenv is generated and frozen into a
The frozen requirements file can also be used to deploy virtual environments
directly, e.g. on an existing VM or bare metal server.
Local development
-----------------
App-manager
~~~~~~~~~~~
It's the cli tools which allows to execute various actions:
- list dependency between swh modules
- manipulate the generation of frozen requirements for our python applications
- helm chart dependency version update
- etc...
.. code::
# Build the app-manager image
docker build -t app-manager scripts
List dependent applications
~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is the listing command to list whatever other applications is depending
on a specification application version.
.. code::
docker run --rm -v $workspace/swh-apps:/src \
app-manager list-dependent-apps \
--application swh.graph --version v6.3.0
Example:
.. code::
docker run --rm -v $workspace/swh-apps:/src \
app-manager list-dependent-apps \
--application swh.graph --version v6.3.0
swh-alter
swh-graph
swh-provenance
swh-vault-cookers
swh-web
Freeze application dependencies
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is in charge of using the existing requirements.txt declared besides the
application's Dockerfile and generate an updated frozen-requirements.txt. This
file is then used within the docker image to build an identical python
environments within the image.
Note that it does update any dependencies within the environments so any new
upstream libraries will get updated if new releases happened.
.. code::
docker run --rm -v $workspace/swh-apps:/src \
app-manager generate-frozen-requirements swh-provenance
.. code::
$ docker run --rm -v $workspace/swh-apps:/src \
app-manager generate-frozen-requirements swh-provenance
Collecting uv
Downloading uv-0.5.26-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.2/16.2 MB 36.4 MB/s eta 0:00:00
Installing collected packages: uv
Successfully installed uv-0.5.26
[notice] A new release of pip is available: 23.0.1 -> 25.0
[notice] To update, run: python3 -m pip install --upgrade pip
Using Python 3.10.15 environment at: /tmp/swh-provenancep16_3kh7
Resolved 3 packages in 46ms
Prepared 3 packages in 86ms
Uninstalled 2 packages in 51ms
Installed 3 packages in 32ms
- pip==23.0.1
+ pip==25.0
- setuptools==65.5.0
+ setuptools==75.8.0
+ wheel==0.45.1
# This file was autogenerated by uv via the following command:
# uv pip compile /src/apps/swh-provenance/requirements.txt -o /src/apps/swh-provenance/tmpk6i4m4rz
Resolved 76 packages in 850ms
aiohappyeyeballs==2.4.4
# via aiohttp
# via -r /src/apps/swh-provenance/requirements.txt
python-magic==0.4.27
# via swh-core
python-mimeparse==2.0.0
# via aiohttp-utils
pyyaml==6.0.2
# via swh-core
redis==5.2.1
# via
# via -r /src/apps/swh-provenance/requirements.txt
...
swh-storage==2.9.0
# via
# swh-dataset
# swh-provenance
tenacity==9.0.0
# via
# swh-core
# swh-journal
# swh-storage
tqdm==4.67.1
# via swh-dataset
types-protobuf==5.29.1.20241207
# via mypy-protobuf
types-requests==2.32.0.20241016
# via swh-dataset
typing-extensions==4.12.2
# via
# multidict
# swh-core
# swh-model
# swh-objstorage
# swh-storage
urllib3==2.3.0
# via
# botocore
# requests
# sentry-sdk
# types-requests
werkzeug==3.1.3
# via flask
wrapt==1.17.2
# via deprecated
yarl==1.18.3
# via aiohttp
Update swh-charts' version
~~~~~~~~~~~~~~~~~~~~~~~~~~
Providing access to both the swh-apps and the swh-charts repositories, this
allows to update the values of values-swh-application-versions.yaml with the
most recent docker image versions. This also bumps the Charts.yaml's version
incrementally.
.. code::
docker run --rm \
--volume $workspace/swh-apps:/src \
--volume $workspace/swh-charts:/tmp/swh-charts \
app-manager update-versions \
--applications-filepath /tmp/s-charts/values-swh-application-versions.yaml \
--chart-filepath /tmp/swh-charts/swh/Chart.yaml
Build application image
~~~~~~~~~~~~~~~~~~~~~~~
Each application is stored in swh-apps:/apps/$application_dir/ folder.
It usually holds the same set of files:
- Dockerfile: The set of instructions to build the application's docker image
- entrypoint.sh: The last Dockerfile instruction refers to this executable to run in the
image.
- requirements.txt: The set of python dependencies the application requires to run.
- requirements-frozen.txt: The derivative frozen set of dependencies to reproduce the
environment in the docker image. It's built by the app-manager out of the
requirements.txt file.
Cli call to build the application locally:
.. code::
$ DOCKER_BUILDKIT=1 docker build -t swh-toolbox:latest apps/swh-toolbox \
--build-arg REGISTRY=
Note: The REGISTRY is empty so we only rely to local docker image. You can avoid
changing it, in which case, it will use the swh's gitlab registry.
Cli call to run the application and be dropped in a shell:
.. code::
$ docker run -it swh-toolbox:latest shell
Note: This is specific command depending on the entrypoint.sh. In that case, the
entrypoint.sh allows a shell option to be dropped in an interactive bash session.
apps/cassandra-checks/Dockerfile
View file @ a8b3a6d3
FROM python:3.10-bullseye
ARG REGISTRY=container-registry.softwareheritage.org/swh/infra/swh-apps/
ARG base_image=${REGISTRY}base
ARG base_image_version=latest
RUN apt-get -y update && \
apt-get -y upgrade && \
apt-get install -y libpq-dev git && \
apt clean && \
addgroup --gid 1000 swh && \
useradd --gid 1000 --uid 1000 -m -d /opt/swh swh && \
mkdir /etc/swh
FROM ${base_image}:${base_image_version}
RUN mkdir /volume && chown root: /volume
ARG user=swh
ARG workdir=/opt/${user}
ARG configdir=/etc/${user}
USER swh
WORKDIR /opt/swh
USER root
RUN mkdir /volume && chown root: /volume
COPY --chown=swh:swh requirements-frozen.txt /opt/swh
COPY --chown=swh:swh entrypoint.sh /opt/swh
COPY --chmod=0644 requirements-frozen.txt ${workdir}
RUN --mount=type=cache,target=.cache,uid=1000,gid=1000,uid=1000,gid=1000 \
uv pip sync requirements-frozen.txt
ENV PYTHONPATH=/opt/swh
ENV PATH=/opt/swh/.local/bin:$PATH
COPY --chmod=0755 entrypoint.sh ${workdir}
RUN chmod u+x /opt/swh/entrypoint.sh && \
/usr/local/bin/python -m pip install --upgrade pip && \
pip install --no-cache-dir -r requirements-frozen.txt
USER ${user}
ENV SWH_CONFIG_FILENAME=/etc/swh/config.yml
ENV SWH_CONFIG_FILENAME=${configdir}/config.yml
ENTRYPOINT ["/opt/swh/entrypoint.sh"]
apps/cassandra-checks/entrypoint.sh
View file @ a8b3a6d3
......@@ -14,13 +14,16 @@ else
fi
case "$1" in
"shell")
shift
echo "Running command $@"
exec bash -i "$@"
;;
*)
echo Starting Cassandra objects checker.
exec $CLONE_DIR/sysadmin/cassandra_checks/get_journal_check_and_replay.py
;;
"shell")
shift
if (( $# == 0)); then
exec bash -i
else
"$@"
fi
;;
*)
echo Starting Cassandra objects checker.
exec $CLONE_DIR/sysadmin/cassandra_checks/get_journal_check_and_replay.py
;;
esac
apps/cassandra-checks/requirements-frozen.txt
View file @ a8b3a6d3
aiohappyeyeballs==2.3.5
aiohttp==3.10.3
# This file was autogenerated by uv via the following command:
# uv pip compile /src/apps/cassandra-checks/requirements.txt -o /src/apps/cassandra-checks/tmpwfuj2y1l
aiohappyeyeballs==2.6.1
# via aiohttp
aiohttp==3.11.14
# via
# aiohttp-utils
# swh-core
aiohttp-utils==3.2.1
aiosignal==1.3.1
async-timeout==4.0.3
attrs==24.2.0
# via swh-core
aiosignal==1.3.2
# via aiohttp
attrs==25.3.0
# via
# aiohttp
# attrs-strict
# hypothesis
# swh-model
attrs-strict==1.0.1
blinker==1.8.2
cassandra-driver==3.29.1
certifi==2024.7.4
cffi==1.17.0
charset-normalizer==3.3.2
click==8.1.7
confluent-kafka==2.5.0
Deprecated==1.2.14
exceptiongroup==1.2.2
Flask==3.0.3
frozenlist==1.4.1
# via swh-model
backports-entry-points-selectable==1.3.0
# via
# swh-core
# swh-storage
blinker==1.9.0
# via
# flask
# swh-core
cassandra-driver==3.29.2
# via swh-storage
certifi==2025.1.31
# via
# requests
# sentry-sdk
cffi==1.17.1
# via swh-perfecthash
charset-normalizer==3.4.1
# via requests
click==8.1.8
# via
# flask
# geomet
# swh-core
# swh-objstorage
# swh-storage
confluent-kafka==2.8.2
# via swh-journal
deprecated==1.2.18
# via
# swh-core
# swh-model
# swh-objstorage
# swh-storage
flask==3.1.0
# via
# swh-core
# swh-counters
# swh-storage
frozenlist==1.5.0
# via
# aiohttp
# aiosignal
geomet==0.2.1.post1
# via cassandra-driver
gunicorn==23.0.0
hypothesis==6.111.0
idna==3.7
# via aiohttp-utils
hypothesis==6.130.4
# via swh-model
idna==3.10
# via
# requests
# yarl
iso8601==2.1.0
# via
# swh-core
# swh-model
# swh-storage
itsdangerous==2.2.0
Jinja2==3.1.4
MarkupSafe==2.1.5
msgpack==1.0.8
multidict==6.0.5
# via flask
jinja2==3.1.6
# via flask
markupsafe==3.0.2
# via
# jinja2
# werkzeug
msgpack==1.1.0
# via
# swh-core
# swh-journal
# swh-objstorage
multidict==6.2.0
# via
# aiohttp
# yarl
mypy-extensions==1.0.0
packaging==24.1
psycopg2==2.9.9
# via swh-storage
packaging==24.2
# via gunicorn
propcache==0.3.1
# via
# aiohttp
# yarl
psycopg==3.2.6
# via
# swh-core
# swh-storage
psycopg-pool==3.2.6
# via
# swh-core
# swh-storage
pycparser==2.22
# via cffi
python-dateutil==2.9.0.post0
# via swh-model
python-magic==0.4.27
python-mimeparse==1.6.0
PyYAML==6.0.2
redis==5.0.8
# via swh-core
python-mimeparse==2.0.0
# via aiohttp-utils
pyyaml==6.0.2
# via swh-core
redis==5.2.1
# via
# swh-counters
# swh-storage
requests==2.32.3
sentry-sdk==2.12.0
six==1.16.0
# via
# swh-core
# swh-objstorage
sentry-sdk==2.24.1
# via swh-core
six==1.17.0
# via
# geomet
# python-dateutil
sortedcontainers==2.4.0
swh.core==3.4.0
swh.counters==0.11.0
swh.journal==1.5.2
swh.model==6.14.0
swh.objstorage==3.3.0
swh.perfecthash==1.3.2
swh.storage==2.7.0
# via hypothesis
swh-core==4.1.0
# via
# swh-counters
# swh-journal
# swh-objstorage
# swh-storage
swh-counters==0.11.0
# via swh-storage
swh-journal==1.5.3
# via swh-counters
swh-model==7.1.0
# via
# -r /src/apps/cassandra-checks/requirements.txt
# swh-journal
# swh-objstorage
# swh-storage
swh-objstorage==4.0.0
# via swh-storage
swh-perfecthash==1.3.2
# via swh-objstorage
swh-storage==3.0.0
# via -r /src/apps/cassandra-checks/requirements.txt
tenacity==9.0.0
typing_extensions==4.12.2
urllib3==2.2.2
Werkzeug==3.0.3
wrapt==1.16.0
yarl==1.9.4
# via
# swh-core
# swh-journal
# swh-storage
typing-extensions==4.13.0
# via
# psycopg
# psycopg-pool
# swh-core
# swh-model
# swh-storage
urllib3==2.3.0
# via
# requests
# sentry-sdk
werkzeug==3.1.3
# via flask
wrapt==1.17.2
# via deprecated
yarl==1.18.3
# via aiohttp
apps/ipfs-gateway/Dockerfile
View file @ a8b3a6d3
......@@ -40,7 +40,7 @@ WORKDIR /home/ipfs
USER ipfs
WORKDIR /data/ipfs
ENV IPFS_PATH /data/ipfs
ENV IPFS_PATH=/data/ipfs
# | true to ignore the empty directory error:
# https://github.com/ipfs/kubo/issues/9155
......
apps/software-stories/Dockerfile
View file @ a8b3a6d3
......@@ -10,13 +10,11 @@ USER node
ENV NODE_OPTIONS=--openssl-legacy-provider
WORKDIR /opt/swh/swh-stories
COPY --chown=node:node entrypoint.sh /opt/swh
RUN chmod u+x /opt/swh/entrypoint.sh && \
git clone https://github.com/ScienceStories/swh-stories.git /opt/swh/swh-stories && \
git checkout 7dcac1ac6edf376afc0a011465094b11c27683c5 && \
rm package-lock.json && \
RUN git clone https://github.com/ScienceStories/swh-stories.git /opt/swh/swh-stories && \
npm install --no-audit && \
npm run build
ENTRYPOINT "/opt/swh/entrypoint.sh"
COPY --chown=node:node entrypoint.sh /opt/swh
RUN chmod u+x /opt/swh/entrypoint.sh
ENTRYPOINT ["/opt/swh/entrypoint.sh"]
apps/swh-add-forge-now/Dockerfile
View file @ a8b3a6d3
FROM python:3.10-bookworm
ARG REGISTRY=container-registry.softwareheritage.org/swh/infra/swh-apps/
ARG base_image=${REGISTRY}base
ARG base_image_version=latest
RUN apt-get -y update && \
apt-get -y upgrade && \
apt-get install -y libcmph-dev netcat-openbsd jq && \
apt clean && \
addgroup --gid 1000 swh && \
useradd --gid 1000 --uid 1000 -m -d /opt/swh swh
FROM ${base_image}:${base_image_version}
USER swh
WORKDIR /opt/swh
ARG user=swh
ARG workdir=/opt/${user}
ARG configdir=/etc/${user}
COPY --chown=swh:swh requirements-frozen.txt /opt/swh
USER root
RUN apt-get -y update && \
apt-get install -y netcat-openbsd jq && \
apt-get clean
ENV PATH=/opt/swh/.local/bin:$PATH
COPY --chmod=0644 requirements-frozen.txt ${workdir}
RUN --mount=type=cache,target=.cache,uid=1000,gid=1000 \
uv pip sync requirements-frozen.txt
RUN /usr/local/bin/python -m pip install --upgrade pip && \
pip install --no-cache-dir -r requirements-frozen.txt
USER ${user}
CMD [ "/bin/bash", "-i" ]
apps/swh-add-forge-now/requirements-frozen.txt
View file @ a8b3a6d3
aiohappyeyeballs==2.3.5
aiohttp==3.10.3
# This file was autogenerated by uv via the following command:
# uv pip compile /src/apps/swh-add-forge-now/requirements.txt -o /src/apps/swh-add-forge-now/tmpee98wbzv
aiofiles==24.1.0
# via python-keycloak
aiohappyeyeballs==2.6.1
# via aiohttp
aiohttp==3.11.14
# via
# aiohttp-utils
# swh-core
aiohttp-utils==3.2.1
aiosignal==1.3.1
amqp==5.2.0
asn1crypto==1.5.1
async-timeout==4.0.3
attrs==24.2.0
# via swh-core
aiosignal==1.3.2
# via aiohttp
amqp==5.3.1
# via kombu
anyio==4.9.0
# via httpx
async-property==0.2.2
# via python-keycloak
attrs==25.3.0
# via
# aiohttp
# attrs-strict
# hypothesis
# swh-model
# swh-scheduler
attrs-strict==1.0.1
billiard==4.2.0
blinker==1.8.2
cassandra-driver==3.29.1
# via
# swh-model
# swh-scheduler
backports-entry-points-selectable==1.3.0
# via
# swh-core
# swh-storage
billiard==4.2.1
# via celery
blinker==1.9.0
# via
# flask
# swh-core
cassandra-driver==3.29.2
# via swh-storage
celery==5.4.0
certifi==2024.7.4
cffi==1.17.0
charset-normalizer==3.3.2
click==8.1.7
# via swh-scheduler
certifi==2025.1.31
# via
# httpcore
# httpx
# requests
# sentry-sdk
cffi==1.17.1
# via
# cryptography
# swh-perfecthash
charset-normalizer==3.4.1
# via requests
click==8.1.8
# via
# celery
# click-didyoumean
# click-plugins
# click-repl
# flask
# geomet
# swh-auth
# swh-core
# swh-objstorage
# swh-scheduler
# swh-storage
click-didyoumean==0.3.1
# via celery
click-plugins==1.1.1
# via celery
click-repl==0.3.0
confluent-kafka==2.5.0
Deprecated==1.2.14
# via celery
confluent-kafka==2.8.2
# via swh-journal
cryptography==44.0.2
# via jwcrypto
deprecated==1.2.18
# via
# swh-core
# swh-model
# swh-objstorage
# swh-storage
deprecation==2.1.0
ecdsa==0.19.0
exceptiongroup==1.2.2
Flask==3.0.3
frozenlist==1.4.1
# via python-keycloak
flask==3.1.0
# via
# swh-core
# swh-counters
# swh-scheduler
# swh-storage
frozenlist==1.5.0
# via
# aiohttp
# aiosignal
geomet==0.2.1.post1
# via cassandra-driver
gunicorn==23.0.0
humanize==4.10.0
hypothesis==6.111.0
idna==3.7
importlib_metadata==8.2.0
# via aiohttp-utils
h11==0.14.0
# via httpcore
httpcore==1.0.7
# via httpx
httpx==0.28.1
# via python-keycloak
humanize==4.12.2
# via swh-scheduler
hypothesis==6.130.4
# via swh-model
idna==3.10
# via
# anyio
# httpx
# requests
# yarl
importlib-metadata==8.6.1
# via swh-scheduler
iso8601==2.1.0
# via
# swh-core
# swh-model
# swh-storage
itsdangerous==2.2.0
Jinja2==3.1.4
kombu==5.4.0
MarkupSafe==2.1.5
msgpack==1.0.8
multidict==6.0.5
# via flask
jinja2==3.1.6
# via flask
jwcrypto==1.5.6
# via python-keycloak
kombu==5.5.1
# via celery
markupsafe==3.0.2
# via
# jinja2
# werkzeug
msgpack==1.1.0
# via
# swh-core
# swh-journal
# swh-objstorage
multidict==6.2.0
# via
# aiohttp
# yarl
mypy-extensions==1.0.0
packaging==24.1
pg8000==1.31.2
# via swh-storage
packaging==24.2
# via
# deprecation
# gunicorn
pika==1.3.2
prompt_toolkit==3.0.47
psycopg2==2.9.9
pyasn1==0.6.0
# via swh-scheduler
prompt-toolkit==3.0.50
# via click-repl
propcache==0.3.1
# via
# aiohttp
# yarl
psycopg==3.2.6
# via
# swh-core
# swh-scheduler
# swh-storage
psycopg-pool==3.2.6
# via
# swh-core
# swh-scheduler
# swh-storage
pycparser==2.22
# via cffi
python-dateutil==2.9.0.post0
python-jose==3.3.0
python-keycloak==3.8.4
# via
# celery
# swh-model
python-keycloak==5.3.1
# via swh-auth
python-magic==0.4.27
python-mimeparse==1.6.0
PyYAML==6.0.2
redis==5.0.8
# via swh-core
python-mimeparse==2.0.0
# via aiohttp-utils
pyyaml==6.0.2
# via
# swh-auth
# swh-core
# swh-scheduler
redis==5.2.1
# via
# swh-counters
# swh-storage
requests==2.32.3
# via
# python-keycloak
# requests-toolbelt
# swh-core
# swh-objstorage
# swh-scheduler
requests-toolbelt==1.0.0
rsa==4.9
scramp==1.4.5
sentry-sdk==2.12.0
six==1.16.0
# via python-keycloak
sentry-sdk==2.24.1
# via
# swh-core
# swh-scheduler
setuptools==78.1.0
# via swh-scheduler
six==1.17.0
# via
# geomet
# python-dateutil
sniffio==1.3.1
# via anyio
sortedcontainers==2.4.0
swh.auth==0.9.1
swh.core==3.4.0
swh.counters==0.11.0
swh.journal==1.5.2
swh.model==6.14.0
swh.objstorage==3.3.0
swh.perfecthash==1.3.2
swh.scheduler==2.4.0
swh.storage==2.7.0
# via hypothesis
swh-auth==0.10.0
# via -r /src/apps/swh-add-forge-now/requirements.txt
swh-core==4.1.0
# via
# swh-auth
# swh-counters
# swh-journal
# swh-objstorage
# swh-scheduler
# swh-storage
swh-counters==0.11.0
# via swh-storage
swh-journal==1.5.3
# via swh-counters
swh-model==7.1.0
# via
# swh-journal
# swh-objstorage
# swh-storage
swh-objstorage==4.0.0
# via swh-storage
swh-perfecthash==1.3.2
# via swh-objstorage
swh-scheduler==3.0.0
# via -r /src/apps/swh-add-forge-now/requirements.txt
swh-storage==3.0.0
# via swh-scheduler
tabulate==0.9.0
# via swh-scheduler
tenacity==9.0.0
testing.common.database==2.0.3
testing.postgresql==1.3.0
typing_extensions==4.12.2
tzdata==2024.1
urllib3==2.2.2
# via
# swh-core
# swh-journal
# swh-storage
typing-extensions==4.13.0
# via
# anyio
# jwcrypto
# psycopg
# psycopg-pool
# swh-core
# swh-model
# swh-scheduler
# swh-storage
tzdata==2025.1
# via
# celery
# kombu
urllib3==2.3.0
# via
# requests
# sentry-sdk
vine==5.1.0
# via
# amqp
# celery
# kombu
wcwidth==0.2.13
Werkzeug==3.0.3
wrapt==1.16.0
yarl==1.9.4
zipp==3.20.0
# via prompt-toolkit
werkzeug==3.1.3
# via flask
wrapt==1.17.2
# via deprecated
yarl==1.18.3
# via aiohttp
zipp==3.21.0
# via importlib-metadata
apps/swh-alter/Dockerfile
View file @ a8b3a6d3
FROM python:3.11-bullseye
ARG REGISTRY=container-registry.softwareheritage.org/swh/infra/swh-apps/
ARG base_image=${REGISTRY}base
ARG base_image_version=latest
FROM ${base_image}:${base_image_version}
ARG user=swh
ARG workdir=/opt/${user}
RUN apt-get -y update && \
USER root
RUN apt-get update && \
apt-get -y upgrade && \
apt-get install -y \
curl \
......@@ -20,15 +25,14 @@ RUN apt-get -y update && \
strace \
telnet \
tmux \
vim \
&& apt clean && \
addgroup --gid 1000 ${user} && \
useradd --gid 1000 --uid 1000 -m -d ${workdir} ${user} && \
mkdir /etc/${user}
vim && \
apt-get clean
USER ${user}
WORKDIR ${workdir}
COPY --chmod=0644 requirements-frozen.txt ${workdir}
RUN --mount=type=cache,target=.cache,uid=1000,gid=1000 \
uv pip sync requirements-frozen.txt
USER ${user}
ENV CARGO_HOME="${workdir}/.cargo"
ENV RUSTUP_HOME="${workdir}/.rustup"
......@@ -38,23 +42,14 @@ RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs -o /tmp/rustup-ini
rm /tmp/rustup-init.sh
ENV PATH="${CARGO_HOME}/bin:${PATH}"
# install rage (for swh-alter)
# Use the latest available version so we can detect if rage behavior has changed when
# testing swh-alter. Otherwise, this might only be detected when processing a
# takedown notice or restoring a bundle… which would not be a nice experience.
RUN cargo install rage
COPY --chown=${user}:${user} requirements-frozen.txt ${workdir}
ENV PYTHONPATH=${workdir}
ENV PATH=${workdir}/.local/bin:$PATH
RUN /usr/local/bin/python -m pip install --upgrade pip && \
pip install --no-cache-dir -r requirements-frozen.txt
RUN --mount=type=cache,target=.cache,uid=1000,gid=1000 \
cargo install rage
COPY --chown=${user}:${user} entrypoint.sh ${workdir}
RUN chmod u+x ${workdir}/entrypoint.sh
COPY --chmod=0755 entrypoint.sh ${workdir}
ENTRYPOINT "/opt/swh/entrypoint.sh"
ENTRYPOINT ["/opt/swh/entrypoint.sh"]
apps/swh-alter/entrypoint.sh
View file @ a8b3a6d3
......@@ -2,10 +2,23 @@
set -e
echo "swh-alter pod: running a tmux server"
tmux set -g exit-empty off \; start
case "$1" in
"shell")
shift
if (( $# == 0)); then
exec bash -i
else
"$@"
fi
exit 0
;;
*)
echo "swh-alter pod: running a tmux server"
tmux set -g exit-empty off \; start
trap "tmux kill-server" INT TERM EXIT
trap "tmux kill-server" INT TERM EXIT
while tmux list-sessions 2>/dev/null; do sleep 5; done
echo "tmux server has exited, bailing"
while tmux list-sessions 2>/dev/null; do sleep 5; done
echo "tmux server has exited, bailing"
;;
esac
apps/swh-alter/requirements-frozen.txt
View file @ a8b3a6d3
aiohappyeyeballs==2.4.0
aiohttp==3.10.5
# This file was autogenerated by uv via the following command:
# uv pip compile /src/apps/swh-alter/requirements.txt -o /src/apps/swh-alter/tmpljkffqaj
aiohappyeyeballs==2.6.1
# via aiohttp
aiohttp==3.11.14
# via
# aiohttp-utils
# swh-core
# swh-graph
aiohttp-utils==3.2.1
aiosignal==1.3.1
# via swh-core
aiosignal==1.3.2
# via aiohttp
apache-libcloud==3.8.0
async-timeout==4.0.3
attrs==24.2.0
# via swh-objstorage
attrs==25.3.0
# via
# aiohttp
# attrs-strict
# hypothesis
# swh-alter
# swh-model
attrs-strict==1.0.1
azure-core==1.30.2
azure-storage-blob==12.22.0
blinker==1.8.2
boto3==1.35.4
botocore==1.35.4
cachetools==5.5.0
cassandra-driver==3.29.1
certifi==2024.7.4
cffi==1.17.0
charset-normalizer==3.3.2
click==8.1.7
confluent-kafka==2.5.0
cryptography==43.0.0
Deprecated==1.2.14
elasticsearch==7.17.9
exceptiongroup==1.2.2
Flask==3.0.3
frozendict==2.4.4
frozenlist==1.4.1
# via swh-model
azure-core==1.32.0
# via azure-storage-blob
azure-storage-blob==12.25.0
# via swh-objstorage
backports-entry-points-selectable==1.3.0
# via
# swh-core
# swh-storage
blinker==1.9.0
# via
# flask
# swh-core
boto3==1.37.20
# via swh-graph
botocore==1.37.20
# via
# boto3
# s3transfer
cachetools==5.5.2
# via pyld
cassandra-driver==3.29.2
# via swh-storage
certifi==2025.1.31
# via
# elasticsearch
# requests
# sentry-sdk
cffi==1.17.1
# via
# cryptography
# swh-perfecthash
charset-normalizer==3.4.1
# via requests
click==8.1.8
# via
# flask
# geomet
# swh-alter
# swh-core
# swh-graph
# swh-indexer
# swh-objstorage
# swh-search
# swh-storage
confluent-kafka==2.8.2
# via swh-journal
cryptography==44.0.2
# via azure-storage-blob
deprecated==1.2.18
# via
# swh-core
# swh-model
# swh-objstorage
# swh-storage
elasticsearch==7.17.12
# via swh-search
flask==3.1.0
# via
# swh-core
# swh-counters
# swh-storage
frozendict==2.4.6
# via
# pyld
# swh-indexer
frozenlist==1.5.0
# via
# aiohttp
# aiosignal
geomet==0.2.1.post1
grpcio==1.66.0
grpcio-tools==1.62.3
# via cassandra-driver
grpcio==1.71.0
# via grpcio-tools
grpcio-tools==1.71.0
# via swh-graph
gunicorn==23.0.0
humanize==4.10.0
hypothesis==6.111.1
idna==3.7
igraph==0.11.6
# via aiohttp-utils
humanize==4.12.2
# via swh-alter
hypothesis==6.130.4
# via swh-model
idna==3.10
# via
# requests
# yarl
igraph==0.11.8
# via swh-alter
iso8601==2.1.0
isodate==0.6.1
# via
# swh-core
# swh-indexer
# swh-model
# swh-search
# swh-storage
isodate==0.7.2
# via azure-storage-blob
itsdangerous==2.2.0
Jinja2==3.1.4
# via flask
jinja2==3.1.6
# via flask
jmespath==1.0.1
# via
# boto3
# botocore
latexcodec==3.0.0
lxml==5.3.0
MarkupSafe==2.1.5
msgpack==1.0.8
multidict==6.0.5
# via pybtex
lxml==5.3.1
# via pyld
markupsafe==3.0.2
# via
# jinja2
# werkzeug
msgpack==1.1.0
# via
# swh-core
# swh-journal
# swh-objstorage
multidict==6.2.0
# via
# aiohttp
# yarl
mypy-extensions==1.0.0
mypy-protobuf==3.2.0
packaging==24.1
plyvel==1.5.1
protobuf==4.25.4
psutil==6.0.0
psycopg2==2.9.9
py4j==0.10.9.7
# via swh-storage
mypy-protobuf==3.6.0
# via swh-graph
packaging==24.2
# via gunicorn
propcache==0.3.1
# via
# aiohttp
# yarl
protobuf==5.29.4
# via
# grpcio-tools
# mypy-protobuf
# swh-graph
psutil==7.0.0
# via swh-graph
psycopg==3.2.6
# via
# swh-core
# swh-storage
psycopg-pool==3.2.6
# via
# swh-core
# swh-storage
py4j==0.10.9.9
# via swh-graph
pybtex==0.24.0
# via swh-indexer
pycparser==2.22
PyLD==2.0.4
pyorc==0.9.0
pyparsing==3.1.2
# via cffi
pyld==2.0.4
# via swh-indexer
pyparsing==3.2.3
# via rdflib
python-dateutil==2.9.0.post0
# via
# botocore
# swh-model
python-magic==0.4.27
python-mimeparse==1.6.0
PyYAML==6.0.2
rdflib==7.0.0
redis==5.0.8
# via
# swh-core
# swh-indexer
python-mimeparse==2.0.0
# via aiohttp-utils
pyyaml==6.0.2
# via
# pybtex
# swh-alter
# swh-core
rdflib==7.1.3
# via swh-indexer
redis==5.2.1
# via
# swh-counters
# swh-storage
requests==2.32.3
s3transfer==0.10.2
sentry-sdk==2.13.0
# via
# apache-libcloud
# azure-core
# swh-core
# swh-objstorage
s3transfer==0.11.4
# via boto3
sentry-sdk==2.24.1
# via
# swh-core
# swh-indexer
setuptools==78.1.0
# via grpcio-tools
shamir-mnemonic==0.3.0
six==1.16.0
# via swh-alter
six==1.17.0
# via
# azure-core
# geomet
# pybtex
# python-dateutil
sortedcontainers==2.4.0
swh.alter==1.2.0
swh.core==3.4.0
swh.counters==0.11.0
swh.dataset==1.7.0
swh.graph==5.1.1
swh.indexer==3.4.0
swh.journal==1.5.2
swh.model==6.14.0
swh.objstorage==3.3.0
swh.perfecthash==1.3.2
swh.search==0.21.0
swh.storage==2.7.0
# via hypothesis
swh-alter==1.4.0
# via -r /src/apps/swh-alter/requirements.txt
swh-core==4.1.0
# via
# swh-alter
# swh-counters
# swh-graph
# swh-indexer
# swh-journal
# swh-objstorage
# swh-search
# swh-storage
swh-counters==0.11.0
# via swh-storage
swh-graph==6.7.1
# via swh-alter
swh-indexer==4.0.0
# via swh-search
swh-journal==1.5.3
# via
# swh-counters
# swh-indexer
# swh-search
swh-model==7.1.0
# via
# swh-alter
# swh-graph
# swh-indexer
# swh-journal
# swh-objstorage
# swh-search
# swh-storage
swh-objstorage==4.0.0
# via
# -r /src/apps/swh-alter/requirements.txt
# swh-alter
# swh-indexer
# swh-storage
swh-perfecthash==1.3.2
# via swh-objstorage
swh-search==0.22.1
# via swh-alter
swh-storage==3.0.0
# via
# swh-alter
# swh-indexer
tabulate==0.9.0
# via swh-alter
tenacity==9.0.0
# via
# swh-core
# swh-journal
# swh-storage
texttable==1.7.0
tqdm==4.66.5
# via igraph
tree-sitter==0.21.3
types-protobuf==5.27.0.20240626
types-requests==2.31.0.6
types-urllib3==1.26.25.14
typing_extensions==4.12.2
urllib3==1.26.19
Werkzeug==3.0.4
wrapt==1.16.0
xmltodict==0.13.0
yarl==1.9.4
# via swh-search
types-protobuf==5.29.1.20250315
# via mypy-protobuf
typing-extensions==4.13.0
# via
# azure-core
# azure-storage-blob
# psycopg
# psycopg-pool
# swh-alter
# swh-core
# swh-indexer
# swh-model
# swh-search
# swh-storage
urllib3==1.26.20
# via
# botocore
# elasticsearch
# requests
# sentry-sdk
werkzeug==3.1.3
# via flask
wrapt==1.17.2
# via deprecated
xmltodict==0.14.2
# via swh-indexer
yarl==1.18.3
# via aiohttp
apps/swh-base/Dockerfile 0 → 100644
View file @ a8b3a6d3
FROM python:3.11-bookworm
ARG userid=1000
ARG groupid=1000
ARG user=swh
ARG workdir=/opt/${user}
ARG venv=${workdir}/venv
ARG configdir=/etc/${user}
RUN apt-get update && \
apt-get upgrade -y && \
apt-get install -y git sed \
gzip lzip zstd cpio \
less && \
apt-get clean && \
addgroup --gid ${userid} ${user} && \
useradd --gid ${groupid} --uid ${userid} -m -d ${workdir} ${user} && \
mkdir ${configdir}
USER ${user}
RUN python -m venv ${venv}
WORKDIR ${workdir}
ENV PYTHONPATH=${venv}
ENV PATH=${venv}/bin:$PATH
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
apps/swh-counters/Dockerfile
View file @ a8b3a6d3
# Deeply inspired from the Dockerfile of the swh-storage project
FROM python:3.10-bullseye
ARG REGISTRY=container-registry.softwareheritage.org/swh/infra/swh-apps/
ARG base_image=${REGISTRY}base
ARG base_image_version=latest
FROM ${base_image}:${base_image_version}
ARG user=swh
ARG workdir=/opt/${user}
ARG configdir=/etc/${user}
USER root
# retry command is used by the refresh-history cron job
# to ensure the rpc service is well up
RUN apt-get -y update && \
apt-get -y upgrade && \
apt-get install -y libcmph-dev librdkafka-dev retry && \
apt clean && \
addgroup --gid 1000 swh && \
useradd --gid 1000 --uid 1000 -m -d /opt/swh swh && \
mkdir /etc/swh
USER swh
WORKDIR /opt/swh
COPY --chown=swh:swh requirements-frozen.txt /opt/swh
ENV PYTHONPATH=/opt/swh
ENV PATH=/opt/swh/.local/bin:$PATH
apt-get install retry && \
apt-get clean
RUN /usr/local/bin/python -m pip install --upgrade pip && \
pip install --no-cache-dir -r requirements-frozen.txt && \
pip install gunicorn
COPY --chmod=0644 requirements-frozen.txt ${workdir}
RUN --mount=type=cache,target=.cache,uid=1000,gid=1000 \
uv pip sync requirements-frozen.txt
COPY --chown=swh:swh entrypoint.sh /opt/swh
RUN chmod u+x /opt/swh/entrypoint.sh
COPY --chmod=0755 entrypoint.sh ${workdir}
ENV SWH_CONFIG_FILENAME=/etc/swh/config.yml
ENV PORT 5011
USER ${user}
ENV SWH_CONFIG_FILENAME=${configdir}/config.yml
ENV PORT=5011
EXPOSE $PORT
ENV THREADS 2
ENV WORKERS 2
ENV TIMEOUT 3600
ENV THREADS=2
ENV WORKERS=2
ENV TIMEOUT=3600
ENTRYPOINT ["/opt/swh/entrypoint.sh"]
apps/swh-counters/entrypoint.sh
View file @ a8b3a6d3
......@@ -17,19 +17,24 @@ case "$1" in
exec swh $@
;;
*)
EXTRA_CLI_FLAGS=""
if [ ! -z "${SWH_LOG_CONFIG_JSON}" ]; then
EXTRA_CLI_FLAGS="--log-config-json ${SWH_LOG_CONFIG_JSON}"
EXTRA_CLI_FLAGS=()
if [ -n "${SWH_LOG_CONFIG_JSON}" ]; then
EXTRA_CLI_FLAGS+=('--log-config-json' "${SWH_LOG_CONFIG_JSON}")
fi
echo Starting the swh-counters API server
exec gunicorn --bind 0.0.0.0:${PORT} \
--log-level ${SWH_LOG_LEVEL:-INFO} \
${EXTRA_CLI_FLAGS} \
--threads ${THREADS} \
--workers ${WORKERS} \
--timeout ${TIMEOUT} \
--statsd-host=${STATSD_HOST}:${STATSD_PORT} \
--statsd-prefix=${STATSD_SERVICE_TYPE} \
if [ -n "${STATSD_HOST}" -a -n "${STATSD_PORT}" ]; then
EXTRA_CLI_FLAGS+=('--statsd-host' "${STATSD_HOST}:${STATSD_PORT}")
fi
if [ -n "${STATSD_SERVICE_TYPE}" ]; then
EXTRA_CLI_FLAGS+=('--statsd-prefix' "${STATSD_SERVICE_TYPE}")
fi
echo 'Starting the swh-counters API server'
exec gunicorn --bind "0.0.0.0:${PORT}" \
--log-level "${SWH_LOG_LEVEL:-INFO}" \
"${EXTRA_CLI_FLAGS[@]}" \
--threads "${THREADS}" \
--workers "${WORKERS}" \
--timeout "${TIMEOUT}" \
--reload \
--config 'python:swh.core.api.gunicorn_config' \
'swh.counters.api.server:make_app_from_configfile()'
......
apps/swh-counters/requirements-frozen.txt
View file @ a8b3a6d3
aiohappyeyeballs==2.4.0
aiohttp==3.10.5
# This file was autogenerated by uv via the following command:
# uv pip compile /src/apps/swh-counters/requirements.txt -o /src/apps/swh-counters/tmpiv8trarv
aiohappyeyeballs==2.6.1
# via aiohttp
aiohttp==3.11.14
# via
# aiohttp-utils
# swh-core
aiohttp-utils==3.2.1
aiosignal==1.3.1
async-timeout==4.0.3
attrs==24.2.0
# via swh-core
aiosignal==1.3.2
# via aiohttp
attrs==25.3.0
# via
# aiohttp
# attrs-strict
# hypothesis
# swh-model
attrs-strict==1.0.1
blinker==1.8.2
certifi==2024.7.4
charset-normalizer==3.3.2
click==8.1.7
confluent-kafka==2.5.0
Deprecated==1.2.14
exceptiongroup==1.2.2
Flask==3.0.3
frozenlist==1.4.1
# via swh-model
backports-entry-points-selectable==1.3.0
# via swh-core
blinker==1.9.0
# via
# flask
# swh-core
certifi==2025.1.31
# via
# requests
# sentry-sdk
charset-normalizer==3.4.1
# via requests
click==8.1.8
# via
# flask
# swh-core
confluent-kafka==2.8.2
# via swh-journal
deprecated==1.2.18
# via
# swh-core
# swh-model
flask==3.1.0
# via
# swh-core
# swh-counters
frozenlist==1.5.0
# via
# aiohttp
# aiosignal
gunicorn==23.0.0
hypothesis==6.111.2
idna==3.8
# via
# -r /src/apps/swh-counters/requirements.txt
# aiohttp-utils
hypothesis==6.130.4
# via swh-model
idna==3.10
# via
# requests
# yarl
iso8601==2.1.0
# via
# swh-core
# swh-model
itsdangerous==2.2.0
Jinja2==3.1.4
MarkupSafe==2.1.5
msgpack==1.0.8
multidict==6.0.5
packaging==24.1
# via flask
jinja2==3.1.6
# via flask
markupsafe==3.0.2
# via
# jinja2
# werkzeug
msgpack==1.1.0
# via
# swh-core
# swh-journal
multidict==6.2.0
# via
# aiohttp
# yarl
packaging==24.2
# via gunicorn
propcache==0.3.1
# via
# aiohttp
# yarl
python-dateutil==2.9.0.post0
python-json-logger==2.0.7
# via swh-model
python-json-logger==3.3.0
# via -r /src/apps/swh-counters/requirements.txt
python-magic==0.4.27
# via swh-core
python-mimeparse==2.0.0
PyYAML==6.0.2
redis==5.0.8
# via aiohttp-utils
pyyaml==6.0.2
# via swh-core
redis==5.2.1
# via swh-counters
requests==2.32.3
sentry-sdk==2.13.0
six==1.16.0
# via swh-core
sentry-sdk==2.24.1
# via swh-core
six==1.17.0
# via python-dateutil
sortedcontainers==2.4.0
swh.core==3.4.0
swh.counters==0.11.0
swh.journal==1.5.2
swh.model==6.14.0
# via hypothesis
swh-core==4.1.0
# via
# swh-counters
# swh-journal
swh-counters==0.11.0
# via -r /src/apps/swh-counters/requirements.txt
swh-journal==1.5.3
# via swh-counters
swh-model==7.1.0
# via swh-journal
tenacity==9.0.0
typing_extensions==4.12.2
urllib3==2.2.2
Werkzeug==3.0.4
wrapt==1.16.0
yarl==1.9.4
# via
# swh-core
# swh-journal
typing-extensions==4.13.0
# via swh-model
urllib3==2.3.0
# via
# requests
# sentry-sdk
werkzeug==3.1.3
# via flask
wrapt==1.17.2
# via deprecated
yarl==1.18.3
# via aiohttp
apps/swh-counters/requirements.txt
View file @ a8b3a6d3
swh.counters
python-json-logger
gunicorn
apps/swh-deposit-checkers/Dockerfile
View file @ a8b3a6d3
# Deeply inspired from the Dockerfile of the swh-graph project
FROM python:3.10-bullseye
ARG REGISTRY=container-registry.softwareheritage.org/swh/infra/swh-apps/
ARG base_image=${REGISTRY}base
ARG base_image_version=latest
RUN apt-get -y update && \
apt-get -y upgrade && \
apt-get install -y libcmph-dev && \
apt clean && \
addgroup --gid 1000 swh && \
useradd --gid 1000 --uid 1000 -m -d /opt/swh swh && \
mkdir /etc/swh
FROM ${base_image}:${base_image_version}
USER swh
WORKDIR /opt/swh
ARG user=swh
ARG workdir=/opt/${user}
ARG configdir=/etc/${user}
COPY --chown=swh:swh requirements-frozen.txt /opt/swh
COPY --chown=swh:swh entrypoint.sh /opt/swh
COPY --chmod=0644 requirements-frozen.txt ${workdir}
RUN --mount=type=cache,target=.cache,uid=1000,gid=1000 \
uv pip sync requirements-frozen.txt
ENV PYTHONPATH=/opt/swh
ENV PATH=/opt/swh/.local/bin:$PATH
COPY --chmod=0755 entrypoint.sh ${workdir}
RUN chmod u+x /opt/swh/entrypoint.sh && \
/usr/local/bin/python -m pip install --upgrade pip && \
pip install --no-cache-dir -r requirements-frozen.txt && \
pip install gunicorn
ENV SWH_CONFIG_FILENAME=/etc/swh/config.yml
USER ${user}
ENV SWH_CONFIG_FILENAME=${configdir}/config.yml
ENV SWH_WORKER_INSTANCE=checker
ENV CONCURRENCY=1
ENV MAX_TASKS_PER_CHILD=5
ENTRYPOINT "/opt/swh/entrypoint.sh"
ENTRYPOINT ["/opt/swh/entrypoint.sh"]
apps/swh-deposit-checkers/entrypoint.sh
View file @ a8b3a6d3
......@@ -3,22 +3,25 @@
set -e
case "$1" in
"shell")
shift
echo "Running command $@"
exec bash -i "$@"
;;
*)
echo Starting the swh Celery worker for ${SWH_WORKER_INSTANCE}
exec python -m celery \
--app=swh.scheduler.celery_backend.config.app \
worker \
--pool=prefork \
--concurrency=${CONCURRENCY} \
--max-tasks-per-child=${MAX_TASKS_PER_CHILD} \
-Ofair \
--loglevel=${SWH_LOG_LEVEL:-INFO} \
--without-gossip --without-mingle --without-heartbeat \
--hostname "${SWH_WORKER_INSTANCE}@%h"
;;
"shell")
shift
if (( $# == 0)); then
exec bash -i
else
"$@"
fi
;;
*)
echo Starting the swh Celery worker for ${SWH_WORKER_INSTANCE}
exec python -m celery \
--app=swh.scheduler.celery_backend.config.app \
worker \
--pool=prefork \
--concurrency=${CONCURRENCY} \
--max-tasks-per-child=${MAX_TASKS_PER_CHILD} \
-Ofair \
--loglevel=${SWH_LOG_LEVEL:-INFO} \
--without-gossip --without-mingle --without-heartbeat \
--hostname "${SWH_WORKER_INSTANCE}@%h"
;;
esac
apps/swh-deposit-checkers/requirements-frozen.txt
View file @ a8b3a6d3
aiohappyeyeballs==2.3.5
aiohttp==3.10.3
# This file was autogenerated by uv via the following command:
# uv pip compile /src/apps/swh-deposit-checkers/requirements.txt -o /src/apps/swh-deposit-checkers/tmpezx7r6y1
aiohappyeyeballs==2.6.1
# via aiohttp
aiohttp==3.11.14
# via
# aiohttp-utils
# swh-core
aiohttp-utils==3.2.1
aiosignal==1.3.1
amqp==5.2.0
asn1crypto==1.5.1
async-timeout==4.0.3
attrs==24.2.0
# via swh-core
aiosignal==1.3.2
# via aiohttp
amqp==5.3.1
# via kombu
attrs==25.3.0
# via
# aiohttp
# attrs-strict
# hypothesis
# swh-model
# swh-scheduler
attrs-strict==1.0.1
billiard==4.2.0
blinker==1.8.2
cassandra-driver==3.29.1
# via
# swh-model
# swh-scheduler
backports-entry-points-selectable==1.3.0
# via
# swh-core
# swh-storage
billiard==4.2.1
# via celery
blinker==1.9.0
# via
# flask
# swh-core
cassandra-driver==3.29.2
# via swh-storage
celery==5.4.0
certifi==2024.7.4
cffi==1.17.0
charset-normalizer==3.3.2
click==8.1.7
# via swh-scheduler
certifi==2025.1.31
# via
# requests
# sentry-sdk
cffi==1.17.1
# via swh-perfecthash
charset-normalizer==3.4.1
# via requests
click==8.1.8
# via
# celery
# click-didyoumean
# click-plugins
# click-repl
# flask
# geomet
# swh-core
# swh-deposit
# swh-objstorage
# swh-scheduler
# swh-storage
click-didyoumean==0.3.1
# via celery
click-plugins==1.1.1
# via celery
click-repl==0.3.0
confluent-kafka==2.5.0
Deprecated==1.2.14
exceptiongroup==1.2.2
Flask==3.0.3
frozenlist==1.4.1
# via celery
confluent-kafka==2.8.2
# via swh-journal
deprecated==1.2.18
# via
# swh-core
# swh-model
# swh-objstorage
# swh-storage
elementpath==4.8.0
# via xmlschema
flask==3.1.0
# via
# swh-core
# swh-counters
# swh-scheduler
# swh-storage
frozenlist==1.5.0
# via
# aiohttp
# aiosignal
geomet==0.2.1.post1
# via cassandra-driver
gunicorn==23.0.0
humanize==4.10.0
hypothesis==6.111.0
idna==3.7
importlib_metadata==8.2.0
# via aiohttp-utils
humanize==4.12.2
# via swh-scheduler
hypothesis==6.130.4
# via swh-model
idna==3.10
# via
# requests
# yarl
importlib-metadata==8.6.1
# via swh-scheduler
iso8601==2.1.0
# via
# swh-core
# swh-deposit
# swh-model
# swh-storage
itsdangerous==2.2.0
Jinja2==3.1.4
kombu==5.4.0
MarkupSafe==2.1.5
msgpack==1.0.8
multidict==6.0.5
# via flask
jinja2==3.1.6
# via flask
kombu==5.5.1
# via celery
markupsafe==3.0.2
# via
# jinja2
# werkzeug
msgpack==1.1.0
# via
# swh-core
# swh-journal
# swh-objstorage
multidict==6.2.0
# via
# aiohttp
# yarl
mypy-extensions==1.0.0
packaging==24.1
pg8000==1.31.2
# via swh-storage
packaging==24.2
# via gunicorn
pika==1.3.2
prompt_toolkit==3.0.47
psycopg2==2.9.9
# via swh-scheduler
prompt-toolkit==3.0.50
# via click-repl
propcache==0.3.1
# via
# aiohttp
# yarl
psycopg==3.2.6
# via
# swh-core
# swh-scheduler
# swh-storage
psycopg-pool==3.2.6
# via
# swh-core
# swh-scheduler
# swh-storage
pycparser==2.22
# via cffi
python-dateutil==2.9.0.post0
python-json-logger==2.0.7
# via
# celery
# swh-model
python-json-logger==3.3.0
# via -r /src/apps/swh-deposit-checkers/requirements.txt
python-magic==0.4.27
python-mimeparse==1.6.0
PyYAML==6.0.2
redis==5.0.8
# via swh-core
python-mimeparse==2.0.0
# via aiohttp-utils
pyyaml==6.0.2
# via
# swh-core
# swh-scheduler
redis==5.2.1
# via
# swh-counters
# swh-storage
requests==2.32.3
scramp==1.4.5
sentry-sdk==2.12.0
six==1.16.0
# via
# swh-core
# swh-deposit
# swh-objstorage
# swh-scheduler
sentry-sdk==2.24.1
# via
# swh-core
# swh-deposit
# swh-scheduler
setuptools==78.1.0
# via swh-scheduler
six==1.17.0
# via
# geomet
# python-dateutil
sortedcontainers==2.4.0
swh.core==3.4.0
swh.counters==0.11.0
swh.deposit==2.2.2
swh.journal==1.5.2
swh.model==6.14.0
swh.objstorage==3.3.0
swh.perfecthash==1.3.2
swh.scheduler==2.4.0
swh.storage==2.7.0
# via hypothesis
swh-core==4.1.0
# via
# swh-counters
# swh-deposit
# swh-journal
# swh-objstorage
# swh-scheduler
# swh-storage
swh-counters==0.11.0
# via swh-storage
swh-deposit==3.0.0
# via -r /src/apps/swh-deposit-checkers/requirements.txt
swh-journal==1.5.3
# via swh-counters
swh-model==7.1.0
# via
# swh-deposit
# swh-journal
# swh-objstorage
# swh-storage
swh-objstorage==4.0.0
# via swh-storage
swh-perfecthash==1.3.2
# via swh-objstorage
swh-scheduler==3.0.0
# via -r /src/apps/swh-deposit-checkers/requirements.txt
swh-storage==3.0.0
# via swh-scheduler
tabulate==0.9.0
# via swh-scheduler
tenacity==9.0.0
testing.common.database==2.0.3
testing.postgresql==1.3.0
typing_extensions==4.12.2
tzdata==2024.1
urllib3==2.2.2
# via
# swh-core
# swh-journal
# swh-storage
typing-extensions==4.13.0
# via
# psycopg
# psycopg-pool
# swh-core
# swh-model
# swh-scheduler
# swh-storage
tzdata==2025.1
# via
# celery
# kombu
urllib3==2.3.0
# via
# requests
# sentry-sdk
vine==5.1.0
# via
# amqp
# celery
# kombu
wcwidth==0.2.13
Werkzeug==3.0.3
wrapt==1.16.0
yarl==1.9.4
zipp==3.20.0
# via prompt-toolkit
werkzeug==3.1.3
# via flask
wrapt==1.17.2
# via deprecated
xmlschema==3.4.5
# via swh-deposit
yarl==1.18.3
# via aiohttp
zipp==3.21.0
# via importlib-metadata
apps/swh-deposit/Dockerfile
View file @ a8b3a6d3
# Deeply inspired from the Dockerfile of the swh-storage project
FROM python:3.10-bullseye
ARG REGISTRY=container-registry.softwareheritage.org/swh/infra/swh-apps/
ARG base_image=${REGISTRY}base
ARG base_image_version=latest
RUN apt-get -y update && \
apt-get -y upgrade && \
apt-get install -y libcmph-dev librdkafka-dev && \
apt clean && \
addgroup --gid 1000 swh && \
useradd --gid 1000 --uid 1000 -m -d /opt/swh swh && \
mkdir /etc/swh
FROM ${base_image}:${base_image_version}
USER swh
WORKDIR /opt/swh
ARG user=swh
ARG workdir=/opt/${user}
ARG configdir=/etc/${user}
COPY --chown=swh:swh requirements-frozen.txt /opt/swh
COPY --chmod=0644 requirements-frozen.txt ${workdir}
RUN --mount=type=cache,target=.cache,uid=1000,gid=1000 \
uv pip sync requirements-frozen.txt
ENV PYTHONPATH=/opt/swh
ENV PATH=/opt/swh/.local/bin:$PATH
COPY --chmod=0755 entrypoint.sh ${workdir}
RUN /usr/local/bin/python -m pip install --upgrade pip && \
pip install --no-cache-dir -r requirements-frozen.txt && \
pip install gunicorn
COPY --chown=swh:swh entrypoint.sh /opt/swh
RUN chmod u+x /opt/swh/entrypoint.sh
ENV SWH_CONFIG_FILENAME=/etc/swh/config.yml
ENV PORT 5006
USER ${user}
ENV SWH_CONFIG_FILENAME=${configdir}/config.yml
ENV PORT=5006
EXPOSE $PORT
ENV THREADS 2
ENV WORKERS 2
ENV TIMEOUT 3600
ENV DJANGO_SETTINGS_MODULE swh.deposit.settings.production
ENV SWH_MAIN_PACKAGE swh.deposit
ENV THREADS=2
ENV WORKERS=2
ENV TIMEOUT=3600
ENV DJANGO_SETTINGS_MODULE=swh.deposit.settings.production
ENV SWH_MAIN_PACKAGE=swh.deposit
ENTRYPOINT ["/opt/swh/entrypoint.sh"]