app_manager: Use more efficient uv cli instead of pip

It's a proposal out of nico's yesterday remark on making that step faster. It's not looking up the cache part.

It's more efficient and faster for computing the requirements (twice as fast with the graph deps) [1] vs [2]

Docker layers were already cached so the time comparison is viable.

Also, I've only focused on the frozen requirements step proposal. I've failed to make it work on other container Dockerfile (since uv wants to work on venv and we don't have any there).

Note that the requirements-frozen.txt generated by uv is slightly different (while still compatible afaict) [3]

[1] Generate frozen requirements with pip in ~22s total

tony@kessel:~/work/inria/repo/swh/sysadm-environment/swh-apps (master) $ time docker run --rm -v $(pwd):/src app-manager generate-frozen-requirements swh-graph
Requirement already satisfied: pip in /tmp/swh-graphzwxbv4a7/lib/python3.10/site-packages (23.0.1)
Collecting pip
  Using cached pip-24.3.1-py3-none-any.whl (1.8 MB)
Requirement already satisfied: setuptools in /tmp/swh-graphzwxbv4a7/lib/python3.10/site-packages (65.5.0)
Collecting setuptools
  Using cached setuptools-75.3.0-py3-none-any.whl (1.3 MB)
Collecting wheel
  Downloading wheel-0.44.0-py3-none-any.whl (67 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67.1/67.1 kB 10.5 MB/s eta 0:00:00
Installing collected packages: wheel, setuptools, pip
  Attempting uninstall: setuptools
    Found existing installation: setuptools 65.5.0
    Uninstalling setuptools-65.5.0:
      Successfully uninstalled setuptools-65.5.0
  Attempting uninstall: pip
    Found existing installation: pip 23.0.1
    Uninstalling pip-23.0.1:
      Successfully uninstalled pip-23.0.1
Successfully installed pip-24.3.1 setuptools-75.3.0 wheel-0.44.0
Collecting swh.graph (from -r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading swh.graph-6.1.0-py3-none-any.whl.metadata (3.9 kB)
Collecting python-json-logger (from -r /src/apps/swh-graph/requirements.txt (line 2))
  Downloading python_json_logger-2.0.7-py3-none-any.whl.metadata (6.5 kB)
Collecting aiohttp (from swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading aiohttp-3.10.10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.6 kB)
Collecting boto3 (from swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading boto3-1.35.54-py3-none-any.whl.metadata (6.7 kB)
Collecting click (from swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting py4j (from swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading py4j-0.10.9.7-py2.py3-none-any.whl.metadata (1.5 kB)
Collecting psutil (from swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading psutil-6.1.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (22 kB)
Collecting protobuf<5.26.0,>=4.21.11 (from swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading protobuf-4.25.5-cp37-abi3-manylinux2014_x86_64.whl.metadata (541 bytes)
Collecting grpcio-tools (from swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading grpcio_tools-1.67.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
Collecting mypy-protobuf<3.3 (from swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading mypy_protobuf-3.2.0-py3-none-any.whl.metadata (493 bytes)
Collecting swh.core>=0.3 (from swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading swh.core-3.6.2-py3-none-any.whl.metadata (3.7 kB)
Collecting swh.model>=6.13.0 (from swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading swh.model-6.16.0-py3-none-any.whl.metadata (2.9 kB)
Collecting swh.dataset (from swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading swh.dataset-1.7.0-py3-none-any.whl.metadata (2.0 kB)
Collecting types-protobuf>=3.19.5 (from mypy-protobuf<3.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading types_protobuf-5.28.3.20241030-py3-none-any.whl.metadata (2.3 kB)
Collecting backports.entry-points-selectable (from swh.core>=0.3->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading backports.entry_points_selectable-1.3.0-py3-none-any.whl.metadata (4.1 kB)
Collecting deprecated (from swh.core>=0.3->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading Deprecated-1.2.14-py2.py3-none-any.whl.metadata (5.4 kB)
Collecting python-magic (from swh.core>=0.3->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading python_magic-0.4.27-py2.py3-none-any.whl.metadata (5.8 kB)
Collecting pyyaml (from swh.core>=0.3->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Using cached PyYAML-6.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB)
Collecting requests (from swh.core>=0.3->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)
Collecting sentry-sdk>=2 (from swh.core>=0.3->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading sentry_sdk-2.18.0-py2.py3-none-any.whl.metadata (9.9 kB)
Collecting tenacity>=8.4.2 (from swh.core>=0.3->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading tenacity-9.0.0-py3-none-any.whl.metadata (1.2 kB)
Collecting aiohttp-utils>=3.1.1 (from swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading aiohttp_utils-3.2.1-py2.py3-none-any.whl.metadata (2.6 kB)
Collecting blinker (from swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading blinker-1.8.2-py3-none-any.whl.metadata (1.6 kB)
Collecting flask (from swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading flask-3.0.3-py3-none-any.whl.metadata (3.2 kB)
Collecting iso8601 (from swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading iso8601-2.1.0-py3-none-any.whl.metadata (3.7 kB)
Collecting msgpack>=1.0.0 (from swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading msgpack-1.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (8.4 kB)
Collecting attrs!=21.1.0 (from swh.model>=6.13.0->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading attrs-24.2.0-py3-none-any.whl.metadata (11 kB)
Collecting attrs-strict>=0.0.7 (from swh.model>=6.13.0->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading attrs_strict-1.0.1-py3-none-any.whl.metadata (6.3 kB)
Collecting hypothesis (from swh.model>=6.13.0->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading hypothesis-6.116.0-py3-none-any.whl.metadata (6.0 kB)
Collecting python-dateutil (from swh.model>=6.13.0->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting typing-extensions (from swh.model>=6.13.0->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
Collecting aiohappyeyeballs>=2.3.0 (from aiohttp->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading aiohappyeyeballs-2.4.3-py3-none-any.whl.metadata (6.1 kB)
Collecting aiosignal>=1.1.2 (from aiohttp->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading aiosignal-1.3.1-py3-none-any.whl.metadata (4.0 kB)
Collecting frozenlist>=1.1.1 (from aiohttp->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading frozenlist-1.5.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (13 kB)
Collecting multidict<7.0,>=4.5 (from aiohttp->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading multidict-6.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.0 kB)
Collecting yarl<2.0,>=1.12.0 (from aiohttp->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading yarl-1.17.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (64 kB)
Collecting async-timeout<5.0,>=4.0 (from aiohttp->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading async_timeout-4.0.3-py3-none-any.whl.metadata (4.2 kB)
Collecting botocore<1.36.0,>=1.35.54 (from boto3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading botocore-1.35.54-py3-none-any.whl.metadata (5.7 kB)
Collecting jmespath<2.0.0,>=0.7.1 (from boto3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading jmespath-1.0.1-py3-none-any.whl.metadata (7.6 kB)
Collecting s3transfer<0.11.0,>=0.10.0 (from boto3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading s3transfer-0.10.3-py3-none-any.whl.metadata (1.7 kB)
INFO: pip is looking at multiple versions of grpcio-tools to determine which version is compatible with other requirements. This could take a while.
Collecting grpcio-tools (from swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading grpcio_tools-1.67.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
  Downloading grpcio_tools-1.66.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
  Downloading grpcio_tools-1.66.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
  Downloading grpcio_tools-1.66.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
  Downloading grpcio_tools-1.65.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
  Downloading grpcio_tools-1.65.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
  Downloading grpcio_tools-1.65.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
INFO: pip is still looking at multiple versions of grpcio-tools to determine which version is compatible with other requirements. This could take a while.
  Downloading grpcio_tools-1.65.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
  Downloading grpcio_tools-1.64.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
  Downloading grpcio_tools-1.64.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
  Downloading grpcio_tools-1.64.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
  Downloading grpcio_tools-1.63.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
  Downloading grpcio_tools-1.63.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
  Downloading grpcio_tools-1.62.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.2 kB)
Collecting grpcio>=1.62.3 (from grpcio-tools->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading grpcio-1.67.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.9 kB)
Requirement already satisfied: setuptools in /tmp/swh-graphzwxbv4a7/lib/python3.10/site-packages (from grpcio-tools->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1)) (75.3.0)
Collecting tqdm (from swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading tqdm-4.66.6-py3-none-any.whl.metadata (57 kB)
Collecting plyvel (from swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading plyvel-1.5.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB)
Collecting swh.journal>=0.9 (from swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading swh.journal-1.5.2-py3-none-any.whl.metadata (1.9 kB)
Collecting swh.storage>=2.3.1 (from swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading swh.storage-2.8.1-py3-none-any.whl.metadata (8.6 kB)
Collecting types-requests (from swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading types_requests-2.32.0.20241016-py3-none-any.whl.metadata (1.9 kB)
Collecting pyorc (from swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading pyorc-0.9.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.8 kB)
Collecting python-mimeparse (from aiohttp-utils>=3.1.1->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading python_mimeparse-2.0.0-py3-none-any.whl.metadata (3.1 kB)
Collecting gunicorn (from aiohttp-utils>=3.1.1->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading gunicorn-23.0.0-py3-none-any.whl.metadata (4.4 kB)
Collecting urllib3!=2.2.0,<3,>=1.25.4 (from botocore<1.36.0,>=1.35.54->boto3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Using cached urllib3-2.2.3-py3-none-any.whl.metadata (6.5 kB)
Collecting six>=1.5 (from python-dateutil->swh.model>=6.13.0->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB)
Collecting certifi (from sentry-sdk>=2->swh.core>=0.3->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading certifi-2024.8.30-py3-none-any.whl.metadata (2.2 kB)
Collecting confluent-kafka!=2.1.0 (from swh.journal>=0.9->swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading confluent_kafka-2.6.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (2.3 kB)
Collecting cassandra-driver!=3.21.0,>=3.19.0 (from swh.storage>=2.3.1->swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading cassandra_driver-3.29.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.2 kB)
Collecting mypy-extensions (from swh.storage>=2.3.1->swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading mypy_extensions-1.0.0-py3-none-any.whl.metadata (1.1 kB)
Collecting psycopg2 (from swh.storage>=2.3.1->swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading psycopg2-2.9.10.tar.gz (385 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting redis (from swh.storage>=2.3.1->swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading redis-5.2.0-py3-none-any.whl.metadata (9.1 kB)
Collecting swh.counters>=v0.8.0 (from swh.storage>=2.3.1->swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading swh.counters-0.11.0-py3-none-any.whl.metadata (1.9 kB)
Collecting swh.objstorage>=2.2.0 (from swh.storage>=2.3.1->swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading swh.objstorage-3.3.0-py3-none-any.whl.metadata (4.5 kB)
Collecting idna>=2.0 (from yarl<2.0,>=1.12.0->aiohttp->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading idna-3.10-py3-none-any.whl.metadata (10 kB)
Collecting propcache>=0.2.0 (from yarl<2.0,>=1.12.0->aiohttp->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading propcache-0.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.7 kB)
Collecting wrapt<2,>=1.10 (from deprecated->swh.core>=0.3->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading wrapt-1.16.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Collecting Werkzeug>=3.0.0 (from flask->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading werkzeug-3.1.2-py3-none-any.whl.metadata (3.7 kB)
Collecting Jinja2>=3.1.2 (from flask->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading jinja2-3.1.4-py3-none-any.whl.metadata (2.6 kB)
Collecting itsdangerous>=2.1.2 (from flask->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading itsdangerous-2.2.0-py3-none-any.whl.metadata (1.9 kB)
Collecting sortedcontainers<3.0.0,>=2.1.0 (from hypothesis->swh.model>=6.13.0->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading sortedcontainers-2.4.0-py2.py3-none-any.whl.metadata (10 kB)
Collecting exceptiongroup>=1.0.0 (from hypothesis->swh.model>=6.13.0->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading exceptiongroup-1.2.2-py3-none-any.whl.metadata (6.6 kB)
Collecting charset-normalizer<4,>=2 (from requests->swh.core>=0.3->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading charset_normalizer-3.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (34 kB)
Collecting geomet<0.3,>=0.1 (from cassandra-driver!=3.21.0,>=3.19.0->swh.storage>=2.3.1->swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading geomet-0.2.1.post1-py3-none-any.whl.metadata (1.0 kB)
Collecting MarkupSafe>=2.0 (from Jinja2>=3.1.2->flask->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading MarkupSafe-3.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB)
Collecting swh.perfecthash>=1.3.0 (from swh.objstorage>=2.2.0->swh.storage>=2.3.1->swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading swh.perfecthash-1.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.9 kB)
Collecting packaging (from gunicorn->aiohttp-utils>=3.1.1->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading packaging-24.1-py3-none-any.whl.metadata (3.2 kB)
Collecting cffi (from swh.perfecthash>=1.3.0->swh.objstorage>=2.2.0->swh.storage>=2.3.1->swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading cffi-1.17.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.5 kB)
Collecting pycparser (from cffi->swh.perfecthash>=1.3.0->swh.objstorage>=2.2.0->swh.storage>=2.3.1->swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading pycparser-2.22-py3-none-any.whl.metadata (943 bytes)
Downloading swh.graph-6.1.0-py3-none-any.whl (183 kB)
Downloading python_json_logger-2.0.7-py3-none-any.whl (8.1 kB)
Downloading mypy_protobuf-3.2.0-py3-none-any.whl (15 kB)
Downloading protobuf-4.25.5-cp37-abi3-manylinux2014_x86_64.whl (294 kB)
Downloading swh.core-3.6.2-py3-none-any.whl (2.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 67.7 MB/s eta 0:00:00
Downloading swh.model-6.16.0-py3-none-any.whl (149 kB)
Downloading aiohttp-3.10.10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 52.1 MB/s eta 0:00:00
Downloading boto3-1.35.54-py3-none-any.whl (139 kB)
Using cached click-8.1.7-py3-none-any.whl (97 kB)
Downloading grpcio_tools-1.62.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.8/2.8 MB 74.5 MB/s eta 0:00:00
Downloading psutil-6.1.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (287 kB)
Downloading py4j-0.10.9.7-py2.py3-none-any.whl (200 kB)
Downloading swh.dataset-1.7.0-py3-none-any.whl (54 kB)
Downloading aiohappyeyeballs-2.4.3-py3-none-any.whl (14 kB)
Downloading aiohttp_utils-3.2.1-py2.py3-none-any.whl (10 kB)
Downloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Downloading async_timeout-4.0.3-py3-none-any.whl (5.7 kB)
Downloading attrs-24.2.0-py3-none-any.whl (63 kB)
Downloading attrs_strict-1.0.1-py3-none-any.whl (14 kB)
Downloading botocore-1.35.54-py3-none-any.whl (12.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.7/12.7 MB 81.0 MB/s eta 0:00:00
Downloading frozenlist-1.5.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (241 kB)
Downloading grpcio-1.67.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.9/5.9 MB 82.2 MB/s eta 0:00:00
Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
Downloading msgpack-1.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (378 kB)
Downloading multidict-6.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (124 kB)
Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Downloading s3transfer-0.10.3-py3-none-any.whl (82 kB)
Downloading sentry_sdk-2.18.0-py2.py3-none-any.whl (317 kB)
Downloading swh.journal-1.5.2-py3-none-any.whl (43 kB)
Downloading swh.storage-2.8.1-py3-none-any.whl (553 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 553.8/553.8 kB 31.8 MB/s eta 0:00:00
Downloading tenacity-9.0.0-py3-none-any.whl (28 kB)
Downloading types_protobuf-5.28.3.20241030-py3-none-any.whl (68 kB)
Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB)
Downloading yarl-1.17.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (318 kB)
Downloading backports.entry_points_selectable-1.3.0-py3-none-any.whl (6.2 kB)
Downloading blinker-1.8.2-py3-none-any.whl (9.5 kB)
Downloading Deprecated-1.2.14-py2.py3-none-any.whl (9.6 kB)
Downloading flask-3.0.3-py3-none-any.whl (101 kB)
Downloading hypothesis-6.116.0-py3-none-any.whl (468 kB)
Downloading iso8601-2.1.0-py3-none-any.whl (7.5 kB)
Downloading plyvel-1.5.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (877 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 877.8/877.8 kB 39.0 MB/s eta 0:00:00
Downloading pyorc-0.9.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.7/2.7 MB 74.5 MB/s eta 0:00:00
Downloading python_magic-0.4.27-py2.py3-none-any.whl (13 kB)
Using cached PyYAML-6.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (751 kB)
Downloading requests-2.32.3-py3-none-any.whl (64 kB)
Downloading tqdm-4.66.6-py3-none-any.whl (78 kB)
Downloading types_requests-2.32.0.20241016-py3-none-any.whl (15 kB)
Downloading cassandra_driver-3.29.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.9/3.9 MB 77.4 MB/s eta 0:00:00
Downloading certifi-2024.8.30-py3-none-any.whl (167 kB)
Downloading charset_normalizer-3.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (144 kB)
Downloading confluent_kafka-2.6.0-cp310-cp310-manylinux_2_28_x86_64.whl (3.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.9/3.9 MB 79.9 MB/s eta 0:00:00
Downloading exceptiongroup-1.2.2-py3-none-any.whl (16 kB)
Downloading idna-3.10-py3-none-any.whl (70 kB)
Downloading itsdangerous-2.2.0-py3-none-any.whl (16 kB)
Downloading jinja2-3.1.4-py3-none-any.whl (133 kB)
Downloading propcache-0.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (208 kB)
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Downloading sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
Downloading swh.counters-0.11.0-py3-none-any.whl (37 kB)
Downloading swh.objstorage-3.3.0-py3-none-any.whl (125 kB)
Using cached urllib3-2.2.3-py3-none-any.whl (126 kB)
Downloading werkzeug-3.1.2-py3-none-any.whl (224 kB)
Downloading wrapt-1.16.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (80 kB)
Downloading gunicorn-23.0.0-py3-none-any.whl (85 kB)
Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)
Downloading python_mimeparse-2.0.0-py3-none-any.whl (5.6 kB)
Downloading redis-5.2.0-py3-none-any.whl (261 kB)
Downloading geomet-0.2.1.post1-py3-none-any.whl (18 kB)
Downloading MarkupSafe-3.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (20 kB)
Downloading swh.perfecthash-1.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (259 kB)
Downloading packaging-24.1-py3-none-any.whl (53 kB)
Downloading cffi-1.17.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (446 kB)
Downloading pycparser-2.22-py3-none-any.whl (117 kB)
Building wheels for collected packages: psycopg2
  Building wheel for psycopg2 (setup.py): started
  Building wheel for psycopg2 (setup.py): finished with status 'done'
  Created wheel for psycopg2: filename=psycopg2-2.9.10-cp310-cp310-linux_x86_64.whl size=479773 sha256=acb2030b880b2f622f4c937004d3bc04fc031f874e28485fa3cf9d100e3434ec
  Stored in directory: /opt/swh/.cache/pip/wheels/51/41/e0/2912ad51b01f454d26dfb26e5cc5923874656749b9e83943a8
Successfully built psycopg2
Installing collected packages: sortedcontainers, py4j, plyvel, confluent-kafka, wrapt, urllib3, typing-extensions, types-protobuf, tqdm, tenacity, six, pyyaml, python-mimeparse, python-magic, python-json-logger, pyorc, pycparser, psycopg2, psutil, protobuf, propcache, packaging, mypy-extensions, msgpack, MarkupSafe, jmespath, itsdangerous, iso8601, idna, grpcio, frozenlist, exceptiongroup, click, charset-normalizer, certifi, blinker, backports.entry-points-selectable, attrs, async-timeout, aiohappyeyeballs, Werkzeug, types-requests, sentry-sdk, requests, redis, python-dateutil, mypy-protobuf, multidict, Jinja2, hypothesis, gunicorn, grpcio-tools, geomet, deprecated, cffi, attrs-strict, aiosignal, yarl, swh.perfecthash, swh.model, swh.core, flask, cassandra-driver, botocore, swh.journal, s3transfer, aiohttp, boto3, aiohttp-utils, swh.objstorage, swh.counters, swh.storage, swh.dataset, swh.graph
Successfully installed Jinja2-3.1.4 MarkupSafe-3.0.2 Werkzeug-3.1.2 aiohappyeyeballs-2.4.3 aiohttp-3.10.10 aiohttp-utils-3.2.1 aiosignal-1.3.1 async-timeout-4.0.3 attrs-24.2.0 attrs-strict-1.0.1 backports.entry-points-selectable-1.3.0 blinker-1.8.2 boto3-1.35.54 botocore-1.35.54 cassandra-driver-3.29.2 certifi-2024.8.30 cffi-1.17.1 charset-normalizer-3.4.0 click-8.1.7 confluent-kafka-2.6.0 deprecated-1.2.14 exceptiongroup-1.2.2 flask-3.0.3 frozenlist-1.5.0 geomet-0.2.1.post1 grpcio-1.67.1 grpcio-tools-1.62.3 gunicorn-23.0.0 hypothesis-6.116.0 idna-3.10 iso8601-2.1.0 itsdangerous-2.2.0 jmespath-1.0.1 msgpack-1.1.0 multidict-6.1.0 mypy-extensions-1.0.0 mypy-protobuf-3.2.0 packaging-24.1 plyvel-1.5.1 propcache-0.2.0 protobuf-4.25.5 psutil-6.1.0 psycopg2-2.9.10 py4j-0.10.9.7 pycparser-2.22 pyorc-0.9.0 python-dateutil-2.9.0.post0 python-json-logger-2.0.7 python-magic-0.4.27 python-mimeparse-2.0.0 pyyaml-6.0.2 redis-5.2.0 requests-2.32.3 s3transfer-0.10.3 sentry-sdk-2.18.0 six-1.16.0 sortedcontainers-2.4.0 swh.core-3.6.2 swh.counters-0.11.0 swh.dataset-1.7.0 swh.graph-6.1.0 swh.journal-1.5.2 swh.model-6.16.0 swh.objstorage-3.3.0 swh.perfecthash-1.3.2 swh.storage-2.8.1 tenacity-9.0.0 tqdm-4.66.6 types-protobuf-5.28.3.20241030 types-requests-2.32.0.20241016 typing-extensions-4.12.2 urllib3-2.2.3 wrapt-1.16.0 yarl-1.17.1
docker run --rm -v $(pwd):/src app-manager generate-frozen-requirements
0.03s user 0.02s system 0% cpu 21.970 total

[2] Generating frozen requirements with uv in ~10.5s total

tony@kessel:~/work/inria/repo/swh/sysadm-environment/swh-apps (master) $ time docker run --rm -v $(pwd):/src app-manager generate-frozen-requirements swh-graph
Collecting uv
  Downloading uv-0.4.30-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.0/14.0 MB 80.5 MB/s eta 0:00:00
Installing collected packages: uv
Successfully installed uv-0.4.30

[notice] A new release of pip is available: 23.0.1 -> 24.3.1
[notice] To update, run: python3 -m pip install --upgrade pip
Using Python 3.10.15 environment at /tmp/swh-graphaf177vlv
Resolved 3 packages in 52ms
Prepared 3 packages in 61ms
Uninstalled 2 packages in 57ms
Installed 3 packages in 31ms
 - pip==23.0.1
 + pip==24.3.1
 - setuptools==65.5.0
 + setuptools==75.3.0
 + wheel==0.44.0
Using Python 3.10.15 environment at /tmp/swh-graphaf177vlv
Resolved 75 packages in 448ms
Prepared 74 packages in 4.49s
Installed 74 packages in 127ms
 + aiohappyeyeballs==2.4.3
 + aiohttp==3.10.10
 + aiohttp-utils==3.2.1
 + aiosignal==1.3.1
 + async-timeout==4.0.3
 + attrs==24.2.0
 + attrs-strict==1.0.1
 + backports-entry-points-selectable==1.3.0
 + blinker==1.8.2
 + boto3==1.35.54
 + botocore==1.35.54
 + cassandra-driver==3.29.2
 + certifi==2024.8.30
 + cffi==1.17.1
 + charset-normalizer==3.4.0
 + click==8.1.7
 + confluent-kafka==2.6.0
 + deprecated==1.2.14
 + exceptiongroup==1.2.2
 + flask==3.0.3
 + frozenlist==1.5.0
 + geomet==0.2.1.post1
 + grpcio==1.67.1
 + grpcio-tools==1.62.3
 + gunicorn==23.0.0
 + hypothesis==6.116.0
 + idna==3.10
 + iso8601==2.1.0
 + itsdangerous==2.2.0
 + jinja2==3.1.4
 + jmespath==1.0.1
 + markupsafe==3.0.2
 + msgpack==1.1.0
 + multidict==6.1.0
 + mypy-extensions==1.0.0
 + mypy-protobuf==3.2.0
 + packaging==24.1
 + plyvel==1.5.1
 + propcache==0.2.0
 + protobuf==4.25.5
 + psutil==6.1.0
 + psycopg2==2.9.10
 + py4j==0.10.9.7
 + pycparser==2.22
 + pyorc==0.9.0
 + python-dateutil==2.9.0.post0
 + python-json-logger==2.0.7
 + python-magic==0.4.27
 + python-mimeparse==2.0.0
 + pyyaml==6.0.2
 + redis==5.2.0
 + requests==2.32.3
 + s3transfer==0.10.3
 + sentry-sdk==2.18.0
 + six==1.16.0
 + sortedcontainers==2.4.0
 + swh-core==3.6.2
 + swh-counters==0.11.0
 + swh-dataset==1.7.0
 + swh-graph==6.1.0
 + swh-journal==1.5.2
 + swh-model==6.16.0
 + swh-objstorage==3.3.0
 + swh-perfecthash==1.3.2
 + swh-storage==2.8.1
 + tenacity==9.0.0
 + tqdm==4.66.6
 + types-protobuf==5.28.3.20241030
 + types-requests==2.32.0.20241016
 + typing-extensions==4.12.2
 + urllib3==2.2.3
 + werkzeug==3.1.2
 + wrapt==1.16.0
 + yarl==1.17.1
docker run --rm -v $(pwd):/src app-manager generate-frozen-requirements
0.02s user 0.01s system 0% cpu 10.447 total

[3] requirement-frozen.txt from pip vs the one from uv

$ diff requirements-frozen.txt apps/swh-graph/requirements-frozen.txt
8c8
< backports.entry-points-selectable==1.3.0
---
> backports-entry-points-selectable==1.3.0
18c18
< Deprecated==1.2.14
---
> deprecated==1.2.14
20c20
< Flask==3.0.3
---
> flask==3.0.3
30c30
< Jinja2==3.1.4
---
> jinja2==3.1.4
32c32
< MarkupSafe==3.0.2
---
> markupsafe==3.0.2
37a38
> pip==24.3.1
50c51
< PyYAML==6.0.2
---
> pyyaml==6.0.2
54a56
> setuptools==75.3.0
57,65c59,67
< swh.core==3.6.2
< swh.counters==0.11.0
< swh.dataset==1.7.0
< swh.graph==6.1.0
< swh.journal==1.5.2
< swh.model==6.16.0
< swh.objstorage==3.3.0
< swh.perfecthash==1.3.2
< swh.storage==2.8.1
---
> swh-core==3.6.2
> swh-counters==0.11.0
> swh-dataset==1.7.0
> swh-graph==6.1.0
> swh-journal==1.5.2
> swh-model==6.16.0
> swh-objstorage==3.3.0
> swh-perfecthash==1.3.2
> swh-storage==2.8.1
70c72
< typing_extensions==4.12.2
---
> typing-extensions==4.12.2
72c74,76
< Werkzeug==3.1.2
---
> uv==0.4.30
> werkzeug==3.1.2
> wheel==0.44.0

To skip the venv step completely, we should consider using https://docs.astral.sh/uv/pip/compile/

I don't see exactly how to properly bypass the venv part in the app-manager code. I can only install uv through pip (in the venv like the other current deps). So far when I tried installing uv at the system level, uv has permission issues to install stuff system-wide after that.

But using the "uv pip compile" part (inside the current venv) without installing the requirements deps like it's done currently, I have a speed gain again (twice as fast, from 10 to 5 seconds [1]).

[1] generated frozen requirements in half the time again

time docker run --rm -v $(pwd):/src app-manager generate-frozen-requirements swh-graph
Collecting uv
  Downloading uv-0.4.30-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.0/14.0 MB 76.1 MB/s eta 0:00:00
Installing collected packages: uv
Successfully installed uv-0.4.30

[notice] A new release of pip is available: 23.0.1 -> 24.3.1
[notice] To update, run: python3 -m pip install --upgrade pip
Using Python 3.10.15 environment at /tmp/swh-graph8faurkg6
Resolved 3 packages in 47ms
Prepared 3 packages in 56ms
Uninstalled 2 packages in 54ms
Installed 3 packages in 32ms
 - pip==23.0.1
 + pip==24.3.1
 - setuptools==65.5.0
 + setuptools==75.3.0
 + wheel==0.44.0
warning: Requirements file /src/apps/swh-graph/tmp62jtha3d does not contain any dependencies
# This file was autogenerated by uv via the following command:
#    uv pip compile /src/apps/swh-graph/requirements.txt -o /src/apps/swh-graph/tmp62jtha3d
aiohappyeyeballs==2.4.3
    # via aiohttp
aiohttp==3.10.10
    # via
    #   aiohttp-utils
    #   swh-core
    #   swh-graph
aiohttp-utils==3.2.1
    # via swh-core
aiosignal==1.3.1
    # via aiohttp
async-timeout==4.0.3
    # via
    #   aiohttp
    #   redis
attrs==24.2.0
    # via
    #   aiohttp
    #   attrs-strict
    #   hypothesis
    #   swh-model
attrs-strict==1.0.1
    # via swh-model
backports-entry-points-selectable==1.3.0
    # via
    #   swh-core
    #   swh-storage
blinker==1.8.2
    # via
    #   flask
    #   swh-core
boto3==1.35.54
    # via
    #   swh-dataset
    #   swh-graph
botocore==1.35.54
    # via
    #   boto3
    #   s3transfer
cassandra-driver==3.29.2
    # via swh-storage
certifi==2024.8.30
    # via
    #   requests
    #   sentry-sdk
cffi==1.17.1
    # via swh-perfecthash
charset-normalizer==3.4.0
    # via requests
click==8.1.7
    # via
    #   flask
    #   geomet
    #   swh-core
    #   swh-dataset
    #   swh-graph
    #   swh-objstorage
    #   swh-storage
confluent-kafka==2.6.0
    # via swh-journal
deprecated==1.2.14
    # via
    #   swh-core
    #   swh-model
    #   swh-objstorage
    #   swh-storage
exceptiongroup==1.2.2
    # via hypothesis
flask==3.0.3
    # via
    #   swh-core
    #   swh-counters
    #   swh-storage
frozenlist==1.5.0
    # via
    #   aiohttp
    #   aiosignal
geomet==0.2.1.post1
    # via cassandra-driver
grpcio==1.67.1
    # via grpcio-tools
grpcio-tools==1.62.3
    # via swh-graph
gunicorn==23.0.0
    # via aiohttp-utils
hypothesis==6.116.0
    # via swh-model
idna==3.10
    # via
    #   requests
    #   yarl
iso8601==2.1.0
    # via
    #   swh-core
    #   swh-model
    #   swh-storage
itsdangerous==2.2.0
    # via flask
jinja2==3.1.4
    # via flask
jmespath==1.0.1
    # via
    #   boto3
    #   botocore
markupsafe==3.0.2
    # via
    #   jinja2
    #   werkzeug
msgpack==1.1.0
    # via
    #   swh-core
    #   swh-journal
    #   swh-objstorage
multidict==6.1.0
    # via
    #   aiohttp
    #   yarl
mypy-extensions==1.0.0
    # via swh-storage
mypy-protobuf==3.2.0
    # via swh-graph
packaging==24.1
    # via gunicorn
plyvel==1.5.1
    # via swh-dataset
propcache==0.2.0
    # via yarl
protobuf==4.25.5
    # via
    #   grpcio-tools
    #   mypy-protobuf
    #   swh-graph
psutil==6.1.0
    # via swh-graph
psycopg2==2.9.10
    # via
    #   swh-core
    #   swh-storage
py4j==0.10.9.7
    # via swh-graph
pycparser==2.22
    # via cffi
pyorc==0.9.0
    # via swh-dataset
python-dateutil==2.9.0.post0
    # via
    #   botocore
    #   swh-model
python-json-logger==2.0.7
    # via -r /src/apps/swh-graph/requirements.txt
python-magic==0.4.27
    # via swh-core
python-mimeparse==2.0.0
    # via aiohttp-utils
pyyaml==6.0.2
    # via swh-core
redis==5.2.0
    # via
    #   swh-counters
    #   swh-storage
requests==2.32.3
    # via
    #   swh-core
    #   swh-objstorage
s3transfer==0.10.3
    # via boto3
sentry-sdk==2.18.0
    # via swh-core
setuptools==75.3.0
    # via grpcio-tools
six==1.16.0
    # via
    #   geomet
    #   python-dateutil
sortedcontainers==2.4.0
    # via hypothesis
swh-core==3.6.2
    # via
    #   swh-counters
    #   swh-dataset
    #   swh-graph
    #   swh-journal
    #   swh-objstorage
    #   swh-storage
swh-counters==0.11.0
    # via swh-storage
swh-dataset==1.7.0
    # via swh-graph
swh-graph==6.1.0
    # via -r /src/apps/swh-graph/requirements.txt
swh-journal==1.5.2
    # via
    #   swh-counters
    #   swh-dataset
swh-model==6.16.0
    # via
    #   swh-dataset
    #   swh-graph
    #   swh-journal
    #   swh-objstorage
    #   swh-storage
swh-objstorage==3.3.0
    # via swh-storage
swh-perfecthash==1.3.2
    # via swh-objstorage
swh-storage==2.8.1
    # via swh-dataset
tenacity==9.0.0
    # via
    #   swh-core
    #   swh-journal
    #   swh-storage
tqdm==4.66.6
    # via swh-dataset
types-protobuf==5.28.3.20241030
    # via mypy-protobuf
types-requests==2.32.0.20241016
    # via swh-dataset
typing-extensions==4.12.2
    # via
    #   multidict
    #   swh-core
    #   swh-model
    #   swh-objstorage
    #   swh-storage
urllib3==2.2.3
    # via
    #   botocore
    #   requests
    #   sentry-sdk
    #   types-requests
werkzeug==3.1.2
    # via flask
wrapt==1.16.0
    # via deprecated
yarl==1.17.1
    # via aiohttp
Resolved 75 packages in 519ms
docker run --rm -v $(pwd):/src app-manager generate-frozen-requirements   0.01s user 0.02s system 0% cpu 5.200 total

The requirements-frozen.txt becomes a tad more verbose but it's probably a helpful minor detail [1].

It could actually help us troubleshoot since it's giving the source of the compulsed dep. (And the interesting part too is we could use the 'uv pip sync' if we unblock its use in the other docker container image).

[1] requirements-frozen.txt

# This file was autogenerated by uv via the following command:
#    uv pip compile /src/apps/swh-graph/requirements.txt -o /src/apps/swh-graph/tmp250toaks
aiohappyeyeballs==2.4.3
    # via aiohttp
aiohttp==3.10.10
    # via
    #   aiohttp-utils
    #   swh-core
    #   swh-graph
aiohttp-utils==3.2.1
    # via swh-core
aiosignal==1.3.1
    # via aiohttp
async-timeout==4.0.3
    # via
    #   aiohttp
    #   redis
attrs==24.2.0
    # via
    #   aiohttp
    #   attrs-strict
    #   hypothesis
    #   swh-model
attrs-strict==1.0.1
    # via swh-model
backports-entry-points-selectable==1.3.0
    # via
    #   swh-core
    #   swh-storage
blinker==1.8.2
    # via
    #   flask
    #   swh-core
boto3==1.35.54
    # via
    #   swh-dataset
    #   swh-graph
botocore==1.35.54
    # via
    #   boto3
    #   s3transfer
cassandra-driver==3.29.2
    # via swh-storage
certifi==2024.8.30
    # via
    #   requests
    #   sentry-sdk
cffi==1.17.1
    # via swh-perfecthash
charset-normalizer==3.4.0
    # via requests
click==8.1.7
    # via
    #   flask
    #   geomet
    #   swh-core
    #   swh-dataset
    #   swh-graph
    #   swh-objstorage
    #   swh-storage
confluent-kafka==2.6.0
    # via swh-journal
deprecated==1.2.14
    # via
    #   swh-core
    #   swh-model
    #   swh-objstorage
    #   swh-storage
exceptiongroup==1.2.2
    # via hypothesis
flask==3.0.3
    # via
    #   swh-core
    #   swh-counters
    #   swh-storage
frozenlist==1.5.0
    # via
    #   aiohttp
    #   aiosignal
geomet==0.2.1.post1
    # via cassandra-driver
grpcio==1.67.1
    # via grpcio-tools
grpcio-tools==1.62.3
    # via swh-graph
gunicorn==23.0.0
    # via aiohttp-utils
hypothesis==6.116.0
    # via swh-model
idna==3.10
    # via
    #   requests
    #   yarl
iso8601==2.1.0
    # via
    #   swh-core
    #   swh-model
    #   swh-storage
itsdangerous==2.2.0
    # via flask
jinja2==3.1.4
    # via flask
jmespath==1.0.1
    # via
    #   boto3
    #   botocore
markupsafe==3.0.2
    # via
    #   jinja2
    #   werkzeug
msgpack==1.1.0
    # via
    #   swh-core
    #   swh-journal
    #   swh-objstorage
multidict==6.1.0
    # via
    #   aiohttp
    #   yarl
mypy-extensions==1.0.0
    # via swh-storage
mypy-protobuf==3.2.0
    # via swh-graph
packaging==24.1
    # via gunicorn
plyvel==1.5.1
    # via swh-dataset
propcache==0.2.0
    # via yarl
protobuf==4.25.5
    # via
    #   grpcio-tools
    #   mypy-protobuf
    #   swh-graph
psutil==6.1.0
    # via swh-graph
psycopg2==2.9.10
    # via
    #   swh-core
    #   swh-storage
py4j==0.10.9.7
    # via swh-graph
pycparser==2.22
    # via cffi
pyorc==0.9.0
    # via swh-dataset
python-dateutil==2.9.0.post0
    # via
    #   botocore
    #   swh-model
python-json-logger==2.0.7
    # via -r /src/apps/swh-graph/requirements.txt
python-magic==0.4.27
    # via swh-core
python-mimeparse==2.0.0
    # via aiohttp-utils
pyyaml==6.0.2
    # via swh-core
redis==5.2.0
    # via
    #   swh-counters
    #   swh-storage
requests==2.32.3
    # via
    #   swh-core
    #   swh-objstorage
s3transfer==0.10.3
    # via boto3
sentry-sdk==2.18.0
    # via swh-core
setuptools==75.3.0
    # via grpcio-tools
six==1.16.0
    # via
    #   geomet
    #   python-dateutil
sortedcontainers==2.4.0
    # via hypothesis
swh-core==3.6.2
    # via
    #   swh-counters
    #   swh-dataset
    #   swh-graph
    #   swh-journal
    #   swh-objstorage
    #   swh-storage
swh-counters==0.11.0
    # via swh-storage
swh-dataset==1.7.0
    # via swh-graph
swh-graph==6.1.0
    # via -r /src/apps/swh-graph/requirements.txt
swh-journal==1.5.2
    # via
    #   swh-counters
    #   swh-dataset
swh-model==6.16.0
    # via
    #   swh-dataset
    #   swh-graph
    #   swh-journal
    #   swh-objstorage
    #   swh-storage
swh-objstorage==3.3.0
    # via swh-storage
swh-perfecthash==1.3.2
    # via swh-objstorage
swh-storage==2.8.1
    # via swh-dataset
tenacity==9.0.0
    # via
    #   swh-core
    #   swh-journal
    #   swh-storage
tqdm==4.66.6
    # via swh-dataset
types-protobuf==5.28.3.20241030
    # via mypy-protobuf
types-requests==2.32.0.20241016
    # via swh-dataset
typing-extensions==4.12.2
    # via
    #   multidict
    #   swh-core
    #   swh-model
    #   swh-objstorage
    #   swh-storage
urllib3==2.2.3
    # via
    #   botocore
    #   requests
    #   sentry-sdk
    #   types-requests
werkzeug==3.1.2
    # via flask
wrapt==1.16.0
    # via deprecated
yarl==1.17.1
    # via aiohttp

changed the description

added 1 commit

• 72f31688 - app_manager: Only compile the frozen deps without installing them

Compare with previous version

Also, I've only focused on the frozen requirements step proposal. I've failed to make it work on other container Dockerfile (since uv wants to work on venv and we don't have any there [1]).

[1] https://docs.astral.sh/uv/pip/environments/

Quoting [1]

A virtual environment is a lightweight way to isolate packages from a Python installation's environment. Unlike pip, uv requires using a virtual environment by default.

I can only install uv through pip (no system deps). So far when I tried installing uv at the system level, uv has permission issues to install stuff system-wide after that.

fwiw, that's what i'm facing when trying to change swh-graph's Dockerfile too to lift uv (permission issue):

RUN /usr/bin/python3 -m pip install --upgrade pip --break-system-packages uv
RUN ls /opt/swh/.local/bin
RUN /opt/swh/.local/bin/uv pip sync --break-system-packages --system requirements-frozen.txt

with:

Step 16/26 : ENV PATH=/opt/swh/.local/bin:$PATH
 ---> Using cache
 ---> 63362dfe01d5
Step 17/26 : RUN /usr/bin/python3 -m pip install --upgrade pip --break-system-packages uv setuptools
 ---> Running in 4fccf331bcbf
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: pip in /usr/lib/python3/dist-packages (23.0.1)
Collecting pip
  Downloading pip-24.3.1-py3-none-any.whl (1.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 21.9 MB/s eta 0:00:00
Collecting uv
  Downloading uv-0.4.30-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.0/14.0 MB 72.1 MB/s eta 0:00:00
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (66.1.1)
Collecting setuptools
  Downloading setuptools-75.3.0-py3-none-any.whl (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 49.7 MB/s eta 0:00:00
Installing collected packages: uv, setuptools, pip
Successfully installed pip-24.3.1 setuptools-75.3.0 uv-0.4.30
Removing intermediate container 4fccf331bcbf
 ---> 3d54a3715605
Step 18/26 : RUN ls /opt/swh/.local/bin
 ---> Running in 7d2b5f9784df
pip
pip3
pip3.11
uv
uvx
Removing intermediate container 7d2b5f9784df
 ---> d3d3efda8300
Step 19/26 : RUN /opt/swh/.local/bin/uv pip sync --break-system-packages --system requirements-frozen.txt
 ---> Running in 5240c13777cc
Using Python 3.10.15 environment at /usr/local
Resolved 75 packages in 438ms
Prepared 75 packages in 5.21s
error: failed to remove file `/usr/local/lib/python3.10/site-packages/_distutils_hack/__init__.py`
  Caused by: Permission denied (os error 13)                                                                       <- from what i understood, it's because there ain't any venv
The command '/bin/sh -c /opt/swh/.local/bin/uv pip sync --break-system-packages --system requirements-frozen.txt' returned a non-zero code: 2
docker build --tag xxx-graph-tryout apps/swh-graph  0.01s user 0.02s system 0% cpu 10.802 total

As a follow up to this work, there is now a tryout [1] And we do gain a bit of time too (less than here but still).

[1] !49 (merged)

added 54 commits

• 72f31688...132468a0 - 52 commits from branch master
• 15af5837 - app_manager: Use more efficient uv cli instead of pip
• 19ca4b06 - app_manager: Only compile the frozen deps without installing them

Compare with previous version

@olasd ok to land as-is and give it a spin btw?

yeah, that's worth a try!

approved this merge request

added 178 commits

• 19ca4b06...2ad46dc1 - 176 commits from branch master
• 771a3581 - app_manager: Use more efficient uv cli instead of pip
• 4aaa9654 - app_manager: Only compile the frozen deps without installing them

Compare with previous version

merged manually

marked this merge request as ready

Expectedly, it seems to do the trick, the build frozen requirements step is shorter now for the last 2 builds.

changed the description

mentioned in merge request !49 (merged)