app_manager: Use more efficient uv cli instead of pip

Review changes
Download
Patches
Plain diff

Merged app_manager: Use more efficient uv cli instead of pip

mr/optimize-requirements-frozen-computations-tryout into master

Overview 8
Commits 2
Pipelines 2
Changes 1

Merged Antoine R. Dumont requested to merge mr/optimize-requirements-frozen-computations-tryout into master 4 months ago

Overview 8
Commits 2
Pipelines 2
Changes 1

It's a proposal out of nico's yesterday remark on making that step faster. It's not looking up the cache part.

It's more efficient and faster for computing the requirements (twice as fast with the graph deps) [1] vs [2]

Docker layers were already cached so the time comparison is viable.

Also, I've only focused on the frozen requirements step proposal. I've failed to make it work on other container Dockerfile (since uv wants to work on venv and we don't have any there).

Note that the requirements-frozen.txt generated by uv is slightly different (while still compatible afaict) [3]

[1] Generate frozen requirements with pip in ~22s total

tony@kessel:~/work/inria/repo/swh/sysadm-environment/swh-apps (master) $ time docker run --rm -v $(pwd):/src app-manager generate-frozen-requirements swh-graph
Requirement already satisfied: pip in /tmp/swh-graphzwxbv4a7/lib/python3.10/site-packages (23.0.1)
Collecting pip
  Using cached pip-24.3.1-py3-none-any.whl (1.8 MB)
Requirement already satisfied: setuptools in /tmp/swh-graphzwxbv4a7/lib/python3.10/site-packages (65.5.0)
Collecting setuptools
  Using cached setuptools-75.3.0-py3-none-any.whl (1.3 MB)
Collecting wheel
  Downloading wheel-0.44.0-py3-none-any.whl (67 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67.1/67.1 kB 10.5 MB/s eta 0:00:00
Installing collected packages: wheel, setuptools, pip
  Attempting uninstall: setuptools
    Found existing installation: setuptools 65.5.0
    Uninstalling setuptools-65.5.0:
      Successfully uninstalled setuptools-65.5.0
  Attempting uninstall: pip
    Found existing installation: pip 23.0.1
    Uninstalling pip-23.0.1:
      Successfully uninstalled pip-23.0.1
Successfully installed pip-24.3.1 setuptools-75.3.0 wheel-0.44.0
Collecting swh.graph (from -r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading swh.graph-6.1.0-py3-none-any.whl.metadata (3.9 kB)
Collecting python-json-logger (from -r /src/apps/swh-graph/requirements.txt (line 2))
  Downloading python_json_logger-2.0.7-py3-none-any.whl.metadata (6.5 kB)
Collecting aiohttp (from swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading aiohttp-3.10.10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.6 kB)
Collecting boto3 (from swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading boto3-1.35.54-py3-none-any.whl.metadata (6.7 kB)
Collecting click (from swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting py4j (from swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading py4j-0.10.9.7-py2.py3-none-any.whl.metadata (1.5 kB)
Collecting psutil (from swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading psutil-6.1.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (22 kB)
Collecting protobuf<5.26.0,>=4.21.11 (from swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading protobuf-4.25.5-cp37-abi3-manylinux2014_x86_64.whl.metadata (541 bytes)
Collecting grpcio-tools (from swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading grpcio_tools-1.67.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
Collecting mypy-protobuf<3.3 (from swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading mypy_protobuf-3.2.0-py3-none-any.whl.metadata (493 bytes)
Collecting swh.core>=0.3 (from swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading swh.core-3.6.2-py3-none-any.whl.metadata (3.7 kB)
Collecting swh.model>=6.13.0 (from swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading swh.model-6.16.0-py3-none-any.whl.metadata (2.9 kB)
Collecting swh.dataset (from swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading swh.dataset-1.7.0-py3-none-any.whl.metadata (2.0 kB)
Collecting types-protobuf>=3.19.5 (from mypy-protobuf<3.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading types_protobuf-5.28.3.20241030-py3-none-any.whl.metadata (2.3 kB)
Collecting backports.entry-points-selectable (from swh.core>=0.3->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading backports.entry_points_selectable-1.3.0-py3-none-any.whl.metadata (4.1 kB)
Collecting deprecated (from swh.core>=0.3->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading Deprecated-1.2.14-py2.py3-none-any.whl.metadata (5.4 kB)
Collecting python-magic (from swh.core>=0.3->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading python_magic-0.4.27-py2.py3-none-any.whl.metadata (5.8 kB)
Collecting pyyaml (from swh.core>=0.3->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Using cached PyYAML-6.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB)
Collecting requests (from swh.core>=0.3->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)
Collecting sentry-sdk>=2 (from swh.core>=0.3->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading sentry_sdk-2.18.0-py2.py3-none-any.whl.metadata (9.9 kB)
Collecting tenacity>=8.4.2 (from swh.core>=0.3->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading tenacity-9.0.0-py3-none-any.whl.metadata (1.2 kB)
Collecting aiohttp-utils>=3.1.1 (from swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading aiohttp_utils-3.2.1-py2.py3-none-any.whl.metadata (2.6 kB)
Collecting blinker (from swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading blinker-1.8.2-py3-none-any.whl.metadata (1.6 kB)
Collecting flask (from swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading flask-3.0.3-py3-none-any.whl.metadata (3.2 kB)
Collecting iso8601 (from swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading iso8601-2.1.0-py3-none-any.whl.metadata (3.7 kB)
Collecting msgpack>=1.0.0 (from swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading msgpack-1.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (8.4 kB)
Collecting attrs!=21.1.0 (from swh.model>=6.13.0->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading attrs-24.2.0-py3-none-any.whl.metadata (11 kB)
Collecting attrs-strict>=0.0.7 (from swh.model>=6.13.0->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading attrs_strict-1.0.1-py3-none-any.whl.metadata (6.3 kB)
Collecting hypothesis (from swh.model>=6.13.0->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading hypothesis-6.116.0-py3-none-any.whl.metadata (6.0 kB)
Collecting python-dateutil (from swh.model>=6.13.0->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting typing-extensions (from swh.model>=6.13.0->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
Collecting aiohappyeyeballs>=2.3.0 (from aiohttp->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading aiohappyeyeballs-2.4.3-py3-none-any.whl.metadata (6.1 kB)
Collecting aiosignal>=1.1.2 (from aiohttp->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading aiosignal-1.3.1-py3-none-any.whl.metadata (4.0 kB)
Collecting frozenlist>=1.1.1 (from aiohttp->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading frozenlist-1.5.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (13 kB)
Collecting multidict<7.0,>=4.5 (from aiohttp->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading multidict-6.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.0 kB)
Collecting yarl<2.0,>=1.12.0 (from aiohttp->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading yarl-1.17.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (64 kB)
Collecting async-timeout<5.0,>=4.0 (from aiohttp->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading async_timeout-4.0.3-py3-none-any.whl.metadata (4.2 kB)
Collecting botocore<1.36.0,>=1.35.54 (from boto3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading botocore-1.35.54-py3-none-any.whl.metadata (5.7 kB)
Collecting jmespath<2.0.0,>=0.7.1 (from boto3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading jmespath-1.0.1-py3-none-any.whl.metadata (7.6 kB)
Collecting s3transfer<0.11.0,>=0.10.0 (from boto3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading s3transfer-0.10.3-py3-none-any.whl.metadata (1.7 kB)
INFO: pip is looking at multiple versions of grpcio-tools to determine which version is compatible with other requirements. This could take a while.
Collecting grpcio-tools (from swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading grpcio_tools-1.67.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
  Downloading grpcio_tools-1.66.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
  Downloading grpcio_tools-1.66.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
  Downloading grpcio_tools-1.66.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
  Downloading grpcio_tools-1.65.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
  Downloading grpcio_tools-1.65.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
  Downloading grpcio_tools-1.65.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
INFO: pip is still looking at multiple versions of grpcio-tools to determine which version is compatible with other requirements. This could take a while.
  Downloading grpcio_tools-1.65.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
  Downloading grpcio_tools-1.64.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
  Downloading grpcio_tools-1.64.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
  Downloading grpcio_tools-1.64.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
  Downloading grpcio_tools-1.63.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
  Downloading grpcio_tools-1.63.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.3 kB)
  Downloading grpcio_tools-1.62.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.2 kB)
Collecting grpcio>=1.62.3 (from grpcio-tools->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading grpcio-1.67.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.9 kB)
Requirement already satisfied: setuptools in /tmp/swh-graphzwxbv4a7/lib/python3.10/site-packages (from grpcio-tools->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1)) (75.3.0)
Collecting tqdm (from swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading tqdm-4.66.6-py3-none-any.whl.metadata (57 kB)
Collecting plyvel (from swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading plyvel-1.5.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB)
Collecting swh.journal>=0.9 (from swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading swh.journal-1.5.2-py3-none-any.whl.metadata (1.9 kB)
Collecting swh.storage>=2.3.1 (from swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading swh.storage-2.8.1-py3-none-any.whl.metadata (8.6 kB)
Collecting types-requests (from swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading types_requests-2.32.0.20241016-py3-none-any.whl.metadata (1.9 kB)
Collecting pyorc (from swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading pyorc-0.9.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.8 kB)
Collecting python-mimeparse (from aiohttp-utils>=3.1.1->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading python_mimeparse-2.0.0-py3-none-any.whl.metadata (3.1 kB)
Collecting gunicorn (from aiohttp-utils>=3.1.1->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading gunicorn-23.0.0-py3-none-any.whl.metadata (4.4 kB)
Collecting urllib3!=2.2.0,<3,>=1.25.4 (from botocore<1.36.0,>=1.35.54->boto3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Using cached urllib3-2.2.3-py3-none-any.whl.metadata (6.5 kB)
Collecting six>=1.5 (from python-dateutil->swh.model>=6.13.0->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB)
Collecting certifi (from sentry-sdk>=2->swh.core>=0.3->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading certifi-2024.8.30-py3-none-any.whl.metadata (2.2 kB)
Collecting confluent-kafka!=2.1.0 (from swh.journal>=0.9->swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading confluent_kafka-2.6.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (2.3 kB)
Collecting cassandra-driver!=3.21.0,>=3.19.0 (from swh.storage>=2.3.1->swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading cassandra_driver-3.29.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.2 kB)
Collecting mypy-extensions (from swh.storage>=2.3.1->swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading mypy_extensions-1.0.0-py3-none-any.whl.metadata (1.1 kB)
Collecting psycopg2 (from swh.storage>=2.3.1->swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading psycopg2-2.9.10.tar.gz (385 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting redis (from swh.storage>=2.3.1->swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading redis-5.2.0-py3-none-any.whl.metadata (9.1 kB)
Collecting swh.counters>=v0.8.0 (from swh.storage>=2.3.1->swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading swh.counters-0.11.0-py3-none-any.whl.metadata (1.9 kB)
Collecting swh.objstorage>=2.2.0 (from swh.storage>=2.3.1->swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading swh.objstorage-3.3.0-py3-none-any.whl.metadata (4.5 kB)
Collecting idna>=2.0 (from yarl<2.0,>=1.12.0->aiohttp->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading idna-3.10-py3-none-any.whl.metadata (10 kB)
Collecting propcache>=0.2.0 (from yarl<2.0,>=1.12.0->aiohttp->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading propcache-0.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.7 kB)
Collecting wrapt<2,>=1.10 (from deprecated->swh.core>=0.3->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading wrapt-1.16.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Collecting Werkzeug>=3.0.0 (from flask->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading werkzeug-3.1.2-py3-none-any.whl.metadata (3.7 kB)
Collecting Jinja2>=3.1.2 (from flask->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading jinja2-3.1.4-py3-none-any.whl.metadata (2.6 kB)
Collecting itsdangerous>=2.1.2 (from flask->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading itsdangerous-2.2.0-py3-none-any.whl.metadata (1.9 kB)
Collecting sortedcontainers<3.0.0,>=2.1.0 (from hypothesis->swh.model>=6.13.0->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading sortedcontainers-2.4.0-py2.py3-none-any.whl.metadata (10 kB)
Collecting exceptiongroup>=1.0.0 (from hypothesis->swh.model>=6.13.0->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading exceptiongroup-1.2.2-py3-none-any.whl.metadata (6.6 kB)
Collecting charset-normalizer<4,>=2 (from requests->swh.core>=0.3->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading charset_normalizer-3.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (34 kB)
Collecting geomet<0.3,>=0.1 (from cassandra-driver!=3.21.0,>=3.19.0->swh.storage>=2.3.1->swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading geomet-0.2.1.post1-py3-none-any.whl.metadata (1.0 kB)
Collecting MarkupSafe>=2.0 (from Jinja2>=3.1.2->flask->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading MarkupSafe-3.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB)
Collecting swh.perfecthash>=1.3.0 (from swh.objstorage>=2.2.0->swh.storage>=2.3.1->swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading swh.perfecthash-1.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.9 kB)
Collecting packaging (from gunicorn->aiohttp-utils>=3.1.1->swh.core[http]>=0.3->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading packaging-24.1-py3-none-any.whl.metadata (3.2 kB)
Collecting cffi (from swh.perfecthash>=1.3.0->swh.objstorage>=2.2.0->swh.storage>=2.3.1->swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading cffi-1.17.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.5 kB)
Collecting pycparser (from cffi->swh.perfecthash>=1.3.0->swh.objstorage>=2.2.0->swh.storage>=2.3.1->swh.dataset->swh.graph->-r /src/apps/swh-graph/requirements.txt (line 1))
  Downloading pycparser-2.22-py3-none-any.whl.metadata (943 bytes)
Downloading swh.graph-6.1.0-py3-none-any.whl (183 kB)
Downloading python_json_logger-2.0.7-py3-none-any.whl (8.1 kB)
Downloading mypy_protobuf-3.2.0-py3-none-any.whl (15 kB)
Downloading protobuf-4.25.5-cp37-abi3-manylinux2014_x86_64.whl (294 kB)
Downloading swh.core-3.6.2-py3-none-any.whl (2.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 67.7 MB/s eta 0:00:00
Downloading swh.model-6.16.0-py3-none-any.whl (149 kB)
Downloading aiohttp-3.10.10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 52.1 MB/s eta 0:00:00
Downloading boto3-1.35.54-py3-none-any.whl (139 kB)
Using cached click-8.1.7-py3-none-any.whl (97 kB)
Downloading grpcio_tools-1.62.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.8/2.8 MB 74.5 MB/s eta 0:00:00
Downloading psutil-6.1.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (287 kB)
Downloading py4j-0.10.9.7-py2.py3-none-any.whl (200 kB)
Downloading swh.dataset-1.7.0-py3-none-any.whl (54 kB)
Downloading aiohappyeyeballs-2.4.3-py3-none-any.whl (14 kB)
Downloading aiohttp_utils-3.2.1-py2.py3-none-any.whl (10 kB)
Downloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Downloading async_timeout-4.0.3-py3-none-any.whl (5.7 kB)
Downloading attrs-24.2.0-py3-none-any.whl (63 kB)
Downloading attrs_strict-1.0.1-py3-none-any.whl (14 kB)
Downloading botocore-1.35.54-py3-none-any.whl (12.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.7/12.7 MB 81.0 MB/s eta 0:00:00
Downloading frozenlist-1.5.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (241 kB)
Downloading grpcio-1.67.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.9/5.9 MB 82.2 MB/s eta 0:00:00
Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
Downloading msgpack-1.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (378 kB)
Downloading multidict-6.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (124 kB)
Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Downloading s3transfer-0.10.3-py3-none-any.whl (82 kB)
Downloading sentry_sdk-2.18.0-py2.py3-none-any.whl (317 kB)
Downloading swh.journal-1.5.2-py3-none-any.whl (43 kB)
Downloading swh.storage-2.8.1-py3-none-any.whl (553 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 553.8/553.8 kB 31.8 MB/s eta 0:00:00
Downloading tenacity-9.0.0-py3-none-any.whl (28 kB)
Downloading types_protobuf-5.28.3.20241030-py3-none-any.whl (68 kB)
Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB)
Downloading yarl-1.17.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (318 kB)
Downloading backports.entry_points_selectable-1.3.0-py3-none-any.whl (6.2 kB)
Downloading blinker-1.8.2-py3-none-any.whl (9.5 kB)
Downloading Deprecated-1.2.14-py2.py3-none-any.whl (9.6 kB)
Downloading flask-3.0.3-py3-none-any.whl (101 kB)
Downloading hypothesis-6.116.0-py3-none-any.whl (468 kB)
Downloading iso8601-2.1.0-py3-none-any.whl (7.5 kB)
Downloading plyvel-1.5.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (877 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 877.8/877.8 kB 39.0 MB/s eta 0:00:00
Downloading pyorc-0.9.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.7/2.7 MB 74.5 MB/s eta 0:00:00
Downloading python_magic-0.4.27-py2.py3-none-any.whl (13 kB)
Using cached PyYAML-6.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (751 kB)
Downloading requests-2.32.3-py3-none-any.whl (64 kB)
Downloading tqdm-4.66.6-py3-none-any.whl (78 kB)
Downloading types_requests-2.32.0.20241016-py3-none-any.whl (15 kB)
Downloading cassandra_driver-3.29.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.9/3.9 MB 77.4 MB/s eta 0:00:00
Downloading certifi-2024.8.30-py3-none-any.whl (167 kB)
Downloading charset_normalizer-3.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (144 kB)
Downloading confluent_kafka-2.6.0-cp310-cp310-manylinux_2_28_x86_64.whl (3.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.9/3.9 MB 79.9 MB/s eta 0:00:00
Downloading exceptiongroup-1.2.2-py3-none-any.whl (16 kB)
Downloading idna-3.10-py3-none-any.whl (70 kB)
Downloading itsdangerous-2.2.0-py3-none-any.whl (16 kB)
Downloading jinja2-3.1.4-py3-none-any.whl (133 kB)
Downloading propcache-0.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (208 kB)
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Downloading sortedcontainers-2.4.0-py2.py3-none-any.whl (29 kB)
Downloading swh.counters-0.11.0-py3-none-any.whl (37 kB)
Downloading swh.objstorage-3.3.0-py3-none-any.whl (125 kB)
Using cached urllib3-2.2.3-py3-none-any.whl (126 kB)
Downloading werkzeug-3.1.2-py3-none-any.whl (224 kB)
Downloading wrapt-1.16.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (80 kB)
Downloading gunicorn-23.0.0-py3-none-any.whl (85 kB)
Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)
Downloading python_mimeparse-2.0.0-py3-none-any.whl (5.6 kB)
Downloading redis-5.2.0-py3-none-any.whl (261 kB)
Downloading geomet-0.2.1.post1-py3-none-any.whl (18 kB)
Downloading MarkupSafe-3.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (20 kB)
Downloading swh.perfecthash-1.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (259 kB)
Downloading packaging-24.1-py3-none-any.whl (53 kB)
Downloading cffi-1.17.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (446 kB)
Downloading pycparser-2.22-py3-none-any.whl (117 kB)
Building wheels for collected packages: psycopg2
  Building wheel for psycopg2 (setup.py): started
  Building wheel for psycopg2 (setup.py): finished with status 'done'
  Created wheel for psycopg2: filename=psycopg2-2.9.10-cp310-cp310-linux_x86_64.whl size=479773 sha256=acb2030b880b2f622f4c937004d3bc04fc031f874e28485fa3cf9d100e3434ec
  Stored in directory: /opt/swh/.cache/pip/wheels/51/41/e0/2912ad51b01f454d26dfb26e5cc5923874656749b9e83943a8
Successfully built psycopg2
Installing collected packages: sortedcontainers, py4j, plyvel, confluent-kafka, wrapt, urllib3, typing-extensions, types-protobuf, tqdm, tenacity, six, pyyaml, python-mimeparse, python-magic, python-json-logger, pyorc, pycparser, psycopg2, psutil, protobuf, propcache, packaging, mypy-extensions, msgpack, MarkupSafe, jmespath, itsdangerous, iso8601, idna, grpcio, frozenlist, exceptiongroup, click, charset-normalizer, certifi, blinker, backports.entry-points-selectable, attrs, async-timeout, aiohappyeyeballs, Werkzeug, types-requests, sentry-sdk, requests, redis, python-dateutil, mypy-protobuf, multidict, Jinja2, hypothesis, gunicorn, grpcio-tools, geomet, deprecated, cffi, attrs-strict, aiosignal, yarl, swh.perfecthash, swh.model, swh.core, flask, cassandra-driver, botocore, swh.journal, s3transfer, aiohttp, boto3, aiohttp-utils, swh.objstorage, swh.counters, swh.storage, swh.dataset, swh.graph
Successfully installed Jinja2-3.1.4 MarkupSafe-3.0.2 Werkzeug-3.1.2 aiohappyeyeballs-2.4.3 aiohttp-3.10.10 aiohttp-utils-3.2.1 aiosignal-1.3.1 async-timeout-4.0.3 attrs-24.2.0 attrs-strict-1.0.1 backports.entry-points-selectable-1.3.0 blinker-1.8.2 boto3-1.35.54 botocore-1.35.54 cassandra-driver-3.29.2 certifi-2024.8.30 cffi-1.17.1 charset-normalizer-3.4.0 click-8.1.7 confluent-kafka-2.6.0 deprecated-1.2.14 exceptiongroup-1.2.2 flask-3.0.3 frozenlist-1.5.0 geomet-0.2.1.post1 grpcio-1.67.1 grpcio-tools-1.62.3 gunicorn-23.0.0 hypothesis-6.116.0 idna-3.10 iso8601-2.1.0 itsdangerous-2.2.0 jmespath-1.0.1 msgpack-1.1.0 multidict-6.1.0 mypy-extensions-1.0.0 mypy-protobuf-3.2.0 packaging-24.1 plyvel-1.5.1 propcache-0.2.0 protobuf-4.25.5 psutil-6.1.0 psycopg2-2.9.10 py4j-0.10.9.7 pycparser-2.22 pyorc-0.9.0 python-dateutil-2.9.0.post0 python-json-logger-2.0.7 python-magic-0.4.27 python-mimeparse-2.0.0 pyyaml-6.0.2 redis-5.2.0 requests-2.32.3 s3transfer-0.10.3 sentry-sdk-2.18.0 six-1.16.0 sortedcontainers-2.4.0 swh.core-3.6.2 swh.counters-0.11.0 swh.dataset-1.7.0 swh.graph-6.1.0 swh.journal-1.5.2 swh.model-6.16.0 swh.objstorage-3.3.0 swh.perfecthash-1.3.2 swh.storage-2.8.1 tenacity-9.0.0 tqdm-4.66.6 types-protobuf-5.28.3.20241030 types-requests-2.32.0.20241016 typing-extensions-4.12.2 urllib3-2.2.3 wrapt-1.16.0 yarl-1.17.1
docker run --rm -v $(pwd):/src app-manager generate-frozen-requirements
0.03s user 0.02s system 0% cpu 21.970 total

[2] Generating frozen requirements with uv in ~10.5s total

tony@kessel:~/work/inria/repo/swh/sysadm-environment/swh-apps (master) $ time docker run --rm -v $(pwd):/src app-manager generate-frozen-requirements swh-graph
Collecting uv
  Downloading uv-0.4.30-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.0/14.0 MB 80.5 MB/s eta 0:00:00
Installing collected packages: uv
Successfully installed uv-0.4.30

[notice] A new release of pip is available: 23.0.1 -> 24.3.1
[notice] To update, run: python3 -m pip install --upgrade pip
Using Python 3.10.15 environment at /tmp/swh-graphaf177vlv
Resolved 3 packages in 52ms
Prepared 3 packages in 61ms
Uninstalled 2 packages in 57ms
Installed 3 packages in 31ms
 - pip==23.0.1
 + pip==24.3.1
 - setuptools==65.5.0
 + setuptools==75.3.0
 + wheel==0.44.0
Using Python 3.10.15 environment at /tmp/swh-graphaf177vlv
Resolved 75 packages in 448ms
Prepared 74 packages in 4.49s
Installed 74 packages in 127ms
 + aiohappyeyeballs==2.4.3
 + aiohttp==3.10.10
 + aiohttp-utils==3.2.1
 + aiosignal==1.3.1
 + async-timeout==4.0.3
 + attrs==24.2.0
 + attrs-strict==1.0.1
 + backports-entry-points-selectable==1.3.0
 + blinker==1.8.2
 + boto3==1.35.54
 + botocore==1.35.54
 + cassandra-driver==3.29.2
 + certifi==2024.8.30
 + cffi==1.17.1
 + charset-normalizer==3.4.0
 + click==8.1.7
 + confluent-kafka==2.6.0
 + deprecated==1.2.14
 + exceptiongroup==1.2.2
 + flask==3.0.3
 + frozenlist==1.5.0
 + geomet==0.2.1.post1
 + grpcio==1.67.1
 + grpcio-tools==1.62.3
 + gunicorn==23.0.0
 + hypothesis==6.116.0
 + idna==3.10
 + iso8601==2.1.0
 + itsdangerous==2.2.0
 + jinja2==3.1.4
 + jmespath==1.0.1
 + markupsafe==3.0.2
 + msgpack==1.1.0
 + multidict==6.1.0
 + mypy-extensions==1.0.0
 + mypy-protobuf==3.2.0
 + packaging==24.1
 + plyvel==1.5.1
 + propcache==0.2.0
 + protobuf==4.25.5
 + psutil==6.1.0
 + psycopg2==2.9.10
 + py4j==0.10.9.7
 + pycparser==2.22
 + pyorc==0.9.0
 + python-dateutil==2.9.0.post0
 + python-json-logger==2.0.7
 + python-magic==0.4.27
 + python-mimeparse==2.0.0
 + pyyaml==6.0.2
 + redis==5.2.0
 + requests==2.32.3
 + s3transfer==0.10.3
 + sentry-sdk==2.18.0
 + six==1.16.0
 + sortedcontainers==2.4.0
 + swh-core==3.6.2
 + swh-counters==0.11.0
 + swh-dataset==1.7.0
 + swh-graph==6.1.0
 + swh-journal==1.5.2
 + swh-model==6.16.0
 + swh-objstorage==3.3.0
 + swh-perfecthash==1.3.2
 + swh-storage==2.8.1
 + tenacity==9.0.0
 + tqdm==4.66.6
 + types-protobuf==5.28.3.20241030
 + types-requests==2.32.0.20241016
 + typing-extensions==4.12.2
 + urllib3==2.2.3
 + werkzeug==3.1.2
 + wrapt==1.16.0
 + yarl==1.17.1
docker run --rm -v $(pwd):/src app-manager generate-frozen-requirements
0.02s user 0.01s system 0% cpu 10.447 total

[3] requirement-frozen.txt from pip vs the one from uv

$ diff requirements-frozen.txt apps/swh-graph/requirements-frozen.txt
8c8
< backports.entry-points-selectable==1.3.0
---
> backports-entry-points-selectable==1.3.0
18c18
< Deprecated==1.2.14
---
> deprecated==1.2.14
20c20
< Flask==3.0.3
---
> flask==3.0.3
30c30
< Jinja2==3.1.4
---
> jinja2==3.1.4
32c32
< MarkupSafe==3.0.2
---
> markupsafe==3.0.2
37a38
> pip==24.3.1
50c51
< PyYAML==6.0.2
---
> pyyaml==6.0.2
54a56
> setuptools==75.3.0
57,65c59,67
< swh.core==3.6.2
< swh.counters==0.11.0
< swh.dataset==1.7.0
< swh.graph==6.1.0
< swh.journal==1.5.2
< swh.model==6.16.0
< swh.objstorage==3.3.0
< swh.perfecthash==1.3.2
< swh.storage==2.8.1
---
> swh-core==3.6.2
> swh-counters==0.11.0
> swh-dataset==1.7.0
> swh-graph==6.1.0
> swh-journal==1.5.2
> swh-model==6.16.0
> swh-objstorage==3.3.0
> swh-perfecthash==1.3.2
> swh-storage==2.8.1
70c72
< typing_extensions==4.12.2
---
> typing-extensions==4.12.2
72c74,76
< Werkzeug==3.1.2
---
> uv==0.4.30
> werkzeug==3.1.2
> wheel==0.44.0