[proposal] apps: Make docker image Install their dependencies faster (!49) · Merge requests · Platform / Infrastructure / Software Heritage virtual environment packaging manifests · GitLab

Snippets Groups Projects

Merged Antoine R. Dumont requested to merge mr/optimize-build-image-time into master 2 months ago

Feb 05, 2025

toolbox: Install python dependencies faster · 7889a4ba

Antoine R. Dumont authored 2 months ago

Current pip install [1] vs uv pip install [2].

[1]
```
$ /usr/bin/time docker build --no-cache -t swh-loader-package apps/swh-loader-package
...
Successfully tagged toolbox:latest
0.08user 0.12system 2:23.18elapsed 0%CPU (0avgtext+0avgdata 40712maxresident)k
0inputs+0outputs (0major+2589minor)pagefaults 0swaps
```

[2]
```
$ /usr/bin/time docker build --no-cache -t swh-loader-package-with-uv apps/swh-loader-package
Successfully tagged toolbox-with-uv:latest
0.09user 0.05system 1:45.10elapsed 0%CPU (0avgtext+0avgdata 38656maxresident)k
0inputs+0outputs (0major+2675minor)pagefaults 0swaps
...

```

7889a4ba

loader-package: Install python dependencies faster · 8ad3d96d

Antoine R. Dumont authored 2 months ago

Current pip install [1] vs uv pip install [2].

[1]
```
$ /usr/bin/time docker build --no-cache -t swh-loader-package apps/swh-loader-package
...
Successfully tagged swh-loader-package:latest
0.03user 0.04system 0:57.30elapsed 0%CPU (0avgtext+0avgdata 40880maxresident)k
8inputs+0outputs (0major+2588minor)pagefaults 0swaps
```

[2]
```
$ /usr/bin/time docker build --no-cache -t swh-loader-package-with-uv apps/swh-loader-package
...
Successfully tagged swh-loader-package-with-uv:latest
0.03user 0.02system 0:51.90elapsed 0%CPU (0avgtext+0avgdata 40516maxresident)k
0inputs+0outputs (0major+2554minor)pagefaults 0swaps
```

8ad3d96d

loader-svn: Install python dependencies faster · 1db963ad

Antoine R. Dumont authored 2 months ago

Current pip install [1] vs uv pip install [2].

[1]
```
$ /usr/bin/time docker build --no-cache -t swh-loader-svn apps/swh-loader-svn
...
Successfully tagged swh-loader-svn:latest
0.05user 0.05system 0:58.04elapsed 0%CPU (0avgtext+0avgdata 38948maxresident)k
0inputs+0outputs (0major+2652minor)pagefaults 0swaps
```

[2]
```
$ /usr/bin/time docker build --no-cache -t swh-loader-svn-with-uv apps/swh-loader-svn
...
Successfully tagged swh-loader-svn-with-uv:latest
0.04user 0.03system 0:46.99elapsed 0%CPU (0avgtext+0avgdata 40972maxresident)k
0inputs+0outputs (0major+2282minor)pagefaults 0swaps

1db963ad

loader-bzr: Install dependencies faster · be1f53a4

Antoine R. Dumont authored 2 months ago

This gains around 10 seconds.

[1]

```
Successfully tagged swh-loader-bzr:latest
0.05user 0.04system 0:44.04elapsed 0%CPU (0avgtext+0avgdata 39236maxresident)k
8inputs+0outputs (0major+2291minor)pagefaults 0swaps
```

[2]

```
Successfully tagged swh-loader-bzr-with-uv:latest
0.03user 0.03system 0:34.54elapsed 0%CPU (0avgtext+0avgdata 40980maxresident)k
0inputs+0outputs (0major+2240minor)pagefaults 0swaps
```

be1f53a4

loader-savecodenow: Install dependencies faster · 72af3192

Antoine R. Dumont authored 2 months ago

This gains around 45 seconds. Unsurprisingly the gain is a bit more
significant since there is a lot of python dependencies in this application.

[1] master
```
$ /usr/bin/time docker build --no-cache -t swh-loader-savecodenow apps/swh-loader-savecodenow

Successfully tagged swh-loader-savecodenow:latest
0.10user 0.05system 1:37.69elapsed 0%CPU (0avgtext+0avgdata 40896maxresident)k
8inputs+0outputs (0major+2645minor)pagefaults 0swaps
```

[2] with improvments
```
$ /usr/bin/time docker build --no-cache -t swh-loader-savecodenow-with-uv apps/swh-loader-savecodenow
...
Successfully tagged swh-loader-savecodenow-with-uv:latest
0.06user 0.05system 0:55.73elapsed 0%CPU (0avgtext+0avgdata 40932maxresident)k
0inputs+0outputs (0major+2587minor)pagefaults 0swaps
```

72af3192

deposit: Install dependencies faster · 04e60fd9

Antoine R. Dumont authored 2 months ago

This gains around 20 seconds. Unsurprisingly the gain is a bit more
significant since there is a lot of dependencies for that kind of frontend
applications.

[1] master
```
$ /usr/bin/time docker build --no-cache -t swh-deposit apps/swh-deposit
...
Successfully tagged swh-deposit:latest
0.11user 0.07system 0:49.05elapsed 0%CPU (0avgtext+0avgdata 38876maxresident)k
8inputs+0outputs (0major+2568minor)pagefaults 0swaps
```

*** uv
```
Successfully tagged swh-deposit-with-uv:latest
0.02user 0.02system 0:28.45elapsed 0%CPU (0avgtext+0avgdata 40412maxresident)k
0inputs+0outputs (0major+2471minor)pagefaults 0swaps
```

04e60fd9

graph: Install dependencies faster · 732ba8f8

Antoine R. Dumont authored 2 months ago

This gains only 4 seconds for the graph. Unsurprisingly the gain is less
significant for the graph as most of the time is spent in installing debian
package or crates dependencies (with lots of compilation).

[1] master

```
$ /usr/bin/time docker build --no-cache -t swh-graph apps/swh-graph
...
Successfully tagged swh-graph:latest
0.23user 0.21system 6:16.24elapsed 0%CPU (0avgtext+0avgdata 41756maxresident)k
0inputs+0outputs (0major+2669minor)pagefaults 0swaps
```

[2]
```
$ /usr/bin/time docker build --no-cache -t swh-graph-with-uv apps/swh-graph
...
Successfully tagged swh-graph-with-uv:latest
0.20user 0.21system 6:12.39elapsed 0%CPU (0avgtext+0avgdata 38716maxresident)k
0inputs+0outputs (0major+2782minor)pagefaults 0swaps
```

732ba8f8

graphql: Install dependencies faster · 2045b62c

Antoine R. Dumont authored 2 months ago

This gains around 6 to 7 seconds for this application only. It's probably a
lot more for some packages with a high number of dependencies.

In any case, if we generalize this, we can gain some minutes when we have to
build lots of images (e.g. from any tag set on swh-storage, swh-objstorage,
swh-core, swh-scheduler, swh-model, etc...).

[1] master
```
/usr/bin/time docker build --no-cache -t swh-graphql apps/swh-graphql

Successfully tagged swh-graphql:latest
0.02user 0.06system 0:33.23elapsed 0%CPU (0avgtext+0avgdata 41112maxresident)k
0inputs+0outputs (0major+2522minor)pagefaults 0swaps
Successfully tagged swh-graphql:latest
0.05user 0.02system 0:32.30elapsed 0%CPU (0avgtext+0avgdata 40768maxresident)k
0inputs+0outputs (0major+2583minor)pagefaults 0swaps
Successfully tagged swh-graphql:latest
0.03user 0.04system 0:32.61elapsed 0%CPU (0avgtext+0avgdata 38588maxresident)k
0inputs+0outputs (0major+2562minor)pagefaults 0swaps
```

[2] dev branch
```
/usr/bin/time docker build --no-cache -t swh-graphql-with-uv apps/swh-graphql

Successfully tagged swh-graphql-with-uv:latest
0.01user 0.03system 0:26.56elapsed 0%CPU (0avgtext+0avgdata 38856maxresident)k
0inputs+0outputs (0major+2570minor)pagefaults 0swaps
Successfully tagged swh-graphql-with-uv:latest
0.03user 0.01system 0:26.87elapsed 0%CPU (0avgtext+0avgdata 39852maxresident)k
0inputs+0outputs (0major+2471minor)pagefaults 0swaps
Successfully tagged swh-graphql-with-uv:latest
0.02user 0.02system 0:25.86elapsed 0%CPU (0avgtext+0avgdata 39524maxresident)k
0inputs+0outputs (0major+2268minor)pagefaults 0swaps
```

2045b62c