jobs: Optimize rust compilation steps

Use cargo cache and do not compile in release mode for swh-environment jobs.

Setup use of sccache in Jenkins production instance in every steps compiling rust code.

Jenkins job jenkins-tools/swh-jenkins-jobs-builder #433 succeeded in 7.9 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

In addition to prepare-cargo-cache (which only gives us a cache for the crates code), this should populate the sccache-related environment variables to get caching for the compiled crates:

CARGO_WRAPPER=sccache, CARGO_INCREMENTAL=false and pulling SCCACHE_REDIS_{ENDPOINT,PASSWORD} from the jenkins credentials store.

I'm not sure what dropping --release achieves but it feels a bit wrong.

In addition to prepare-cargo-cache (which only gives us a cache for the crates code), this should populate the sccache-related environment variables to get caching for the compiled crates:

Ack, I thought both achieved the same.

I'm not sure what dropping --release achieves but it feels a bit wrong.

It disables rust compiler optimizations which decreases compilation time and CPU usage significantly. That pipeline only run code quality checks and Python tests so we do not need rust release builds (but we still need compiled binaries for the swh-graph Python tests).

CARGO_WRAPPER=sccache

Should not it be RUSTC_WRAPPER as explained in sccache README ? There is no reference to CARGO_WRAPPER in rust documentation.

yeah, it should, thanks

confirmed as I now have this error when changing from CARGO_WRAPPER to RUSTC_WRAPPER when testing locally:

error: could not execute process `sccache /home/jenkins/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/bin/rustc -vV` (never executed)

likely related to my local docker image container-registry.softwareheritage.org/swh/infra/ci-cd/swh-jenkins-dockerfiles/tox not up to date, currently pulling it.

indeed, works much better once the docker image up to date.

added 2 commits

• 606ffb44 - jobs/swh-environment: Optimize rust compilation in production jenkins
• aed665e0 - jobs/includes/stage-rust-*: Setup use of sccache in production jenkins

Compare with previous version

Jenkins job jenkins-tools/swh-jenkins-jobs-builder #434 succeeded in 6.5 sec.
See Console Output, Blue Ocean and Coverage Report for more details.

changed title from jobs/swh-environment: Optimize rust compilation step to jobs: Optimize rust compilation steps

changed the description

approved this merge request

Thanks for going the rest of the way for sccache!

I tested in our Jenkins production instance by hacking on the jobs configuration and it seems to work fine, see https://jenkins.softwareheritage.org/blue/organizations/jenkins/DGRPH%2Fgitlab-builds/detail/gitlab-builds/2255/pipeline/23 for instance.

while I am on it: swh-jenkins-dockerfiles!75 (merged)

merged manually