swh-pipeline: Execute all code quality checks instead of just flake8

Prefer to rely on pre-commit to run all code quality checks in order to also benefit from black, codespell and isort checks.

This will enforce newly introduced code to pass those checks and force external contributors to install the pre-commit hooks.

I tested the changes in docker and also processed all swh repositories to fix failing checks after the recent version bumps of code quality tools in pre-commit configuration.

Jenkins job jenkins-tools/swh-jenkins-jobs-builder #408 succeeded .
See Console Output, Blue Ocean and Coverage Report for more details.

Unless i'm mistaken, this will also unify the versions used between the machines (because the plugin versions are declared in the pre-commit configuration file or some such).

So for my part.

Currently, configuration for black and flake8 can also be found in tox.ini files:

[testenv:black]
skip_install = true
deps =
  black==24.8.0
commands =
  {envpython} -m black --check swh

[testenv:flake8]
skip_install = true
deps =
  flake8==7.1.1
  flake8-bugbear==24.4.26
  pycodestyle==2.12.1
commands =
  {envpython} -m flake8

We could indeed remove them after these changes as these tox environments are not (black) or will no longer (flake8) be executed by Jenkins.

added 1 commit

• 4868f37b - swh-pipeline: Execute all code quality checks instead of just flake8

Compare with previous version

Jenkins job jenkins-tools/swh-jenkins-jobs-builder #409 succeeded .
See Console Output, Blue Ocean and Coverage Report for more details.

^ Remove twine-check from SKIP list.

added 4 commits

• 4868f37b...ac9467b1 - 3 commits from branch swh/infra/ci-cd:master
• bef17d31 - swh-pipeline: Execute all code quality checks instead of just flake8

Compare with previous version

^ Rebase

Running pre-commit should happen before rust building, shouldn't it? (cc. @vlorentz)

Could we make the skip list configurable in the job template yaml? We have a couple of modules with extra checks that we might want to be able disable.

Could we make the skip list configurable in the job template yaml? We have a couple of modules with extra checks that we might want to be able disable.

Good idea, will update.

Jenkins job jenkins-tools/swh-jenkins-jobs-builder #418 succeeded .
See Console Output, Blue Ocean and Coverage Report for more details.

added 1 commit

• c567576a - swh-pipeline: Execute all code quality checks instead of just flake8

Compare with previous version

Jenkins job jenkins-tools/swh-jenkins-jobs-builder #420 succeeded .
See Console Output, Blue Ocean and Coverage Report for more details.

^ Declare precommit checks to skip in job templates

added 1 commit

• 70b16a66 - swh-pipeline: Execute all code quality checks instead of just flake8

Compare with previous version

Jenkins job jenkins-tools/swh-jenkins-jobs-builder #421 succeeded .
See Console Output, Blue Ocean and Coverage Report for more details.

^ Also remove clippy precommit check as it is already executed by the jenkins pipeline

cc. @vlorentz, is there a reason to pick one or the other?

there isn't. they both start a build phase that is mutually exclusive, but Cargo manages locks itself, so it should be fine to run Clippy in parallel with the Rust tests. (and even save us some time on the parallelizable phase)

So, if I understand correctly, clippy can only run after the crate has been built; In that case it indeed makes sense to not run it with pre-commit in the jenkins pipeline

No, it can also run before the crate is built

Let me rephrase the question once more then.

Should the jenkins pipeline run clippy with pre-commit ?

yes

But it is already executed by the Rust tests step no ?

yes, and it should be removed from the Rust tests if we run it through pre-commit

added 2 commits

• 27bdee29 - 1 commit from branch swh/infra/ci-cd:master
• 05a9dc2b - swh-pipeline: Execute all code quality checks instead of just flake8

Compare with previous version

Jenkins job jenkins-tools/swh-jenkins-jobs-builder #423 succeeded .
See Console Output, Blue Ocean and Coverage Report for more details.

approved this merge request

added 2 commits

• 2206092b - 1 commit from branch swh/infra/ci-cd:master
• 2b0c2835 - swh-pipeline: Execute all code quality checks instead of just flake8

Compare with previous version

^ Rebase

Jenkins job jenkins-tools/swh-jenkins-jobs-builder #426 succeeded .
See Console Output, Blue Ocean and Coverage Report for more details.

merged manually