Jérémy Bobbio (Lunar) authored 2 years ago and Jérémy Bobbio (Lunar) committed 2 years ago

Related to swh/meta#4959

Related to swh/meta#4960

- pre-commit from 4.1.0 to 4.3.0,
- codespell from 2.2.1 to 2.2.2,
- black from 22.3.0 to 22.10.0 and
- flake8 from 4.0.1 to 5.0.4. Also freeze flake8 dependencies.

Also change flake8's repo config to github (the gitlab mirror
being outdated).

black is considered stable since release 22.1.0 and the version
we are currently using is quite outdated and not compatible with
click 8.1.0, so it is time to bump it to its latest stable release.

Please note that E501 pycodestyle warning related to line length
is replaced by B950 one from flake8-bugbear as recommended by black.
https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html#line-length

Related to T3922

This also drops spurious copyright headers to those files if present.

Related to T3812

Enable to check package documentation can be built without producing
sphinx warnings.

The sphinx environment is designed to be used in continuous integration
in order to prevent breaking documentation build when committing changes.

The sphinx-dev environment is designed to be used inside a full swh
development environment.

Related to T3258

Needs djangorestframework-stubs>=1.4 for proper mypy checking.

If djangorestframework-stubs does not contain the stub for
`format_suffix_patterns`, then mypy does not see any issue on passing it
a `List[object]` as argument, so the `# type: ignore` is useless,
so mypy complains about it.

For some reason, the previous commit ended up messing pip while
selecting the extras of swh.core to be used, making it not select the
[http] extra.

It's a tad more colorful

I noticed this because my virtualenv ended up with the djangorestframework-stubs
package installed, which made the mypy pre-commit hook fail.

This commit adds drf-stubs to the tox.ini, and does the relevant updates to our
type annotations/signatures.

Turns out that upstream DRF has all `request` parameters named that way, so we
rename all our `req` parameters to match, which appeases mypy's signature
checks.

- blackify all the python files,
- enable black in pre-commit,
- add a black tox environment.

For this we:

- ensure we create the DepositRequest with str as raw_metadata (instead
  of a bytes), and
- make atom_dataset return a str instead of a bytes

  Every time this chunk of data is sent on the wire (http req to the django app)
  this needs to be an str, otherwise we have things like:

    raw_metadata = "b'[...]'"

  especially when testing against django 2, which is the only django version
  available on debian sid, so this is needed to make it possible to build the
  deb file on sid *with* tests enabled.
- add a py3-django2 environment in tox
- bump the dependency to django <3 in requirements-server.txt (otherwise it
  will take precendence on the deps: field of the tox file, thus execute tests
  in py3-django2 with django 1.11).

Note that no other tests than executing pytest have been done with django2, so
do not expect it works flawlessly.

so the scheduler interaction code is executed.

Note that this does not test for correctness in these interactions yet.

also move tests/__init__.py content in tests/conftest.py and adapt test code
accordingly.

This also ensures retries_left is set otherwise tests may fail when using
the local sheduler.

Current djangos-stubs version (1.2.0) is not compatible with latest mypy one (0.750).
So pin their version until the issue is fixed.

hat tip to @anlambert for the tip

That also fix the pytest tests run

Without this, some dependencies from swh.core[http] are not installed
(because swh.core, without extras, is installed first from
requirements-swh.txt).

Probably similar to this issue https://github.com/pypa/pip/issues/6506

We need to explicitely specify the location where pytest looks for tests
to prevent the kind of nasty error described here:

  https://tox.readthedocs.io/en/latest/example/pytest.html#known-issues-and-limitations

Also note that migrations scripts are excluded from flake8 in the tox file.

related to T1247.