Related to T2610

flake8 hook has been removed from https://github.com/pre-commit/pre-commit-hooks
so now use the one from https://gitlab.com/pycqa/flake8

by moving import statements in functions.

Related to T2575.

Related to D3899#96289

The gitea lister can be run on multiple instances which could use the same id.
So listing another gitea instance, the current code would fail to insert data
for such case.

This commit fixes that behavior by prefixing the uid with the instance name.

Related to T2577

Existing listers use the `is_within_bound` [1] method from the base lister.
This method uses inclusive boundaries in all cases.

As some "range" task listers [2] [3] are using `split_range` function to create
"overlapping" ranges, this can cause concurrent insert issues down the line [4].

This commit adapts the function `split_range` to make the generated ranges no
longer overlap.

[1]
https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/core/lister_base.py$194-199

[2]
https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/gitlab/tasks.py$37-41

[3]
https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/gitea/tasks.py$36-41

Related to T2577

This fixes [1]

[1] https://sentry.softwareheritage.org/share/issue/b0119b56f24347bcb58ac28c68685c62/

Related to T2358

Prior to this commit, all listers were instantiated at the same time even if
only one was needed. This commit separates those instantiations.

The only drawback to this is the db model initialization which now happens at
each lister instantiation. This can be dealt with if needed at another time
though.

This should fix the remaining blocking problems in the jenkins build failure
report [1]

[1] https://jenkins.softwareheritage.org/view/Debian%20packages/job/debian/job/packages/job/DLS/job/gbp-buildpackage/78/consoleFull

Then drop all unneeded and indirect imports

This should fix the build [1]

[1] https://jenkins.softwareheritage.org/view/Debian%20packages/job/debian/job/packages/job/DLS/job/gbp-buildpackage/77/console

pytest wastes a lot of time in .hypothesis and .git; this commit excludes them.

This allows us to actually run the lister instantiation code instead of relying
on the underlying structure of the lister object. In turn, this allows future
listers to use the scheduler right in their __init__.

The previous include would pull all .mypy_cache directories too, which are quite
large as they include the SQLAlchemy stubs.

Léni Gauffier authored 4 years ago and Léni Gauffier committed 4 years ago

Summary: Lister implementation for Gitea, works for (T2313). For now because of https://github.com/go-gitea/gitea/issues/9165 it would require setting its param limit to 50.

Reviewers: #reviewers, ardumont

Reviewed By: #reviewers, ardumont

Subscribers: ardumont

Differential Revision: https://forge.softwareheritage.org/D3107

Related to T2367

Summary:
Related to T1734

From abandonned D2799

Reviewers: ardumont

Reviewed By: ardumont

Differential Revision: https://forge.softwareheritage.org/D2974

- blackify all the python files,
- enable black in pre-commit,
- add a black tox environment.

This commit fixes the repository mapping to model. It broke when the listed
repository was either None or missing the id field [1]

[1] https://sentry.softwareheritage.org/share/issue/532d682182fc43d6a7a99400e3928811/

Related to T2188

Following the github's documentation [1]

[1] https://developer.github.com/v3/guides/best-practices-for-integrators/#dealing-with-abuse-rate-limits

Related to T2170

Prior to this commit, we sent the origin url as a versioned artifact.
Now we send the origin url as a CRAN's canonical one, and the associated list
of artifacts found there (only 1 today).