Skip to content

Reimplement Gitea lister using new Lister API

Tenma requested to merge generated-differential-D4907-source into generated-differential-D4907-target

The lister is stateless and has only full listing capability. It can request the Gitea API using HTTP token authentication. Rate-limiting was not encountered but is handled generically. Added support for getting repo last update date through API.

Also add Gitea lister mandatory params to lister/tests/test_cli.

Depends on !177 (closed) for build Closes #2971 (closed)

Test Plan

Tests have good coverage but must be splitted.

Example test run:

pytest --log-cli-level=WARNING swh/lister/gitea/tests/
==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.7.3, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
plugins: hypothesis-6.0.1, flask-1.1.0, requests-mock-1.8.0, django-4.1.0, postgresql-2.5.2, forked-1.3.0, mock-3.5.1, asyncio-0.14.0, xdist-2.2.0, dash-1.18.1, cov-2.11.0, swh.journal-0.6.1, swh.core-0.11.1.dev4+g40018f2
collected 8 items                                                                                                                                                                            

swh/lister/gitea/tests/test_lister.py::test_gitea_full_listing 
--------------------------------------------------------------------------------------- live log call ----------------------------------------------------------------------------------------
WARNING  swh.lister.gitea.lister:lister.py:79 No authentication token set in configuration, using anonymous mode
WARNING  swh.lister.gitea.lister:lister.py:97 Unexpected HTTP status code 429 on https://try.gitea.io/api/v1/repos/search?sort=id&order=asc&limit=3&page=2: b''
WARNING  swh.lister.gitea.lister:before_sleep.py:45 Retrying swh.lister.gitea.lister.GiteaLister.page_request in 1.0 seconds as it raised HTTPError: 429 Client Error: None for url: https://try.gitea.io/api/v1/repos/search?sort=id&order=asc&limit=3&page=2.
PASSED                                                                                                                                                                                 [ 12%]
swh/lister/gitea/tests/test_lister.py::test_gitea_auth_instance 
--------------------------------------------------------------------------------------- live log call ----------------------------------------------------------------------------------------
WARNING  swh.lister.gitea.lister:lister.py:75 Using authentication token from user u
PASSED                                                                                                                                                                                 [ 25%]
swh/lister/gitea/tests/test_lister.py::test_gitea_list_http_error[400] 
--------------------------------------------------------------------------------------- live log call ----------------------------------------------------------------------------------------
WARNING  swh.lister.gitea.lister:lister.py:79 No authentication token set in configuration, using anonymous mode
WARNING  swh.lister.gitea.lister:lister.py:97 Unexpected HTTP status code 400 on https://try.gitea.io/api/v1/repos/search?sort=id&order=asc&limit=3&page=1: b''
PASSED                                                                                                                                                                                 [ 37%]
swh/lister/gitea/tests/test_lister.py::test_gitea_list_http_error[500] 
--------------------------------------------------------------------------------------- live log call ----------------------------------------------------------------------------------------
WARNING  swh.lister.gitea.lister:lister.py:79 No authentication token set in configuration, using anonymous mode
WARNING  swh.lister.gitea.lister:lister.py:97 Unexpected HTTP status code 500 on https://try.gitea.io/api/v1/repos/search?sort=id&order=asc&limit=3&page=1: b''
PASSED                                                                                                                                                                                 [ 50%]
swh/lister/gitea/tests/test_lister.py::test_gitea_list_http_error[502] 
--------------------------------------------------------------------------------------- live log call ----------------------------------------------------------------------------------------
WARNING  swh.lister.gitea.lister:lister.py:79 No authentication token set in configuration, using anonymous mode
WARNING  swh.lister.gitea.lister:lister.py:97 Unexpected HTTP status code 502 on https://try.gitea.io/api/v1/repos/search?sort=id&order=asc&limit=3&page=1: b''
PASSED                                                                                                                                                                                 [ 62%]
swh/lister/gitea/tests/test_tasks.py::test_ping PASSED                                                                                                                                 [ 75%]
swh/lister/gitea/tests/test_tasks.py::test_full_listing PASSED                                                                                                                         [ 87%]
swh/lister/gitea/tests/test_tasks.py::test_full_listing_params PASSED                                                                                                                  [100%]

===================================================================================== 8 passed in 10.73s =====================================================================================

Migrated from D4907 (view on Phabricator)

Merge request reports