The http_retry decorator has been moved to swh-core package in order
to ease its reuse across swh packages.

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Also include them the proper sphinx way.

Related to T4496

Numerous listers were using the same page_request method or equivalent
in their implementation so prefer to deduplicate that code by adding
an http_request method in base lister class: swh.lister.pattern.Lister.

That method simply wraps a call to requests.Session.request and logs
some useful info for debugging and error reporting, also an HTTPError
will be raised if a request ends up with an error.

All listers using that new method now benefit of requests retry when
an HTTP error occurs thanks to the use of the http_retry decorator.

Instead of retrying HTTP requests only for 429 status code by default,
prefer to use the generic retry policy enabling to also retry for status
codes >= 500 but also on ConnectionError exceptions.

Rename throttling_retry decorator to http_retry to reflect this change.

Related to T3922

Related to D6967

Related to T3629

This does not yet enter into the registration of a new lister.

Related to T3629

tuleap-lister: fix args in test_task.

tuleap-lister: Add rate-limiting test + fix debug and typo.

tuleap-lister: code review: fix mocker + tests/setup_cli.

tuleap-lister: code review: fix relister > lister.

tuleap-lister: code review: fix test_task kwargs.

tuleap-lister: code review: Remove authentication useless lines + fix typos.

tuleap-lister: code review: improve results_simplified for svn repos.

tuleap-lister: code review: add name to CONTRIBUTORS file.

tuleap-lister: code review: Update tutorial for misc files to edit.

tuleap-lister: code review: Update copyright to 2021 exactly.

tuleap-lister: code review: Update py files perms -X.

tuleap-lister: code review: minimise json files.

tuleap-lister: code review: fix chmod on json files.

tuleap-lister: code review: fix var names + add tests.

tuleap-lister: code review: fix useless indirection.

tuleap-lister: code review: Add empty repo test, minor typo fixes.

Sphinx complains because it's an orphan document.

Tenma authored 4 years ago and Antoine R. Dumont committed 4 years ago

This adds a new tutorial which details how to currently write the new listers (both
incremental or stateless). This proposes a python template file to start a new lister.

At last, this renames the previous tutorial into tutorial-2017.

Related to T3073

Remove outdated part about listers database and use swh CLI in
README for executing a lister instead of raw Python code.

Related to T2188

Related T1984

It's redundant with the swh modules in itself.

Add new page in lister tutorial which guides
through the process of running a new lister in
docker.

Add testing section for lister. Also added how to run a
new lister section which elaborates about the steps
required to run the now lister in docker

related to T1330

originally appeared on the Software Heritage blog at
https://www.softwareheritage.org/2017/03/24/list-the-content-of-your-favorite-forge-in-just-a-few-steps/
by Avi Kelman

Closes T762

change cherry picked from python module template
commit 71b117ba0cf9f1251b1cac26d0994df03a4c787d