Related to T2988

Related to T2988

Port debian lister to `swh.lister.pattern.Lister` API.

The new implementation will produce one instance of ListedOrigin model
per package, notably containing the set of parameters expected by the
debian loader.

The lister is also stateful, meaning only new packages and those with
new found versions since the last listing will be returned.

Closes T2979

All the fixtures declared in them are not used anymore in the
tests of the listers ported to the new Lister API.

The previous pagination implementation has a hard-coded limit server side [1]

[1]

```
{"error":"Offset pagination has a maximum allowed offset of 50000 for requests that return objects of type Project. Remaining records can be retrieved using keyset pagination."}
```

Related to T2994

R package last update date can be found in the "Packaged" field of
package info returned by tools::CRAN_package_db().

So retrieve it and parse it as a datetime to provide as last_update
parameter value in ListedOrigin model.

Closes T2989

Related to T2989

Related to T2987

Related to T2984

The PaginatedListedOriginList model has been updated in
rDSCHb93aa5be2c2d5dc2130e1027698f3e1255052d8d and the origins
field has been renamed to results.

The lister is stateless and has full listing capability.
It can request the Gitea API using HTTP token authentication.
Rate-limiting was not encountered but is handled generically.
Added support for getting repo last update date through API.

Related to T2987

Related to T2984

Use random credentials from the list in configuration and improve related
logging messages.

Related to T2987

It's used in multiple module tests now.

Related to T2987

Related to T2987

Note that the current implementation will start back the new visit from the last
next_page link seen (that's what is stored in the lister state to avoid computing back
the url). This means that this page will be seen at least 2 times, on the first visit
and on the next. This should not pose any problems as the listing is idempotent.

Related to T2987

Related to T2987

Related to T2987

Port npm lister to `swh.lister.pattern.Lister` API.

As before, the lister can be run in full or incremental mode.
When using incremental mode, only new and modified packages will
be returned since the last incremental listing process.
Otherwise, all packages will be listed in lexicographical order.

One major improvement to be noted, latest package update date
is now retrieved when available and sent to scheduler database.

Closes T2972

The new lister has only full listing capability.
It scrapes pypi.org list of packages.
Rate-limiting was not encountered but is handled generically.

The new lister has incremental and full listing capability.
It can request the Bitbucket API in anonymous and HTTP basic authentication
modes. Rate-limiting is not aggressive and is handled.

Fix error when a configuration value loaded from a config file is also
given as keyword parameter to the from_configfile method.

Override configuration loaded from config file only if the provided
value is not None.

This ensures that a celery task will be marked as failed if a request error
happens when listing origins.

Also remove no longer used JSON files.

Add swh.lister.utils.throttling_retry decorator enabling to retry a
function that performs an HTTP request who can return a 429 status code.

The implementation is based on the tenacity module and it is assumed
that the requests library is used when querying an URL.

The default wait strategy is based on exponential backoff.

The default max number of attempts is set to 5, HTTPError exception
will then be reraised.

All tenacity.retry parameters can also be overridden in client code.

Previously, the following error was raised when the task has finished
its execution: "Object of type ListerStats is not JSON serializable".

So ensure ListerStats object gets converted to dict before returning it.

Also add missing test for task function.

This is useful when one wants to test the lister in docker environment.