Add a new django application to manage bulk archival of origins.

As a first step, add a new Web API endpoint to submit a list of origins
to archive and a new endpoint enabling to retrieve that list of origins
in a paginated way.

The Web API endpoint enables an user with specific permission to submit
a list of origin URLs and their visit types through a POST request.
The endpoint performs some basic checks on the received origins data to
verify origin URLs are well formed but also if provided visit types are
supported.

If provided origins data are valid, the request is accepted and a oneshot
scheduler task is created to execute the bulk-save lister. That lister will
consume that list of origins, perform some extra but more costly checks on
them and those validated by it will then be scheduled for loading into the
archive.

If some origins data are not valid, the request is rejected and a list of
bogus origins along the reasons of their rejections is returned to the user
in the Web API response.

The other endpoint is dedicated to be consumed by the bulk-save lister to
retrieve the submitted origins list in a paginated way. It is better to
store submitted origins data on the webapp side to avoid bloating the
scheduler database with large JSON documents in case a big list of origins
is submitted by an user.

Related to #4802.

Synchronize it with the one used to render code elements in all pages
of the webapp as it enables to better highlight them.

Antoine Lambert authored 10 months ago and Antoine Lambert committed 10 months ago

This should help avoiding implementation mistakes as mypy will now
check typing when a client code use these interfaces.

Fix some typing issues that were uncovered by this change.

Instead of defining our own django assertion wrappers, prefer to
use those from pytest-django.

OptimizeCSSAssetsPlugin is no longer maintained and must be replaced by
CssMinimizerPlugin.

It implied the migration of the eslint configuration files to the
new flat format (thanks to the eslint-migrate-config CLI tool).

- Create new route with origin url as query param
- Redirect old route

The oldest part of the scheduler API was updated to use model classes
(based on attr package) instead of dictionaries in order to improve
typing.

swh-model was updated to fix deserializing of origin qualifier in
a QualifiedSWHID instance which breaks a test of the webapp.

Align test with changes in swh-model and improve it to check that
a qualified SWHID containing special characters that must be escaped
can be successfully resolved to a browse URL but also that the latter
can be successfully browsed.

Improve the main template to use better rules to show the
"Administration" section of the left menu.

Add a "Users & groups" entry if the swh.web.admin app is enabled. This
later opens the users admin page (either keycloak or the django admin
page) in a new tab.

Set hypothesis max_examples to 1 and remove subtest fixture use.

It fixes issues with autodoc processing as this django application is
disabled by default.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 11 months ago

Enable to use django administration site in swh-web by including "swh.web.admin"
application in swh_extra_django_apps list from swh-web configuration.

When enabled, the admin site is available through the new /manage/ endpoint.

It allows an user to modify some django models of interest (adding, deleting or
modifying rows) if he/she has adequate permissions to do so.

Currently, only the add forge now request and save code now request models can
be managed with the admin site through the use of the following permissions:

- swh_web_add_forge_now.add_request
- swh_web_add_forge_now.change_request
- swh_web_add_forge_now.delete_request
- swh_web_add_forge_now.view_request
- swh_web_save_code_now.add_saveoriginrequest
- swh_web_save_code_now.change_saveoriginrequest
- swh_web_save_code_now.delete_saveoriginrequest
- swh_web_save_code_now.view_saveoriginrequest

When swh-web is using the default django authentication backend, the admin site
also enables and administrator to manage user accounts and their permissions.

When swh-web is using Keycloak as the authentication backend, the permissions
listed above must be added as swh-web client roles and these roles can then be
set for an user through the Keycloak admin console in order to use the features
offered by the admin site.

It messes up with django permissions creation when using pytest-xdist.