Instead of a generic 500.

This will be useful for debugging and checking records are indeed
configured.

Hypothesis can generate multiple times the same origin URL which
makes test fail due to the use of django cache in get_origin_visits.

So force the generation of a single hypothesis example for that test.

There is a lot of origins without any full visit in the archive so in
that case the origin visits page looks empty when reaching it as the
default is to display full visits with different snapshots.

So prefer to display all visits by default and disable the full visits
switch radio buttons when there is no such visits.

The command refreshes all the active mailmaps, and disables every
mailmap that has recently been updated and marked as disabled, once.

To do these actions as atomically as possible, we make sure to select
all rows for update, and to defer all _date fields in the default
queryset manager for UserMailmap objects to avoid resetting them to
earlier values by mistake.

Antoine Lambert authored 3 years ago and Nicolas Dandrimont committed 3 years ago

pytest v7.0.0 removed _pytest.tmpdir.TempdirFactory, which is used
by some of the pytest plugins we use.

The command refreshes all the active mailmaps, and disables every
mailmap that has recently been updated and marked as disabled, once.

To do these actions as atomically as possible, we make sure to select
all rows for update, and to defer all _date fields in the default
queryset manager for UserMailmap objects to avoid resetting them to
earlier values by mistake.

Related to T3908

pytest v7.0.0 removed _pytest.tmpdir.TempdirFactory, which is used
by some of the pytest plugins we use.

This clarifies the current code and optimizes the re-fetching of all origin visits /
origin visit statuses when new visits happen. It's an optimization providing previous
visits are in the cache already.

The second part serves also as a workaround of some problematic origins (the ones with a
high number of visits). It's only a workaround though because when voiding the
cache (e.g. restart the cache service), the initial error (e.g. crash 500 or 502, ...)
will still happen.

Another change is on its way to try and fix that specific problem.

Related to T3905

Add a new database table to store mailmap settings that will be
used to add display names for an authors that can be found in the
archive.

Add endpoints to add or update these mailmap settings restricted
to staff users or users with special permission.
They will be used in the upcoming Web UI to manage mailmaps.

Signature of storage.revision_log has changed in 4544d7ca8e9e and the
limit parameter is not at the same position as before, so better
passing it as keyword parameter to avoid future errors of this type.

The campaign is now over.

This reverts commit 615ef02e.

In the Save Code Now Web UI, add the link to browse a saved origin
only if the visit status is valid.

This prevents to browse the saved origin when loading data are not
yet available in storage database (replication lag for instance).

Related to T3848

Simulate HTML form behavior by adding a keyup event listener on the
vault modals dedicated to tasks creation.

Related to T3845

Some fields returned in the elasticsearch document got renamed since so
fix their extraction.

Add support for new programming languages to highlight in the webapp
by integrating highlight.js language plugins in the highlightjs chunk
generated by webpack.

Fix issue in Python code that was preventing to map a mimetype to a
language through its aliases.

Some module folders do not have package.json file in it but licenses can
still be overridden using the licenseOverride option of the plugin.

Previously, package.json file was attempted to be read regardless of
possible license override, which was causing error.

Fix regression introduced in 835a8601: History link was no longer
displayed for all origins in snapshot directory view.

The elasticsearch request used to get detailed info about a save code
now request was not returning any result anymore as some fields in
the swh-workers-* index got renamed.

The next release of swh-model will remove this attribute, because it was
only a special case, and we will replace it with the generalized 'offset_bytes'.

A future release of swh-model will change its constructor's signature (replace 'offset'
and 'negative_utc_offset' with 'offset_bytes).

In order to allow public access to the graph API, we need to prevent
users to DoS the graph service with resource intensive queries.

Fortunately, the graph API supports a max_edges parameter to limit the
number of edges traversed by the graph.

According to the type of user querying the graph service through the
Web API proxy, a maximum value for the max_edges will be set before
sending a request to the graph API.

Three type of users are considered: anonymous, standard and staff.
The maximum values for the max_edges query parameter according to
user types are retrieved from the webapp configuration.

Related to T3836

It is possible to pass query parameters in the graph_query URL argument
of the Web API graph endpoint if the ? character is quoted.

So add extra processing in the endpoint implementation to merge the
query parameters extracted from the graph_query value and those
coming from the django request.

The purpose is to avoid some query parameters to be silently ignored
when requesting the graph API.

If the request sent to the graph API returns an error, forward
the raw error response as the Web API response instead of trying
to parse a JSON response that usually fails.