When loading task types are available for these visit types in the
swh deployment environment (currently staging and docker), display
the associated counters in the coverage widget.

Related to T3945

Related to T3835

If one of the URL links to parse contains a comma character, the parsing
was incorrect.

So fix that issue by updating the regexp use to format the links for
display in the API Web UI.

Related to T3967

When calling "./bin/install" from swh-environment in a freshly created
venv, for some reason pip will end up installing flask==1.1.2 when
processing swh-web (likely related to the venv state after installing
other swh modules).

That flask version is quite outdated and will fail to import the json
module from the version of itsdangerous installed in the venv.

So as a workaround pin flask to the first version that does not trigger
that ImportError issue.

markupsafe 2.1.0 removes functions that were used by Jinja2 < 3,
the latter being a dependency of pybadges.

As Jinja2 < 3 is installed in the venv, pinning markupsafe to
a version still compatible with Jinja2 < 3 is the only way to
avoid error when running pytest as some plugins import jinja2.

When attempting to re-encode non UTF-8 textual content, use chardet
to find the encoding first and use it if the detection confidence
is really high.

Previously some encoding like SHIFT_JIS (for japanese language) were
not correctly detected and thus content were badly rendered in the
browse Web UI.

Add support for Bazaar repository loading in Save Code Now.

Simplify test to ensure it does not fail when building debian packages.

Tarballs are no longer loaded as branches but as releases in the
nixguix origin snapshots.

Currently, in case of a high number of visits for an origin, this makes the page
querying the result crash (with an empty cache). Note that even with a populated cache,
the current version could take some time to process.

The following diff opens a function which uses simpler storage api calls to retrieve the
most recent visit from a given point in time (with its most recent associated visit
status) instead of filtering all visits of an origin to retrieve the information. Hence
no longer relying on the cache of all visits for that origin either.

Related to T3905

To install the new hook:

  $ pre-commit install -t commit-msg

Numerous tests are broken since rDSCHc46ffadf7adf24c7eb3ffce062e8ade3818c79cc
has landed in swh-scheduler.

Ensure to restore the old loading task names for tests to succeed.

So we can show this to beta-testers who aren't staff.

This will log all incoming HTTP requests on update events (assuming the user
is authorized), for audit purposes.

Instead of a generic 500.

This will be useful for debugging and checking records are indeed
configured.

Hypothesis can generate multiple times the same origin URL which
makes test fail due to the use of django cache in get_origin_visits.

So force the generation of a single hypothesis example for that test.

There is a lot of origins without any full visit in the archive so in
that case the origin visits page looks empty when reaching it as the
default is to display full visits with different snapshots.

So prefer to display all visits by default and disable the full visits
switch radio buttons when there is no such visits.

The command refreshes all the active mailmaps, and disables every
mailmap that has recently been updated and marked as disabled, once.

To do these actions as atomically as possible, we make sure to select
all rows for update, and to defer all _date fields in the default
queryset manager for UserMailmap objects to avoid resetting them to
earlier values by mistake.

Antoine Lambert authored 3 years ago and Nicolas Dandrimont committed 3 years ago

pytest v7.0.0 removed _pytest.tmpdir.TempdirFactory, which is used
by some of the pytest plugins we use.

The command refreshes all the active mailmaps, and disables every
mailmap that has recently been updated and marked as disabled, once.

To do these actions as atomically as possible, we make sure to select
all rows for update, and to defer all _date fields in the default
queryset manager for UserMailmap objects to avoid resetting them to
earlier values by mistake.

Related to T3908

pytest v7.0.0 removed _pytest.tmpdir.TempdirFactory, which is used
by some of the pytest plugins we use.

This clarifies the current code and optimizes the re-fetching of all origin visits /
origin visit statuses when new visits happen. It's an optimization providing previous
visits are in the cache already.

The second part serves also as a workaround of some problematic origins (the ones with a
high number of visits). It's only a workaround though because when voiding the
cache (e.g. restart the cache service), the initial error (e.g. crash 500 or 502, ...)
will still happen.

Another change is on its way to try and fix that specific problem.

Related to T3905

Add a new database table to store mailmap settings that will be
used to add display names for an authors that can be found in the
archive.

Add endpoints to add or update these mailmap settings restricted
to staff users or users with special permission.
They will be used in the upcoming Web UI to manage mailmaps.

Signature of storage.revision_log has changed in 4544d7ca8e9e and the
limit parameter is not at the same position as before, so better
passing it as keyword parameter to avoid future errors of this type.