Traceback was escaped for HTML views as they are displayed in error page
when debug mode is activated, so ensure to unescape it before displaying
it when the pytest assertion fails.

The charset_normalizer.detect function can return a dict filled with
None values when it failed to detect an encoding so ensure to add a
None check to avoid runtime error when a content encoding cannot be
detected.

It is now only used in the test suites so no need to keep it as
a runtime dependency, also remove not required lxml dependency.

HtmlPrettifyMiddleware uses the `prettify()` method from BeautifulSoup.
Its documentation says:

> Since it adds whitespace (in the form of newlines), `prettify()` changes
> the meaning of an HTML document and should not be used to reformat one.
> The goal of `prettify()` is to help you visually understand the structure
> of the documents you work with.

Having HtmlPrettifyMiddleware enabled therefore leads to surprising
behavior, like extra spaces surrounding “something” when writing
`<code>something</code>` in a template.

Now that Django templates are prettified using `djlint`, this middleware
only has downsides. Let’s remove it entirely.

It has been observed that some users are scraping the browse content
views at a high rate by sending too many requests in a short amount
of time.

So add a rate limit of sixty requests per minute to content views to
avoid such abuse by using the django-ratelimit package.

Related to #4795.

Chardet does not seem really maintained and failed to properly detect some
encodings, resulting in 500 errors when attempting to browse some contents.

So prefer to use the charset-normalizer package instead which is better at
detecting encoding while being faster than chardet.

Paul Wise authored 11 months ago and Antoine Lambert committed 11 months ago

Uses a hidden disabled selected option with an empty value to do this.

Prevents accidentally submitting forges as the first forge type,
which is currently bitbucket, which is rarely used on the web.

The AFN submission API already rejects an empty forge type,
giving a reasonably good error to the end-user.

Using a postcss plugin now results in a lot of displayed warnings so
prefer to use a dedicated webpack plugin to trigger CSS linting.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

Restore previous behavior and select by default the latest visit of
an origin regardless of its type.

This reverts commit 9c2e3a12.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

Align returned data with other endpoints related to origins by adding
visit types info in order to fix errors in the search Web UI.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

This code was used at the time to get extra info (notably stack trace) about
a save code now task but this no longer works since a while and it is redundant
with sentry so better removing it.

Those are no longer rendered after previous commits preventing XSS
so prefer to simply display URLs instead.

This avoids an XSS avenue within errors

swh-storage revision_log() method is expected to return Revision objects
in a near future, so make sure we handle this properly.

It seems we depend on v3 now (v2 generates an AttributeError for
webpack_loader.loaders).

This matches what swh-search returns and expects. This could make
test_lookup_origin() (in test_archive.py) fail when the bad luck (?) made
hypothesis generate a test set where swh-search's in-memory backend
tries to merge the visit_types with an existing one (thus trying to add
a pair set objects).

Adapt the definition of OriginInfo (more or less the local version of
swh-search's OriginDict) accordingly.

Since the release of pytest 8.1, some pytest options are no longer
needed and editable install can be used when running tests using tox.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

When searching for origins with the Web UI, if the search pattern is equal
to the URL of an archived origin then user gets automatically redirected to
the interface for browsing it.

Previously the visit type was not taken into account prior performing the
redirection so ensure to redirect only if the origin has a visit type
that matches the selected one.

Instead of writing a dummy webpack-stats.json file into the static folder
of swh-web when pytest is configured, prefer to autouse a session scoped
fixture mocking the return value of WebpackLoader.load_assets.

Use new option consider_namespace_packages to fix tests execution
with latest pytest release.

Some Jupyter notebooks rendered by the webapp might require the use of
mathjax extensions that were not available in the static assets.

So ensure those extensions are available and can be lazily loaded when
required.

Those are not of interest and generate a lot of noises in swh-web dashboard
from Sentry Web UI.

To support secret key rotation, allow setting SECRET_KEY_FALLBACKS in
the swh-web yaml config.

Introduce support of SECRET_KEY_FALLBACKS for inbound email validation.

After the migration to Django 4, the signature algorithm for signed
addresses was upgraded to sha256. However, sha1-based addresses still
exist in the wild, so we need to keep supporting them.

The current setup wouldn't allow Bcc's, bounces or aliases, throwing a
400 error.

It seems it is possible for the _savable_visit_types() helper function
to be called with an empty 'visit_types' (case found in docker tests).

Prevent an IndexError.

Datatables 2.0 can send a request to restore default ordering based
on data index but it was not properly handled by the related views.

Versions of extids related to nix/guix were bumped to 1 as some issues
were detected in production after the first batch of related loadings.

Related to swh/infra/sysadm-environment#5222.