This avoids an XSS avenue within errors

swh-storage revision_log() method is expected to return Revision objects
in a near future, so make sure we handle this properly.

It seems we depend on v3 now (v2 generates an AttributeError for
webpack_loader.loaders).

This matches what swh-search returns and expects. This could make
test_lookup_origin() (in test_archive.py) fail when the bad luck (?) made
hypothesis generate a test set where swh-search's in-memory backend
tries to merge the visit_types with an existing one (thus trying to add
a pair set objects).

Adapt the definition of OriginInfo (more or less the local version of
swh-search's OriginDict) accordingly.

Since the release of pytest 8.1, some pytest options are no longer
needed and editable install can be used when running tests using tox.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

When searching for origins with the Web UI, if the search pattern is equal
to the URL of an archived origin then user gets automatically redirected to
the interface for browsing it.

Previously the visit type was not taken into account prior performing the
redirection so ensure to redirect only if the origin has a visit type
that matches the selected one.

Instead of writing a dummy webpack-stats.json file into the static folder
of swh-web when pytest is configured, prefer to autouse a session scoped
fixture mocking the return value of WebpackLoader.load_assets.

Use new option consider_namespace_packages to fix tests execution
with latest pytest release.

Some Jupyter notebooks rendered by the webapp might require the use of
mathjax extensions that were not available in the static assets.

So ensure those extensions are available and can be lazily loaded when
required.

Those are not of interest and generate a lot of noises in swh-web dashboard
from Sentry Web UI.

To support secret key rotation, allow setting SECRET_KEY_FALLBACKS in
the swh-web yaml config.

Introduce support of SECRET_KEY_FALLBACKS for inbound email validation.

After the migration to Django 4, the signature algorithm for signed
addresses was upgraded to sha256. However, sha1-based addresses still
exist in the wild, so we need to keep supporting them.

The current setup wouldn't allow Bcc's, bounces or aliases, throwing a
400 error.

It seems it is possible for the _savable_visit_types() helper function
to be called with an empty 'visit_types' (case found in docker tests).

Prevent an IndexError.

Datatables 2.0 can send a request to restore default ordering based
on data index but it was not properly handled by the related views.

Versions of extids related to nix/guix were bumped to 1 as some issues
were detected in production after the first batch of related loadings.

Related to swh/infra/sysadm-environment#5222.

It enables to have a fixed header when browsing a table with a large
amount of entries per page.

When increasing the number of entries per page to display, it enables
to scroll the page instead of scrolling a table of fixed height.

When no visit type for an origin is explicitly provided as query parameter,
ensure a full VCS visit type (git, svn or hg) is selected by default when
an origin has multiple visit types.

Related to #4786.

Previously, the checksums for each file was stored in the cache but those
are not exploited by the browse code so we can safely remove that dict
to optimize the cached entry size and avoid reaching memcached size limit
for large directories in production.

When visit type filtering is not enabled in the search options, ensure
to display all visit types for an origin in search results.

Related to #4786.

Some origins in the archive can have multiple visit types since
recently and could lead to confusing display in browse interface
as the latest visit is selected by default.

So add a select element at the top right part of the UI enabling
to switch the visit type for the browsed origin and update links
by adding the visit_type query parameter to keep visit type context
when navigating.

Related to #4786.

importlib.metadata is the Python standard library replacement for
pkg_resources.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

Instead of letting the MemcacheServerError exception pass through, prefer
to catch it and report the error to Sentry.

It enables to avoid returning a 500 error when caching failed, nevertheless
Web UI will be slower when such case is encountered.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

To avoid retrieving the list of branches in each page when browsing a
snapshot, it is put in cache.

However the messages associated to branches are also stored in cache
but those can have a large size and the cache size limit might be
reached.

As the messages of the branches are only exploited in the branches
and releases view, ensure to not store them in get_snapshot_content
cache to avoid MemcacheServerError being thrown due to a too large
value to cache.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

Harmonize cache key format.

Add functions cache_set and cache_get wrapping calls to django cache API.

Dump value to cache as binary using msgpack and store a compressed version
of it to avoid hitting cache size limit.

Related to #4785.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

It ensures cache keys are changing after each deployment of swh-web.

Nicolas Dandrimont authored 1 year ago and Antoine Lambert committed 1 year ago

This will be useful to introduce a web-based view that handles the
message delivery, rather than a management command.