importlib.metadata is the Python standard library replacement for
pkg_resources.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

Instead of letting the MemcacheServerError exception pass through, prefer
to catch it and report the error to Sentry.

It enables to avoid returning a 500 error when caching failed, nevertheless
Web UI will be slower when such case is encountered.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

To avoid retrieving the list of branches in each page when browsing a
snapshot, it is put in cache.

However the messages associated to branches are also stored in cache
but those can have a large size and the cache size limit might be
reached.

As the messages of the branches are only exploited in the branches
and releases view, ensure to not store them in get_snapshot_content
cache to avoid MemcacheServerError being thrown due to a too large
value to cache.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

Harmonize cache key format.

Add functions cache_set and cache_get wrapping calls to django cache API.

Dump value to cache as binary using msgpack and store a compressed version
of it to avoid hitting cache size limit.

Related to #4785.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

It ensures cache keys are changing after each deployment of swh-web.

Nicolas Dandrimont authored 1 year ago and Antoine Lambert committed 1 year ago

This will be useful to introduce a web-based view that handles the
message delivery, rather than a management command.

Nicolas Dandrimont authored 1 year ago and Antoine Lambert committed 1 year ago

This will allow introducing a new web-based view to inject inbound
emails into the webapp, instead of having to use the django management command.

In contrary to whitespace characters, django do not automatically quote
plus characters in URL arguments so we must ensure those are not quoted
backend side when processing an origin URL submitted to Save Code Now
to avoid a loading issue.

In order to support origin with multiple visit types in browse interface,
update lookup_origin* implementation to fetch the visit types associated
to an origin in an efficient way using swh-search.

Related to #4786.

Modify signatures of functions archive.lookup_origin and those in
origin_visits module to take a simple origin_url string instead of
a dictionary containing a single url key.

Related to swh/meta#5075.

Django model BinaryField can either be mapped to bytes or memoryview.

pytest 8.0 modified the handling of default parameter when defining
configuration options, which breaks the use of 'pytest-postgresql < 4'.

Workaround that issue by explicitly defining postgresql_load option
in pytest.ini file.

See https://github.com/pytest-dev/pytest/issues/11282.

It enables to filter visits by type when querying the latest origin visit.

Use that new parameter in the search results view to ensure latest visits
with correct type are fetched through XHRs.

When requesting the generation of a SVG badge for a SWH object, ensure
to cache the HTTP response when the targeted object is found in the
archive.

Related to #4784.

Add extid types related to Guix and Nix in /extid/ endpoint documentation.

Related to swh/meta#3781.

Add new visit type counters resulting from the deployment of the
new nixguix lister.

Related to swh/meta#3781.

The statuses of save code now requests will be updated in a push way
through use of webhooks soon in production and as a consequence the
"scheduled" loading task status will no longer be reported as webhooks
first reported status is "running".

So rename the "not yet scheduled" status to "pending" to allow better
transitions for loading task statuses.

Previous status transitions were:  not created -> not yet scheduled
-> scheduled -> running -> (succeeded|failed).

New status transitions are: not created -> pending -> running
-> (succeeded|failed).

Related to swh/infra/sysadm-environment#5165.

Related to #4779.

Add endpoint /api/1/extid/(extid_type)/(extid_format):(extid)/ to get
info about an external identifier of a given type.

Add endpoint /api/1/extid/target/(swhid)/ to retrieve a list of external
identifiers targeting a given SWHID.

Related to #3435.

Text located after a link was not displayed as links were not
inserted inline so ensure they are.

Fixes #4783.

vlorentz authored 1 year ago and Antoine Lambert committed 1 year ago

In practice, RemoteException happens only when a backend service has an internal
error (which is already logged in its own Sentry project) or a query timeout;
and TransientRemoteException only on temporary 502 errors.

Neither are helpful, and they clutter Sentry by bursting a dozen such errors
a couple of times a week.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

It fixes acceptance or rejection of a save request for an origin URL that
needs to be percent encoded when provided as URL argument.

Previous implementation was using the builtin hash function but
it is seeded with a random seed, so the cache keys were completely
random across processes and webapp django cache was not shared across
gunicorn workers in production and staging environment.

Use hashlib.md5(usedforsecurity=False) instead, which is a cheap,
repeatable and short hash function.

Related to #4782

Enable to log Python tracebacks in terminal when executing cypress tests
to ease debugging.

Nevertheless, only display Python logs whose level is >= WARNING when
running cypress tests and change log level of tracebacks from debug to
error.

Since Django 5.0, the logout view of the default authentication backend
no longer accepts GET requests but only POST ones so adapt swh-web
to take this change into account and enable proper logout operation
when using the default authentication backend with Django >= 5 (it is
used when running cypress tests or executing the webapp in docker for
instance).

Also fix a couple of issues related to login/logout in templates
and improve styling a bit.

And add a (loosely related) test for the content API endpoint

Add around 30 new programming languages that can he highlighted using
the highlight.js library by registering associated hljs plugins.