Nicolas Dandrimont authored 1 year ago and Antoine Lambert committed 1 year ago

This will allow introducing a new web-based view to inject inbound
emails into the webapp, instead of having to use the django management command.

In contrary to whitespace characters, django do not automatically quote
plus characters in URL arguments so we must ensure those are not quoted
backend side when processing an origin URL submitted to Save Code Now
to avoid a loading issue.

In order to support origin with multiple visit types in browse interface,
update lookup_origin* implementation to fetch the visit types associated
to an origin in an efficient way using swh-search.

Related to #4786.

Modify signatures of functions archive.lookup_origin and those in
origin_visits module to take a simple origin_url string instead of
a dictionary containing a single url key.

Related to swh/meta#5075.

Django model BinaryField can either be mapped to bytes or memoryview.

pytest 8.0 modified the handling of default parameter when defining
configuration options, which breaks the use of 'pytest-postgresql < 4'.

Workaround that issue by explicitly defining postgresql_load option
in pytest.ini file.

See https://github.com/pytest-dev/pytest/issues/11282.

It enables to filter visits by type when querying the latest origin visit.

Use that new parameter in the search results view to ensure latest visits
with correct type are fetched through XHRs.

When requesting the generation of a SVG badge for a SWH object, ensure
to cache the HTTP response when the targeted object is found in the
archive.

Related to #4784.

Add extid types related to Guix and Nix in /extid/ endpoint documentation.

Related to swh/meta#3781.

Add new visit type counters resulting from the deployment of the
new nixguix lister.

Related to swh/meta#3781.

The statuses of save code now requests will be updated in a push way
through use of webhooks soon in production and as a consequence the
"scheduled" loading task status will no longer be reported as webhooks
first reported status is "running".

So rename the "not yet scheduled" status to "pending" to allow better
transitions for loading task statuses.

Previous status transitions were:  not created -> not yet scheduled
-> scheduled -> running -> (succeeded|failed).

New status transitions are: not created -> pending -> running
-> (succeeded|failed).

Related to swh/infra/sysadm-environment#5165.

Related to #4779.

Add endpoint /api/1/extid/(extid_type)/(extid_format):(extid)/ to get
info about an external identifier of a given type.

Add endpoint /api/1/extid/target/(swhid)/ to retrieve a list of external
identifiers targeting a given SWHID.

Related to #3435.

Text located after a link was not displayed as links were not
inserted inline so ensure they are.

Fixes #4783.

vlorentz authored 1 year ago and Antoine Lambert committed 1 year ago

In practice, RemoteException happens only when a backend service has an internal
error (which is already logged in its own Sentry project) or a query timeout;
and TransientRemoteException only on temporary 502 errors.

Neither are helpful, and they clutter Sentry by bursting a dozen such errors
a couple of times a week.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

It fixes acceptance or rejection of a save request for an origin URL that
needs to be percent encoded when provided as URL argument.

Previous implementation was using the builtin hash function but
it is seeded with a random seed, so the cache keys were completely
random across processes and webapp django cache was not shared across
gunicorn workers in production and staging environment.

Use hashlib.md5(usedforsecurity=False) instead, which is a cheap,
repeatable and short hash function.

Related to #4782

Enable to log Python tracebacks in terminal when executing cypress tests
to ease debugging.

Nevertheless, only display Python logs whose level is >= WARNING when
running cypress tests and change log level of tracebacks from debug to
error.

Since Django 5.0, the logout view of the default authentication backend
no longer accepts GET requests but only POST ones so adapt swh-web
to take this change into account and enable proper logout operation
when using the default authentication backend with Django >= 5 (it is
used when running cypress tests or executing the webapp in docker for
instance).

Also fix a couple of issues related to login/logout in templates
and improve styling a bit.

And add a (loosely related) test for the content API endpoint

Add around 30 new programming languages that can he highlighted using
the highlight.js library by registering associated hljs plugins.

Some SPDX expressions located in package.json can be invalid, full license
name instead of SPDX code for instance.

In order to workaround this type of issues, use the spdx-correct package
enabling to get correct SPDX expression from an invalid one based on
some heuristics.

As django unquotes arguments when parsing URL, we need to requote origin
URL provided to the save code now API endpoint or the URL validation is
failing otherwise due to invalid character(s) in it.

The previously used regular permission was too permissive and arguments parsing
for save code now API view were failing if the /url/url/ substring was contained
in the origin URL also provided as URL argument.

Add webapp1.internal.softwareheritage.org as production server name.

Allow to set the production server names through a new configuration
entry named production_server_names.

Related to swh/infra/sysadm-environment#5183.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

It exist cases where bytes of a README file are not available in objstorage
(for instance in mirrors due to replication lag) so ensure to handle such
cases to avoid returning a 500 error in views rendering a README file.

Related to #4773.

Antoine Lambert authored 1 year ago and Antoine Lambert committed 1 year ago

The resolve_swhid function was not supporting the resolving of a directory
SWHID with path qualifier set and anchor qualifier set to a release targeting
a directory.