The oldest part of the scheduler API was updated to use model classes
(based on attr package) instead of dictionaries in order to improve
typing.

_complete_deposit() schedules a loading task; but it was called before the
DepositRequest object was saved to the database.

This means that if the loading tasks executes very fast, or the DepositRequest
takes a while to save, then the loader won't see the current DepositRequest when
it queries the deposit's current state.

This is especially likely to happen with large code uploads, as saving them to
disk happens before the DepositRequest row is written to the database.

Related to swh/meta#5075.

Side effect of https://github.com/sissaschool/xmlschema/commit/cdae888.

Turns these references into simple links in order for the rst checks
performed by PyPI to pass when uploading a new package version.

As the known_first_party isort option was not set yet, previous
imports ordering got broken so restore it.

It was placed in mypy.plugins.django-stubs section so it was not
taken into account.

The former does not always plays well with PEP420...

vlorentz authored 1 year ago and vlorentz committed 1 year ago

It was deprecated years ago, and no client used it in staging or prod
since 2022-04-11

vlorentz authored 1 year ago and vlorentz committed 1 year ago

These are Codemeta terms, but they were used in the Atom namespace.
As they are not read by the tests using them, we can just remove them.

vlorentz authored 1 year ago and vlorentz committed 1 year ago

A future commit will reject these invalid tags

vlorentz authored 1 year ago and vlorentz committed 1 year ago

I found some deposits on staging which use
<entry xmlns="https://doi.org/10.5063/SCHEMA/CODEMETA-2.0"> instead

This ensures the deposit checks in the elastic infra (using azure objstorage) is
working. There is currently no way to prevent the content-encoding from being detected
and stored in the blobstorage (various tryouts have been tested to no avail). That
content-encoding, is then preventing the checking/reading to work (failing [2]).

Following the documentation, a workaround has been implemented to reset a posteriori the
content-encoding [1].

It's not proper but, from repl experimentation, it's working which is better than the
actual failure.

[1] https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-properties-metadata-python

[2]
```
HttpResponseError: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing data: incorrect header check'))
```

Refs. swh/infra/sysadm-environment#5129

Use None instead of empty string to try to prevent content-encoding to be set when
writing.

Refs. swh/infra/sysadm-environment#5129

Currently, the writing in the azure objstorage adds extra headers to the blobs. Which
prevents the correct reading of those blobs when checking the tarballs.

According to the documentation, this option opening should allow to configure more
appropriately.

Refs. swh/infra/sysadm-environment#5129

If done on the full path, the conditional technically fails the check for the wrong
reason. This also adds tests on this case.

Refs. swh/infra/sysadm-environment#5129

Refs. swh/infra/sysadm-environment#5129

Similar adaptation of the deposit_check endpoint to allow reading remote tarballs when
using remote storage backends.

Refs. swh/infra/sysadm-environment#5129

Using the File api is consistent across various storage backends. The previous
implementation was only working for file storage. As we are migrating towards remote
backends, this no longer is enough.

Unfortunately though, we cannot rely on zipfile.is_zipfile or tarfile.is_tarfile as
those only works with local filepath. So we need to rely on testing & failing with
exception trapping to read properly the tarballs...

Refs. swh/infra/sysadm-environment#5129

They were only checking one tarball case (the ones reference in the ready_deposit_ok
deposit). This now creates as much deposit as there are extensions to check.

This only had to adapt the deposit so it's provided with a valid metadata. Otherwise,
the metadata checks were invalid, failing the overall tests.

Refs. swh/infra/sysadm-environment#5129

Refs. swh/infra/sysadm-environment#5129

This reverts commit bb22867c.

Not ready yet

This adapts the memcached import the same way the swh.web module did.

Follow up 9210c012

Refs. swh/infra/sysadm-environment#5129

This adapts the memcached import the same way the swh.web module did.

Follow up on 4698e8c9

Refs. swh/infra/sysadm-environment#5129

That uses django.db.models' by default, and fall back to the old contrib
one if need be.

This later has been deprecated for some time and is now producing
errors.

Doing this should work in any situation. The migration is actually only
a matter of the name of the type in the field metadata, so using a
stable one (swh.deposit.model.JSONField) make it backend-agnostic.