This ensures the deposit checks in the elastic infra (using azure objstorage) is
working. There is currently no way to prevent the content-encoding from being detected
and stored in the blobstorage (various tryouts have been tested to no avail). That
content-encoding, is then preventing the checking/reading to work (failing [2]).

Following the documentation, a workaround has been implemented to reset a posteriori the
content-encoding [1].

It's not proper but, from repl experimentation, it's working which is better than the
actual failure.

[1] https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-properties-metadata-python

[2]
```
HttpResponseError: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing data: incorrect header check'))
```

Refs. swh/infra/sysadm-environment#5129

Use None instead of empty string to try to prevent content-encoding to be set when
writing.

Refs. swh/infra/sysadm-environment#5129

Currently, the writing in the azure objstorage adds extra headers to the blobs. Which
prevents the correct reading of those blobs when checking the tarballs.

According to the documentation, this option opening should allow to configure more
appropriately.

Refs. swh/infra/sysadm-environment#5129

If done on the full path, the conditional technically fails the check for the wrong
reason. This also adds tests on this case.

Refs. swh/infra/sysadm-environment#5129

Refs. swh/infra/sysadm-environment#5129

Similar adaptation of the deposit_check endpoint to allow reading remote tarballs when
using remote storage backends.

Refs. swh/infra/sysadm-environment#5129

Using the File api is consistent across various storage backends. The previous
implementation was only working for file storage. As we are migrating towards remote
backends, this no longer is enough.

Unfortunately though, we cannot rely on zipfile.is_zipfile or tarfile.is_tarfile as
those only works with local filepath. So we need to rely on testing & failing with
exception trapping to read properly the tarballs...

Refs. swh/infra/sysadm-environment#5129

They were only checking one tarball case (the ones reference in the ready_deposit_ok
deposit). This now creates as much deposit as there are extensions to check.

This only had to adapt the deposit so it's provided with a valid metadata. Otherwise,
the metadata checks were invalid, failing the overall tests.

Refs. swh/infra/sysadm-environment#5129

Refs. swh/infra/sysadm-environment#5129

This reverts commit bb22867c.

Not ready yet

This adapts the memcached import the same way the swh.web module did.

Follow up 9210c012

Refs. swh/infra/sysadm-environment#5129

This adapts the memcached import the same way the swh.web module did.

Follow up on 4698e8c9

Refs. swh/infra/sysadm-environment#5129

That uses django.db.models' by default, and fall back to the old contrib
one if need be.

This later has been deprecated for some time and is now producing
errors.

Doing this should work in any situation. The migration is actually only
a matter of the name of the type in the field metadata, so using a
stable one (swh.deposit.model.JSONField) make it backend-agnostic.

GitLab will display the content of the README file when browsing the
repository. But in case the file is a symlink, it will display the
path pointed by the symlink. There is a 6 year old issue about this:
https://gitlab.com/gitlab-org/gitlab/-/issues/15093

We can workaround the issue by having the content at the root of the
repository and a symlink to this file in the `docs/` directory.

Tested in swh/devel/swh-py-template!27

It also enables to remove the version restriction on psycopg2.

Related to swh/infra/sysadm-environment#4772

Jérémy Bobbio (Lunar) authored 2 years ago and Jérémy Bobbio (Lunar) committed 2 years ago

Related to swh/meta#4959

Related to swh/meta#4960

This fixes python 3.7 support due to poetry, a dependency of isort, that
removed support for that Python version in a recent release.

In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Also include them the proper sphinx way.

Related to T4496