Ensure azure blob storage do not serve uploaded tarballs with gzip encoding
as they are usually already compressed and uncompressed tarballs with wrong
extensions (.tgz instead .tar) are erroneously detected as gzip encoded by
django-storages due to the use of the mimetypes Python module.

Fixes #5589.

Checking deposit archives can be a costly operation as the checker
must download the archives to list their content.

It has been observed in production that if a large archive has been
uploaded with a deposit, requesting the check endpoint of the private
deposit API can end up with gunicorn worker being killed as the
time to download the archive exceeds the worker timeout.

So instead of using the private API endpoint performs the checks,
prefer to move these operations in the celery worker executing the
check-deposit task.

Fixes #4658.

Add a new private Web API endpoint to get a list of URLs for downloading
the tarballs uploaded with a deposit.

In development or docker mode, the tarballs are stored in the local
filesystem and served by django.

In production mode, the tarballs are stored in an azure blob storage
and temporary download links with a shared access signature are
generated when requesting the endpoint.

It enables to move costly operations related to downloading and processing
tarballs in celery workers instead of letting the deposit server performing
those tasks.

Related to #4657 and #4658.

Antoine Lambert authored 1 month ago and Antoine Lambert committed 4 weeks ago

This was used at the time we were building debian packages for
swh components but we no longer do that.

Use importlib.* equivalents instead.

Bump development tools: mypy, codespell, isort, ...

Move all tools configuration in pyproject.toml.

Remove no longer needed mypy overrides.

Numerous large tarballs stored in production azure blob storage fail
to be downloaded as the default connection timeout of 20 seconds is
reached and thus downloads are aborted, so bump azure connection
timeout to a larger value.

Related to #4654.

This class is used in the deposit CLI and should be usable without
the server part of swh-deposit installed.

This reverts commit cede455a as
these changes are no longer needed.

Django 5.x is used in production so we can remove the pinning to
inferior versions in tox environments.

When the django storage backend storing deposited tarballs is a remote
one (azure for instance), IncompleteRead errors can happen if the size
of the tarball to read is large.

Prefer to read tarball bytes by chunks instead to avoid these kind of
issues.

Fixes #4653.

Recent changes in swh-deposit codebase introduce cascading import of
swh.deposit.models.Deposit when importing DepositChecker class from
the swh.deposit.loader.checker module.

Importing swh.deposit.models.Deposit requires django to be properly
initialized which is usually not the case when the celery task modules
are imported through the swh_scheduler_celery_worker pytest fixture,
so numerous tests related to scheduler tasks in various swh packages
were failing.

So move import of DepositChecker class from top level to the task
function to avoid such side effect.

Previously, trying to execute the following commands:

- swh deposit upload ...
- swh deposit status ...
- swh deposit list ...
- swh deposit metadata-only ...

was returning the following error:

Could not load subcommand swh.deposit:
ImproperlyConfigured('Requested setting INSTALLED_APPS, but settings are not
configured. You must either define the environment variable DJANGO_SETTINGS_MODULE
or call settings.configure() before accessing settings.')

So ensure to init django at the beginning of each command to fix their usability.

Also prevent a similar side effect when swh.deposit.cli.client module is imported
by moving the import of swh.deposit.utils.NAMESPACES from top level to the function
it is used, this prevents the error message above to be displayed when using swh
command shell autocompletion feature.

PBKDF2 was taking about 30% of the runtime on my computer, possibly more on Jenkins.

This should fix timeouts we have been getting on Jenkins lately

The current logic would bypass the password check when the stored user
information was younger than `refresh_expires_at`, which is a pretty
long timeline.

The oldest part of the scheduler API was updated to use model classes
(based on attr package) instead of dictionaries in order to improve
typing.

_complete_deposit() schedules a loading task; but it was called before the
DepositRequest object was saved to the database.

This means that if the loading tasks executes very fast, or the DepositRequest
takes a while to save, then the loader won't see the current DepositRequest when
it queries the deposit's current state.

This is especially likely to happen with large code uploads, as saving them to
disk happens before the DepositRequest row is written to the database.

Related to swh/meta#5075.