Django 5.x is used in production so we can remove the pinning to
inferior versions in tox environments.

When the django storage backend storing deposited tarballs is a remote
one (azure for instance), IncompleteRead errors can happen if the size
of the tarball to read is large.

Prefer to read tarball bytes by chunks instead to avoid these kind of
issues.

Fixes #4653.

Recent changes in swh-deposit codebase introduce cascading import of
swh.deposit.models.Deposit when importing DepositChecker class from
the swh.deposit.loader.checker module.

Importing swh.deposit.models.Deposit requires django to be properly
initialized which is usually not the case when the celery task modules
are imported through the swh_scheduler_celery_worker pytest fixture,
so numerous tests related to scheduler tasks in various swh packages
were failing.

So move import of DepositChecker class from top level to the task
function to avoid such side effect.

Previously, trying to execute the following commands:

- swh deposit upload ...
- swh deposit status ...
- swh deposit list ...
- swh deposit metadata-only ...

was returning the following error:

Could not load subcommand swh.deposit:
ImproperlyConfigured('Requested setting INSTALLED_APPS, but settings are not
configured. You must either define the environment variable DJANGO_SETTINGS_MODULE
or call settings.configure() before accessing settings.')

So ensure to init django at the beginning of each command to fix their usability.

Also prevent a similar side effect when swh.deposit.cli.client module is imported
by moving the import of swh.deposit.utils.NAMESPACES from top level to the function
it is used, this prevents the error message above to be displayed when using swh
command shell autocompletion feature.

PBKDF2 was taking about 30% of the runtime on my computer, possibly more on Jenkins.

This should fix timeouts we have been getting on Jenkins lately

The current logic would bypass the password check when the stored user
information was younger than `refresh_expires_at`, which is a pretty
long timeline.

The oldest part of the scheduler API was updated to use model classes
(based on attr package) instead of dictionaries in order to improve
typing.

_complete_deposit() schedules a loading task; but it was called before the
DepositRequest object was saved to the database.

This means that if the loading tasks executes very fast, or the DepositRequest
takes a while to save, then the loader won't see the current DepositRequest when
it queries the deposit's current state.

This is especially likely to happen with large code uploads, as saving them to
disk happens before the DepositRequest row is written to the database.

Related to swh/meta#5075.

Side effect of https://github.com/sissaschool/xmlschema/commit/cdae888.

Turns these references into simple links in order for the rst checks
performed by PyPI to pass when uploading a new package version.

As the known_first_party isort option was not set yet, previous
imports ordering got broken so restore it.

It was placed in mypy.plugins.django-stubs section so it was not
taken into account.

The former does not always plays well with PEP420...

vlorentz authored 1 year ago and vlorentz committed 1 year ago

It was deprecated years ago, and no client used it in staging or prod
since 2022-04-11

vlorentz authored 1 year ago and vlorentz committed 1 year ago

These are Codemeta terms, but they were used in the Atom namespace.
As they are not read by the tests using them, we can just remove them.

vlorentz authored 1 year ago and vlorentz committed 1 year ago

A future commit will reject these invalid tags