Related T1061

Summary: When mkdtemp is called, shutil.rmtree must be called as well

Test Plan:
Look at /tmp before and after running tests, notice no new
directories instead of 60.

Reviewers: #reviewers!

Differential Revision: https://forge.softwareheritage.org/D331

Summary:
The behavior of storage when the underlying objstorage had an exception was
never actually tested. This new test weeded out a bug in the threaded
implementation for copy_to.

Test Plan: the new test passes

Reviewers: #reviewers!

Differential Revision: https://forge.softwareheritage.org/D330

This allows connection reuse for postgresql and potential remote backends such
as for the object storage, rather than reinitiating all connections on every
request.

Summary: This allows to use swh.storage with a modicum of concurrency

Test Plan: clearly, make test should still pass

Reviewers: #reviewers!

Differential Revision: https://forge.softwareheritage.org/D325

Summary:
Add a level of indirection to allow swapping out the implementation of
the db attribute

Test Plan: once again, make test keeps on working

Reviewers: #reviewers!

Differential Revision: https://forge.softwareheritage.org/D324

Summary:
This avoids reusing a potentially stale connection handle. Also allows testing
potential connection pooling behavior.

This forces us to do proper cursor sanitation as well, a bunch of
"transactional" operations weren't actually transactional.

Test Plan: another round of make test still working

Reviewers: #reviewers!

Differential Revision: https://forge.softwareheritage.org/D323

Summary: Helps avoid lingering postgresql connections when a test fails

Test Plan: make test still works ;)

Reviewers: #reviewers!

Differential Revision: https://forge.softwareheritage.org/D322

Related T1036

This commit adds the implementation of an efficient algorithm for comparing
two directory trees in order to compute the list of introduced file changes
in terms of addition / deletion / modification/ renaming.
It can be found in the diff module located in the new namespace
swh.storage.algos

That algorithm is used to extend the storage API with the following methods:

  - diff_directories: compute diff between two arbitrary directories
  - diff_revisions: compute diff between two arbitrary revisions
  - diff_revision: compute diff between a revision and its first parent

Related T921

Closes D295

This table allows counting objects by bucket, keeping the transactions for
counting objects short (a few dozen seconds at most).

Also add a "single_update" boolean field to the main object_counts table to be
able to discriminate tables that are counted via buckets and tables counted on
one go.

The main table is updated every 256 counted buckets to avoid too much churn on
the table.

Close T962.

This function does _not_ have any backwards compatibility provisions.

this way doc (re)building is more robust

Related T918

Related 08121f71

rationale: swh-init.sql should only contain stuff that requires being Postgres
superuser

Summary:
Avoids writing a bunch of temporary files for no good reason. T866 was a symptom
of this.

Test Plan: No regressions in the unit tests

Reviewers: #reviewers!

Differential Revision: https://forge.softwareheritage.org/D282

swh-enums: Reference new mercurial revision type

Related T329

the new sphinx seems to be confused by relative symlinks when we merge
all docs together

was: "swh"

this way we can ship the images as is in the global doc, with minimal clash risks