vlorentz authored 1 year ago and vlorentz committed 7 months ago

They allow looking for high cardinality values without decompressing most stripes
not containing any (equality) match.

vlorentz authored 1 year ago and vlorentz committed 1 year ago

This lowers the number of orphan nodes.

Additionally, Kafka consumers occasionally hang forever on the .close() call after
getting offsets, and changing the other this way seems to fix it.

(And even if it does not fix the issue, this change makes them hang shortly after
the process started so it can be restarted quickly, instead of waiting hours before
it hangs.)

Related to swh/meta#5075.

As it might be easy to overlook that the dataset we provide are covered
by both an Ethical Charter and Terms of Usage for bulk access, put the
index behind a “I will respect the Ethical Charter and Terms of Use”.

This in done via some simple JavaScript that hides the index sections,
and add a button that will make them reappear on click. We remember
the click for next vists using the browser local storage.

pyorc 0.9.0 dropped support for py37.

Fixes 41fe4969.

boto3 has a dependency constraint `urllib2<2` which is clashing with
recent types-requests (see
https://github.com/python/typeshed/blob/main/stubs/requests/METADATA.toml)

This ensures dependencies of swh.dataset do not get hit by this clash,
making the dependecy solver of pip very slow (and selecting very old
versions of some of these dependencies).

To support the setup on Maxxi, in which the root directory is not deletable
by the Luigi process

It now needs types-click which is indeed a dependency of
swh.core[testing].