Skip to content
Snippets Groups Projects

Add random directory sampling policy

Closed Add random directory sampling policy
generated-differential-D8539-source into master
Closed Raphaël Gomès requested to merge generated-differential-D8539-source into master
  1. Jan 06, 2023
    • Raphaël Gomès's avatar
      Add random directory sampling policy · 326618a5
      Raphaël Gomès authored 
      This makes use of the new discovery algorithm introduced in
``swh-loader-core``, which should help speed up large (think Linux
kernel or way larger) scans.

Most of the time is spend walking the on-disk directory and hashing,
which is where the new optimizations in ``swh-model==6.5.0`` should come
in handy. Python is close to its limit in that regard, some future
endeavor should look into setting up SWH for native extensions.
      326618a5