Skip to content

Rework from_disk.Directory.from_disk() implementation

David Douard requested to merge douardda/swh-model:rework-fromdisk into master

It used to traverse all the directory and filter elements of said tree only afterwards; this version should be a bit smarter and not go too far deep in directories that should be ignored.

We cannot just use the subtree eviction mechanism of 'os.walk(topdown=True)' because the filtering callback takes some context of the subdirectory content (typically to be able to evict empty directories).

This version of the code is a bit more complex but should do the trick.

One questionable point is about the behavior of the ignore_directories_patterns() filter (see the added test the revision 'Rework from_disk.Directory.from_disk() implementation' of this MR for more details)

Also comes with a small refactoring introducing a helper mk_tree() function to make tests involving a directory tree structure a bit easier to read.

On !358 (merged)

Merge request reports