Skip to content
Snippets Groups Projects
  • Martin Kirchgessner's avatar
    4185f757
    New configuration option: use a compressed graph to build the folder structure. · 4185f757
    Martin Kirchgessner authored
    This abstracts methods needed for
    
    * folder structure creation, behind `GraphBackend`, and
    * BLOB access, behind `ObjBackend`.
    
    Former methods have been bundled in `WebApiBackend`, which implements both interfaces.
    
    This MR also introduces a new `GraphBackend`: `CompressedGraphBackend`. It can be enabled via configuration. It's accessed and tested via its gRPC interface - tests rely directly on data in `swh.graph`.
    
    Existing tests have been moved to `tests/graph`. I also changed the `fuse_mntdir` fixture so it starts FUSE as a thread instead of a sub-process. That way, we can see its log and exceptions' stacktraces if needed, which happens often as that FUSE process is what we're actually testing.
    
    The implementation is as robust as possible against the variation of data available at all nodes/edges, while keeping the possibility to plug different backends.
    4185f757
    History
    New configuration option: use a compressed graph to build the folder structure.
    Martin Kirchgessner authored
    This abstracts methods needed for
    
    * folder structure creation, behind `GraphBackend`, and
    * BLOB access, behind `ObjBackend`.
    
    Former methods have been bundled in `WebApiBackend`, which implements both interfaces.
    
    This MR also introduces a new `GraphBackend`: `CompressedGraphBackend`. It can be enabled via configuration. It's accessed and tested via its gRPC interface - tests rely directly on data in `swh.graph`.
    
    Existing tests have been moved to `tests/graph`. I also changed the `fuse_mntdir` fixture so it starts FUSE as a thread instead of a sub-process. That way, we can see its log and exceptions' stacktraces if needed, which happens often as that FUSE process is what we're actually testing.
    
    The implementation is as robust as possible against the variation of data available at all nodes/edges, while keeping the possibility to plug different backends.
This project manages its dependencies using pip. Learn more