In all case I will make another mr as an improvment of this one for progressive approach.
To be sure of template time computation I've added a counter and runned it with linux kernel repo, it takes 1m27 for computation and discovery then 12s (9s on heptapod not 20 as I was saying in previous comment) rendering results page. Faster but still long on very large repos.
Actually cloning the linux kernel to see how it behave against a big one. I've gone the simplest and naïve way for sure. Need to try things for an incremental/progressive approach but I guess it at least meant using javascript for rendering plus a json endpoint to retrieve children of a directory. Will experiment this way and report results here.
Franck Bret (9abf1e55) at 27 Mar 10:09
dashboard: Replace Dash dashboard with a Flask one
... and 3 more commits
Franck Bret (18eb6402) at 27 Mar 10:00
Franck Bret (916fee0e) at 27 Mar 10:00
cli: set exclude templates through configuration file
Exclude templates list can now be set within global or per project configuration file through ["scanner"]["exclude_templates"].
Franck Bret (18eb6402) at 27 Mar 09:55
cli: set exclude templates through configuration file
... and 1 more commit
Franck Bret (5393f9cb) at 27 Mar 09:50
Franck Bret (a835490d) at 27 Mar 09:50
cli: Add progress information to scanner scan command
Add a generic callback method for the scan command to retrieve and display progress information. It takes a context argument to distinguish the caller. Adapt the existing policy discovery callback.
The progress information for now is shown for the following steps:
Depends on swh-model!344
It actually get the whole source_tree/nodes_data results and render the whole tree in the html dom through recursive Jinja templating.
The biggest sources I've tried without exclusion for now is https://foss.heptapod.net/heptapod/heptapod
I takes 1min on my machine computing swhid before saying the dashboard is ready. When dashboard is ready clicking results take approximatively 20s before displaying results. Adding progress loader can help waiting.
I can sure check on bigger one and report some metrics.
The previous dashboard was using Dash with Pandas dataframe and solely consists of rendering a sunburst chart of known files. It wasn't really useful especially with big source tree. What is at least needed is the ability to browse the results with a file browser like pattern. We also want to have a simpler and more maintainable stack with less dependencies.
Here is a first attempt using a simple Flask app using Jinja templates, vanilla css, vanilla js.
It renders two routes for now:
Beautifulsoup has been added to requirements-test so we can test some parts of the dom rendering. For now the plots, dash, pandas dataframe things are kept as long as we keep 'sunburst' as an option of cli scan output-format.
Some screenshots:
Franck Bret (f8d60028) at 27 Mar 08:47
dashboard: Replace Dash dashboard with a Flask one
... and 6 more commits
Fixes and rebased. Ok for merging it?
@douardda Can we merge this one or do you have some more feedback?
Franck Bret (5c889615) at 05 Mar 10:46
cli: set exclude templates through configuration file
... and 1 more commit
Franck Bret (5393f9cb) at 05 Mar 10:44
cli: display progress information while the swh scanner scan
runs
... and 1 more commit
Franck Bret (a1e09490) at 04 Mar 13:26
cli: set exclude templates through configuration file
... and 1 more commit
Good catch, that line needs one more indentation. Fixed.
There is the case where the command line option exclude_templates is None, but a configuration file can have set it.