Skip to content

various optimisation to the model validation logic.

Pierre-Yves David requested to merge generated-differential-D8512-source into generated-differential-D8512-target 
commit 313cc97581a06b3cfe8fc561d4e0b38ccccf152c (HEAD -> faster_validation, [D8512](https://forge.softwareheritage.org/D8512 'view original for D8512 on Phabricator'))
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Thu Sep 22 14:55:26 2022 +0200

    model: inline the call to `_check_swhid`
    
    This reduce the number of function call and should be faster.
    
    The mashup of blind optimisation in the previous changeset yield some
    interesting results in total.
    
    It would be insightful to measure them individually, but that would
    take more time than we currently have.
    
    When testing all the validator changes on our previous "benchmark" we
    see quite interesting improvement.
    
        swh loader run mercurial https://foss.heptapod.net/mercurial/mercurial-devel directory=/data/repos/mercurial-devel
    
    = Median time of 3 run =
    base:   17 minutes 48 seconds
    before: 11 minutes 50 seconds
    after:  11 minutes 11 seconds
    
    On a profile of the same run, the `to_model` call of the from_disk's `Directory` class took the following percentage:
    base:   43%
    before: 15%
    after:  11%

commit 54a19a2aa522787ec368f3707336824db7f13b76
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Thu Sep 22 14:48:03 2022 +0200

    model: optimization pass on custom validator
    
    (This commit is actually doing two things /o\)
    
    - we inline the type-checking in the custom validator to reduce the
      number of function call.
    
    - we optimize some of the custom validator by skipping the creation of
      intermediate tuples.

commit 14ba3c3e19f8c0d8505b4428aa158a594cca6f9d
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Thu Sep 22 11:50:39 2022 +0200

    model: delete unused validator code
    
    Since all `generic_type_validator` are optimized away, the code will no
    longer be called. So we remove that code to avoid any drifting.
    
    A nice "exception" is provided in case this start getting called again
    in the future.

commit b851355ccf71b4d66bfcf11025f40fe8de5275f5
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Thu Sep 22 11:40:40 2022 +0200

    model: remove the try/except
    
    Since try/except context are known to be expensive in Python, it seems
    useful to remove them.

commit 6add176754eefb6c7b37c7a0b21695ae1ae8f7b9
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Tue Sep 20 18:53:01 2022 +0200

    model: also optimize combined validator
    
    This ensure we don't have any remaining `generic_type_validator` call
    that have not been optimized away.

commit cc61b9d3860de28afc2eb23dc35da372fdaf1964
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Tue Sep 20 17:09:04 2022 +0200

    model: drop the `type_validator()` indirection
    
    This indirection seems useless and is probably the remains of some long
    forgotten rituals.

commit bf6fe4be7697ad032cca6901cf51e5c1cc481781
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Tue Sep 20 16:51:32 2022 +0200

    model: implement specialized attribute-validator functions
    
    This should reduces function calls and speeds things up.
    
    It might be useful to introduce even more specialized validator in the
    future. It would also be useful to skip the intermediate try/except.
    
    Some of this will be done in later changesets.

commit 1dfea32477023bec219334eb4cf4921203e76f78
Author: Pierre-Yves David <pierre-yves.david@ens-lyon.org>
Date:   Tue Sep 20 15:48:55 2022 +0200

    model: prepare the filtering of type_validator into something faster
    
    This is currently doing nothing, but prepare for actually changing the
    generic validator into faster specialized variants.

Test Plan

ran tox on each commit, timing and profiling on the full stack.

Migrated from D8512 (view on Phabricator)

Merge request reports