Skip to content
Snippets Groups Projects

nixguix: Allow lister to ignore specific extensions

2 unresolved threads

Those extensions can be extended through configuration. They default to some binary format already encountered during docker runs.

Related to T3781 Depends on !450 (closed)


Migrated from D8763 (view on Phabricator)

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Build is green

    Patch application report for D8763 (id=31591)

    Could not rebase; Attempt merge onto 31eb5f63...

    Updating 31eb5f6..eea39c1
    Fast-forward
     swh/lister/nixguix/lister.py                       |  36 ++++++-
     swh/lister/nixguix/tests/data/sources-failure.json | 110 +++++++++++++++++++++
     swh/lister/nixguix/tests/data/sources-success.json |  94 +++++++++++++++++-
     swh/lister/nixguix/tests/test_lister.py            |  30 +++++-
     4 files changed, 264 insertions(+), 6 deletions(-)
    Changes applied before test
    commit eea39c1877b754a1653a786ad91b723c99ad4e2b
    Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
    Date:   Tue Oct 25 10:39:01 2022 +0200
    
        nixguix: Allow lister configuration to ignore specific extensions
        
        Related to [T3781](https://forge.softwareheritage.org/T3781 'view original for T3781 on Phabricator')
    
    commit 08c87b67f04585d5036ff85e1886fbbb518d9ec4
    Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
    Date:   Tue Oct 25 10:07:43 2022 +0200
    
        nixguix/test: Add all supported tarball extensions to test manifest
        
        Next step is to add some extensions filtering so might as well harden the test dataset
        first.
        
        Related to [T3781](https://forge.softwareheritage.org/T3781 'view original for T3781 on Phabricator')

    See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/822/ for more details.

  • LGTM, one thought posted as inline comment.

  • Merge request was accepted

  • Antoine Lambert approved this merge request

    approved this merge request

    • Rebase
    • Refactor common code on extension detection
    • Adapt docstring and commit message
    • Default extensions to ignore
    • Adapt constructor to extend extension to ignore if any is passed around
  • Rename variable parameter with a better name

  • Build was aborted

    Patch application report for D8763 (id=31595)

    Could not rebase; Attempt merge onto 4f6b3f3f...

    Updating 4f6b3f3..8c5c1b7
    Fast-forward
     swh/lister/nixguix/lister.py                       |  65 ++++++++++--
     swh/lister/nixguix/tests/data/sources-failure.json | 117 +++++++++++++++++++++
     swh/lister/nixguix/tests/data/sources-success.json |  94 ++++++++++++++++-
     swh/lister/nixguix/tests/test_lister.py            |  32 +++++-
     4 files changed, 295 insertions(+), 13 deletions(-)
    Changes applied before test
    commit 8c5c1b7ab8e282935073eb782546d60b8c89c76b
    Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
    Date:   Tue Oct 25 10:39:01 2022 +0200
    
        nixguix: Allow lister to ignore specific extensions
        
        Those extensions can be extended through configuration. They default to some binary
        format already encountered during docker runs.
        
        Related to [T3781](https://forge.softwareheritage.org/T3781 'view original for T3781 on Phabricator')
    
    commit d96a39d5b0569ae82bc15ccb09e25aa16e53e72c
    Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
    Date:   Tue Oct 25 10:07:43 2022 +0200
    
        nixguix/test: Add all supported tarball extensions to test manifest
        
        Next step is to add some extensions filtering so might as well harden the test dataset
        first.
        
        Related to [T3781](https://forge.softwareheritage.org/T3781 'view original for T3781 on Phabricator')

    Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/825/ See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/825/console

  • Build has FAILED

    Patch application report for D8763 (id=31596)

    Could not rebase; Attempt merge onto 4f6b3f3f...

    Updating 4f6b3f3..eddf45b
    Fast-forward
     swh/lister/nixguix/lister.py                       |  69 ++++++++++--
     swh/lister/nixguix/tests/data/sources-failure.json | 117 +++++++++++++++++++++
     swh/lister/nixguix/tests/data/sources-success.json |  94 ++++++++++++++++-
     swh/lister/nixguix/tests/test_lister.py            |  32 +++++-
     4 files changed, 299 insertions(+), 13 deletions(-)
    Changes applied before test
    commit eddf45b619b05ef299ee4cc3864401e89215f957
    Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
    Date:   Tue Oct 25 10:39:01 2022 +0200
    
        nixguix: Allow lister to ignore specific extensions
        
        Those extensions can be extended through configuration. They default to some binary
        format already encountered during docker runs.
        
        Related to [T3781](https://forge.softwareheritage.org/T3781 'view original for T3781 on Phabricator')
    
    commit d96a39d5b0569ae82bc15ccb09e25aa16e53e72c
    Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
    Date:   Tue Oct 25 10:07:43 2022 +0200
    
        nixguix/test: Add all supported tarball extensions to test manifest
        
        Next step is to add some extensions filtering so might as well harden the test dataset
        first.
        
        Related to [T3781](https://forge.softwareheritage.org/T3781 'view original for T3781 on Phabricator')

    Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/826/ See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/826/console

    • Add missing test
    • Fix another test
  • Build is green

    Patch application report for D8763 (id=31598)

    Could not rebase; Attempt merge onto 4f6b3f3f...

    Updating 4f6b3f3..77c2b87
    Fast-forward
     swh/lister/nixguix/lister.py                       |  75 ++++++++++---
     swh/lister/nixguix/tests/data/sources-failure.json | 117 +++++++++++++++++++++
     swh/lister/nixguix/tests/data/sources-success.json |  94 ++++++++++++++++-
     swh/lister/nixguix/tests/test_lister.py            |  48 ++++++++-
     4 files changed, 316 insertions(+), 18 deletions(-)
    Changes applied before test
    commit 77c2b875dc9a3290fb345e7658e4d57ca38dece3
    Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
    Date:   Tue Oct 25 10:39:01 2022 +0200
    
        nixguix: Allow lister to ignore specific extensions
        
        Those extensions can be extended through configuration. They default to some binary
        format already encountered during docker runs.
        
        Related to [T3781](https://forge.softwareheritage.org/T3781 'view original for T3781 on Phabricator')
    
    commit d96a39d5b0569ae82bc15ccb09e25aa16e53e72c
    Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
    Date:   Tue Oct 25 10:07:43 2022 +0200
    
        nixguix/test: Add all supported tarball extensions to test manifest
        
        Next step is to add some extensions filtering so might as well harden the test dataset
        first.
        
        Related to [T3781](https://forge.softwareheritage.org/T3781 'view original for T3781 on Phabricator')

    See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/827/ for more details.

271 297 # either full fqdn NixOS/nixpkgs or guix repository urls
272 298 # maybe add an assert on those specific urls?
273 299 self.origin_upstream = origin_upstream
300 self.extensions_to_ignore = DEFAULT_EXTENSIONS_TO_IGNORE + extensions_to_ignore
  • 37 37 logger = logging.getLogger(__name__)
    38 38
    39 39
    40 # By default, ignore binary files and archives containing binaries
  • Adapt according to last review (merge variable comment, drop spurious .copy)

  • Build is green

    Patch application report for D8763 (id=31600)

    Could not rebase; Attempt merge onto 4f6b3f3f...

    Updating 4f6b3f3..ca4ab7f
    Fast-forward
     swh/lister/nixguix/lister.py                       |  74 ++++++++++---
     swh/lister/nixguix/tests/data/sources-failure.json | 117 +++++++++++++++++++++
     swh/lister/nixguix/tests/data/sources-success.json |  94 ++++++++++++++++-
     swh/lister/nixguix/tests/test_lister.py            |  48 ++++++++-
     4 files changed, 315 insertions(+), 18 deletions(-)
    Changes applied before test
    commit ca4ab7f277dc51efc62dbb5c6866dc424d117d6a
    Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
    Date:   Tue Oct 25 10:39:01 2022 +0200
    
        nixguix: Allow lister to ignore specific extensions
        
        Those extensions can be extended through configuration. They default to some binary
        format already encountered during docker runs.
        
        Related to [T3781](https://forge.softwareheritage.org/T3781 'view original for T3781 on Phabricator')
    
    commit d96a39d5b0569ae82bc15ccb09e25aa16e53e72c
    Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
    Date:   Tue Oct 25 10:07:43 2022 +0200
    
        nixguix/test: Add all supported tarball extensions to test manifest
        
        Next step is to add some extensions filtering so might as well harden the test dataset
        first.
        
        Related to [T3781](https://forge.softwareheritage.org/T3781 'view original for T3781 on Phabricator')

    See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/828/ for more details.

  • Merge request was merged

  • Please register or sign in to reply
    Loading