Skip to content
Snippets Groups Projects

Setup storage and store last revision/release date

Last revision/release dates are excellent candidates for sorting options and filters. These changes will store these values whenever an OriginVisitStatus with a valid snapshot id is received. The dates are fetched from swh.storage by passing the snapshot id. So these changes also configure swh.storage client in swh.search


Migrated from D5883 (view on Phabricator)

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Build has FAILED

    Patch application report for D5883 (id=21078)

    Could not rebase; Attempt merge onto 69364b3b...

    Auto-merging swh/search/tests/test_search.py
    Auto-merging swh/search/elasticsearch.py
    Merge made by the 'recursive' strategy.
     swh/search/elasticsearch.py             |  53 ++++++++++++
     swh/search/in_memory.py                 |  69 ++++++++++++++-
     swh/search/interface.py                 |   7 ++
     swh/search/journal_client.py            |  64 ++++++++++++++
     swh/search/tests/test_journal_client.py |   5 ++
     swh/search/tests/test_search.py         | 143 ++++++++++++++++++++++++++++++++
     6 files changed, 340 insertions(+), 1 deletion(-)
    Changes applied before test
    commit e1b4bd354c173a95f1006b44dc3b74134dbc7557
    Merge: 69364b3 b35a010
    Author: Jenkins user <jenkins@localhost>
    Date:   Thu Jun 17 10:46:48 2021 +0000
    
        Merge branch 'diff-target' into HEAD
    
    commit b35a01098f9c48d9a211b719a0afda9218fdac93
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Thu Jun 17 10:39:17 2021 +0000
    
        Add last_revision_daate and last_release_date
    
    commit 7506af8e63ea6a6db963118de599b553908d8596
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Thu Jun 17 09:04:13 2021 +0000
    
        Reset index at the end of permutation loops
    
    commit d67c56353231d2fa849c3aa044aee16b8330bedc
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Thu Jun 17 07:11:08 2021 +0000
    
        Fix scope of variables used in permutation test
    
    commit f0881fd5ebb9691deeb4c8695d1d1a8490f01afc
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Wed Jun 16 17:26:52 2021 +0000
    
        Add test for last_eventful_visit_date with permutations
    
    commit 140db0a3b7cf6f041bdb4f89b6e2d4c70b6abffb
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Wed Jun 16 08:15:31 2021 +0000
    
        Store last_eventful_visit_date

    Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/146/ See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/146/console

  • Author Contributor
    • Remove duplicate cls param for get_storage
  • Build has FAILED

    Patch application report for D5883 (id=21079)

    Could not rebase; Attempt merge onto 69364b3b...

    Auto-merging swh/search/tests/test_search.py
    Auto-merging swh/search/elasticsearch.py
    Merge made by the 'recursive' strategy.
     swh/search/elasticsearch.py             |  53 ++++++++++++
     swh/search/in_memory.py                 |  69 ++++++++++++++-
     swh/search/interface.py                 |   7 ++
     swh/search/journal_client.py            |  63 ++++++++++++++
     swh/search/tests/test_journal_client.py |   5 ++
     swh/search/tests/test_search.py         | 143 ++++++++++++++++++++++++++++++++
     6 files changed, 339 insertions(+), 1 deletion(-)
    Changes applied before test
    commit b1f59431eaf01a759cf6b48b16871a58d3da23f0
    Merge: 69364b3 617d6e1
    Author: Jenkins user <jenkins@localhost>
    Date:   Thu Jun 17 10:55:55 2021 +0000
    
        Merge branch 'diff-target' into HEAD
    
    commit 617d6e138f2c7016369f273c57d4946498aa6a87
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Thu Jun 17 10:54:48 2021 +0000
    
        FRemove duplicate cls param for get_storage
    
    commit b35a01098f9c48d9a211b719a0afda9218fdac93
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Thu Jun 17 10:39:17 2021 +0000
    
        Add last_revision_daate and last_release_date
    
    commit 7506af8e63ea6a6db963118de599b553908d8596
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Thu Jun 17 09:04:13 2021 +0000
    
        Reset index at the end of permutation loops
    
    commit d67c56353231d2fa849c3aa044aee16b8330bedc
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Thu Jun 17 07:11:08 2021 +0000
    
        Fix scope of variables used in permutation test
    
    commit f0881fd5ebb9691deeb4c8695d1d1a8490f01afc
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Wed Jun 16 17:26:52 2021 +0000
    
        Add test for last_eventful_visit_date with permutations
    
    commit 140db0a3b7cf6f041bdb4f89b6e2d4c70b6abffb
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Wed Jun 16 08:15:31 2021 +0000
    
        Store last_eventful_visit_date

    Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/147/ See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/147/console

  • Author Contributor
    • Fix failing tests (because of datetime parsing error in painless)
  • Build has FAILED

    Patch application report for D5883 (id=21080)

    Could not rebase; Attempt merge onto 69364b3b...

    Auto-merging swh/search/tests/test_search.py
    Auto-merging swh/search/elasticsearch.py
    Merge made by the 'recursive' strategy.
     swh/search/elasticsearch.py             |  53 ++++++++++++
     swh/search/in_memory.py                 |  69 ++++++++++++++-
     swh/search/interface.py                 |   7 ++
     swh/search/journal_client.py            |  63 ++++++++++++++
     swh/search/tests/test_journal_client.py |   5 ++
     swh/search/tests/test_search.py         | 143 ++++++++++++++++++++++++++++++++
     6 files changed, 339 insertions(+), 1 deletion(-)
    Changes applied before test
    commit 17aa25dff4462558b13f6ab1ae413d51f0f8e992
    Merge: 69364b3 2d8c990
    Author: Jenkins user <jenkins@localhost>
    Date:   Thu Jun 17 11:32:05 2021 +0000
    
        Merge branch 'diff-target' into HEAD
    
    commit 2d8c9903b042a66fc173f0cd0bd93246bf141df0
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Thu Jun 17 11:30:23 2021 +0000
    
        Fix failing tests
    
    commit 617d6e138f2c7016369f273c57d4946498aa6a87
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Thu Jun 17 10:54:48 2021 +0000
    
        FRemove duplicate cls param for get_storage
    
    commit b35a01098f9c48d9a211b719a0afda9218fdac93
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Thu Jun 17 10:39:17 2021 +0000
    
        Add last_revision_daate and last_release_date
    
    commit 7506af8e63ea6a6db963118de599b553908d8596
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Thu Jun 17 09:04:13 2021 +0000
    
        Reset index at the end of permutation loops
    
    commit d67c56353231d2fa849c3aa044aee16b8330bedc
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Thu Jun 17 07:11:08 2021 +0000
    
        Fix scope of variables used in permutation test
    
    commit f0881fd5ebb9691deeb4c8695d1d1a8490f01afc
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Wed Jun 16 17:26:52 2021 +0000
    
        Add test for last_eventful_visit_date with permutations
    
    commit 140db0a3b7cf6f041bdb4f89b6e2d4c70b6abffb
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Wed Jun 16 08:15:31 2021 +0000
    
        Store last_eventful_visit_date

    Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/148/ See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/148/console

  • Author Contributor
    • Adjust tests related to previous commit
  • Build is green

    Patch application report for D5883 (id=21081)

    Could not rebase; Attempt merge onto 69364b3b...

    Auto-merging swh/search/tests/test_search.py
    Auto-merging swh/search/elasticsearch.py
    Merge made by the 'recursive' strategy.
     swh/search/elasticsearch.py             |  53 ++++++++++++
     swh/search/in_memory.py                 |  69 ++++++++++++++-
     swh/search/interface.py                 |   7 ++
     swh/search/journal_client.py            |  63 ++++++++++++++
     swh/search/tests/test_journal_client.py |   5 ++
     swh/search/tests/test_search.py         | 143 ++++++++++++++++++++++++++++++++
     6 files changed, 339 insertions(+), 1 deletion(-)
    Changes applied before test
    commit 2d0bf5ca8ed808d5e0668f9303ea7477b1ac7f99
    Merge: 69364b3 0a5cfcf
    Author: Jenkins user <jenkins@localhost>
    Date:   Thu Jun 17 11:43:38 2021 +0000
    
        Merge branch 'diff-target' into HEAD
    
    commit 0a5cfcf2cb3eac34cf0203b5aeccd8ee2c06a7cd
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Thu Jun 17 11:43:02 2021 +0000
    
        Adjust tests related to previous commit
    
    commit 2d8c9903b042a66fc173f0cd0bd93246bf141df0
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Thu Jun 17 11:30:23 2021 +0000
    
        Fix failing tests
    
    commit 617d6e138f2c7016369f273c57d4946498aa6a87
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Thu Jun 17 10:54:48 2021 +0000
    
        FRemove duplicate cls param for get_storage
    
    commit b35a01098f9c48d9a211b719a0afda9218fdac93
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Thu Jun 17 10:39:17 2021 +0000
    
        Add last_revision_daate and last_release_date
    
    commit 7506af8e63ea6a6db963118de599b553908d8596
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Thu Jun 17 09:04:13 2021 +0000
    
        Reset index at the end of permutation loops
    
    commit d67c56353231d2fa849c3aa044aee16b8330bedc
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Thu Jun 17 07:11:08 2021 +0000
    
        Fix scope of variables used in permutation test
    
    commit f0881fd5ebb9691deeb4c8695d1d1a8490f01afc
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Wed Jun 16 17:26:52 2021 +0000
    
        Add test for last_eventful_visit_date with permutations
    
    commit 140db0a3b7cf6f041bdb4f89b6e2d4c70b6abffb
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Wed Jun 16 08:15:31 2021 +0000
    
        Store last_eventful_visit_date

    See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/149/ for more details.

  • Author Contributor
    • Load swh.storage from config
  • Author Contributor

    Squash commits

  • Author Contributor
    • Store last_eventful_visit_date
    • Add last_revision_daate and last_release_date
  • Author Contributor

    Rebase and only include new commit

  • Build is green

    Patch application report for D5883 (id=21114)

    Rebasing onto 35570b42...

    First, rewinding head to replay your work on top of it...
    Applying: Store last_eventful_visit_date
    Using index info to reconstruct a base tree...
    M	swh/search/elasticsearch.py
    M	swh/search/in_memory.py
    M	swh/search/interface.py
    M	swh/search/journal_client.py
    M	swh/search/tests/test_journal_client.py
    M	swh/search/tests/test_search.py
    Falling back to patching base and 3-way merge...
    Auto-merging swh/search/tests/test_search.py
    Auto-merging swh/search/elasticsearch.py
    No changes -- Patch already applied.
    Applying: Add last_revision_daate and last_release_date
    Changes applied before test
    commit 2bfb347ec05e6a414b7a642a73f6f6eb196e5e2d
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Thu Jun 17 10:39:17 2021 +0000
    
        Add last_revision_daate and last_release_date
        
        FRemove duplicate cls param for get_storage
        
        Fix failing tests
        
        Adjust tests related to previous commit
        
        Load swh.storage from config

    See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/152/ for more details.

  • Build is green

    Patch application report for D5883 (id=21115)

    Rebasing onto 35570b42...

    Current branch diff-target is up to date.
    Changes applied before test
    commit fe74c7e858310cba1232ca9885a38d3940a773e3
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Thu Jun 17 10:39:17 2021 +0000
    
        Add last_revision_daate and last_release_date
        
        FRemove duplicate cls param for get_storage
        
        Fix failing tests
        
        Adjust tests related to previous commit
        
        Load swh.storage from config

    See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/153/ for more details.

  • Merge request was returned for changes

  • (cosmetic) Also, careful on the commit message / diff description typo ;)

  • Author Contributor
    • merge the fetch_last_*_date methods
    • Use snapshot_get_all_branches instead of snapshot_get_branches
    • Fix diff/commit description/message
    • Use in-memory backend in test_cli.py
  • Build is green

    Patch application report for D5883 (id=21128)

    Rebasing onto 35570b42...

    Current branch diff-target is up to date.
    Changes applied before test
    commit 835edc7e5b1efd21d8486b228252a1d1056ebc92
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Thu Jun 17 10:39:17 2021 +0000
    
        Setup storage and store last revision/release date
        
        Last revision/release dates are excellent candidates for sorting options and filters. These changes will store these values whenever an OriginVisitStatus with a valid snapshot id is received. The dates are fetched from swh.storage by passing the snapshot id. So these changes also configure swh.storage client in swh.search

    See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/154/ for more details.

  • Author Contributor
    • Include last revision/release date only when available
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
Please register or sign in to reply
Loading