Skip to content
Snippets Groups Projects

translator.py: Translate search query language to ES DSL

4 unresolved threads

Translate swh search query language queries into Elasticsearch DSL

Depends on !64 (closed)


Migrated from D6025 (view on Phabricator)

Merge request reports

Closed by Phabricator Migration userPhabricator Migration user 3 years ago (Jul 30, 2021 1:03pm UTC)

Merge details

  • The changes were not merged into .

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Build has FAILED

    Patch application report for D6025 (id=21782)

    Could not rebase; Attempt merge onto 2edbbbe8...

    Updating 2edbbbe..757b046
    Fast-forward
     query_language/grammar.js                   |  28 +++--
     query_language/test/corpus/combinations.txt |  13 ++-
     swh/search/tests/test_translator.py         | 111 +++++++++++++++++++
     swh/search/translator.py                    | 161 ++++++++++++++++++++++++++++
     4 files changed, 300 insertions(+), 13 deletions(-)
     create mode 100644 swh/search/tests/test_translator.py
     create mode 100644 swh/search/translator.py
    Changes applied before test
    commit 757b0462d0bd11e8663c1e1d5f3de0b0ddc07773
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Tue Jul 27 00:18:11 2021 +0530
    
        translator.py: Translate search query language to ES DSL
        
        Summary:
        
        Translate swh search query language queries into Elasticsearch DSL
        
        Depends on !64
        
        Test Plan:
        
        Reviewers:
        
        Subscribers:
    
    commit c27cb9a0a4d6a657e8ba02ab0f705bb77cc9b6ee
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Tue Jul 13 16:21:51 2021 +0530
    
        grammar.js: Segregate sort_by and limit from filters
        
        Summary:
        The grammar should not allow using sort_by and limit more than once throughout
        the query.
        
        Unlike other filters, these two must not be concatenated by 'and' or 'or'
        
        Reviewers: #reviewers
        
        Differential Revision: https://forge.softwareheritage.org/!64

    Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/236/ See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/236/console

  • you should use raise NotImplementedError("blah blah blah") instead of return "TODO", it makes it clearer that you reach an unimplemented case, and which one.

  • Author Contributor
    • translator.py: Translate search query language to ES DSL
    • translator.py: Complete all the filters and add new tests
    • translator.py: Import swh_ql.so using pkg_resources
  • Build has FAILED

    Patch application report for D6025 (id=21787)

    Rebasing onto 05efa541...

    Current branch diff-target is up to date.
    Changes applied before test
    commit 715a2eb2206cd946bd3ad5fdc87b1cee3fb885ba
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Tue Jul 27 17:55:31 2021 +0530
    
        translator.py: Import swh_ql.so using pkg_resources
    
    commit c3174265485ee9758507fd2fc41f52f46d35dd09
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Tue Jul 27 00:58:31 2021 +0530
    
        translator.py: Complete all the filters and add new tests
    
    commit 3d030756b3f63082148253d44df06eb0ea1f447a
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Tue Jul 27 00:18:11 2021 +0530
    
        translator.py: Translate search query language to ES DSL
        
        Translate swh search query language queries into Elasticsearch DSL

    Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/238/ See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/238/console

  • Author Contributor
    • translator: Try fixing swh_ql.so path
  • Build has FAILED

    Patch application report for D6025 (id=21803)

    Rebasing onto 05efa541...

    Current branch diff-target is up to date.
    Changes applied before test
    commit 16c254dc569b0d0cc8adb9ad1e6f70ad1ab39759
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Tue Jul 27 21:20:47 2021 +0530
    
        translator: Try fixing swh_ql.so path
    
    commit 715a2eb2206cd946bd3ad5fdc87b1cee3fb885ba
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Tue Jul 27 17:55:31 2021 +0530
    
        translator.py: Import swh_ql.so using pkg_resources
    
    commit c3174265485ee9758507fd2fc41f52f46d35dd09
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Tue Jul 27 00:58:31 2021 +0530
    
        translator.py: Complete all the filters and add new tests
    
    commit 3d030756b3f63082148253d44df06eb0ea1f447a
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Tue Jul 27 00:18:11 2021 +0530
    
        translator.py: Translate search query language to ES DSL
        
        Translate swh search query language queries into Elasticsearch DSL

    Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/239/ See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/239/console

  • Author Contributor
    • translator.py: Fix resource path for tox environment
  • Build is green

    Patch application report for D6025 (id=21805)

    Rebasing onto 05efa541...

    Current branch diff-target is up to date.
    Changes applied before test
    commit 905889ddf0f3f4676ed841c325774d3a6a72cef8
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Tue Jul 27 17:55:31 2021 +0530
    
        translator.py: Fix resource path for tox environment
    
    commit c3174265485ee9758507fd2fc41f52f46d35dd09
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Tue Jul 27 00:58:31 2021 +0530
    
        translator.py: Complete all the filters and add new tests
    
    commit 3d030756b3f63082148253d44df06eb0ea1f447a
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Tue Jul 27 00:18:11 2021 +0530
    
        translator.py: Translate search query language to ES DSL
        
        Translate swh search query language queries into Elasticsearch DSL

    See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/240/ for more details.

  • _traverse should raise errors instead of silently ignoring unexpected nodes.

  • Author Contributor
    • translator: Changes suggested by vlorentz
  • Build is green

    Patch application report for D6025 (id=21821)

    Rebasing onto 05efa541...

    Current branch diff-target is up to date.
    Changes applied before test
    commit 3575aac1c04744a934ef5b2ff5309b1fa084e03b
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Wed Jul 28 14:44:00 2021 +0530
    
        translator: Changes suggested by vlorentz
    
    commit 905889ddf0f3f4676ed841c325774d3a6a72cef8
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Tue Jul 27 17:55:31 2021 +0530
    
        translator.py: Fix resource path for tox environment
    
    commit c3174265485ee9758507fd2fc41f52f46d35dd09
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Tue Jul 27 00:58:31 2021 +0530
    
        translator.py: Complete all the filters and add new tests
    
    commit 3d030756b3f63082148253d44df06eb0ea1f447a
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Tue Jul 27 00:18:11 2021 +0530
    
        translator.py: Translate search query language to ES DSL
        
        Translate swh search query language queries into Elasticsearch DSL

    See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/241/ for more details.

  • Author Contributor

    Rebase:

    • translator.py: Translate search query language to ES DSL
    • translator.py: Complete all the filters and add new tests
    • translator.py: Fix resource path for tox environment
    • translator: Changes suggested by vlorentz
    • translator.py: Update list of possible ql paths (suggested by vlorentz)
  • Build is green

    Patch application report for D6025 (id=21857)

    Rebasing onto c34011b7...

    Current branch diff-target is up to date.
    Changes applied before test
    commit 3dab4f5d16d6f40e8ee4d1052211ab799183e3af
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Thu Jul 29 14:12:10 2021 +0530
    
        translator.py: Update list of possible ql paths
    
    commit 847c5fa366445bab38096c2979ab74a11f876f06
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Wed Jul 28 14:44:00 2021 +0530
    
        translator: Changes suggested by vlorentz
    
    commit c0b78c26c12c085062240b0d65969f04c89f9f6b
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Tue Jul 27 17:55:31 2021 +0530
    
        translator.py: Fix resource path for tox environment
    
    commit 82a2dec905ecc66d1201687e17abd12312a5b038
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Tue Jul 27 00:58:31 2021 +0530
    
        translator.py: Complete all the filters and add new tests
    
    commit 7d8feb2ab0373c4f05e6fbaf1371fe817fbc2a9f
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Tue Jul 27 00:18:11 2021 +0530
    
        translator.py: Translate search query language to ES DSL
        
        Translate swh search query language queries into Elasticsearch DSL

    See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/243/ for more details.

  • vlorentz
    vlorentz @vlorentz started a thread on the diff
272 }
273 }
274 return {
275 "range": {
276 f"{name}_date": {
277 self.RANGE_OPERATOR_MAP[op]: value.replace("Z", "+00:00"),
278 }
279 }
280 }
281
282 if category == "sortBy":
283 return value
284
285 if category == "limit":
286 return value
287
  • vlorentz
    vlorentz @vlorentz started a thread on the diff
  • 73 return {"bool": {"must": [filters1, filters2]}}
    74 if conj_op == "or":
    75 return {"bool": {"should": [filters1, filters2]}}
    76
    77 if node.type == "filter":
    78 filter_category = node.children[0]
    79 return self._parse_filter(filter_category)
    80
    81 if node.type == "sortBy":
    82 return self._parse_filter(node)
    83
    84 if node.type == "limit":
    85 return self._parse_filter(node)
    86
    87 return Exception(
    88 f"Unknown node type ({node.type}) "
  • Author Contributor
    • translator.py: Mention filter category and name in exception
    • Squash
  • Build is green

    Patch application report for D6025 (id=21865)

    Rebasing onto c34011b7...

    Current branch diff-target is up to date.
    Changes applied before test
    commit f67cc169e66e1d30aaf41c3544b2d523480cc823
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Fri Jul 30 00:19:21 2021 +0530
    
        translator.py: Mention filter category and name in exception
    
    commit 2d596405d7571ef15ba0f19f7ed3dfd6b88ad1cb
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Tue Jul 27 00:18:11 2021 +0530
    
        translator.py: Translate search query language to ES DSL
        
        Translate swh search query language queries into Elasticsearch DSL

    See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/244/ for more details.

  • Author Contributor

    translator.py: Mention number of children while throwing exception in _traverse

  • Build is green

    Patch application report for D6025 (id=21868)

    Rebasing onto c34011b7...

    Current branch diff-target is up to date.
    Changes applied before test
    commit 734370ade926508fa14d31595b27429794f2f52c
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Tue Jul 27 00:18:11 2021 +0530
    
        translator.py: Translate search query language to ES DSL
        
        Translate swh search query language queries into Elasticsearch DSL

    See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/245/ for more details.

  • vlorentz
    vlorentz @vlorentz started a thread on the diff
  • 152 "nested": {
    153 "path": "intrinsic_metadata",
    154 "query": {
    155 "multi_match": {
    156 "query": r"""word1 word2 \" ' word3""",
    157 "fields": [
    158 get_expansion("keywords", ".") + "^2",
    159 get_expansion("descriptions", "."),
    160 ],
    161 }
    162 },
    163 }
    164 }
    165 }
    166
    167 _test_results(query, expected)
  • vlorentz
    vlorentz @vlorentz started a thread on the diff
  • 13 "<": "lt",
    14 ">=": "gte",
    15 "<=": "lte",
    16 }
    17
    18 def __init__(self):
    19 ql_rel_paths = [
    20 "swh_ql.so", # installed
    21 "../../query_language/swh_ql.so", # development
    22 ]
    23 for ql_rel_path in ql_rel_paths:
    24 ql_path = resource_filename("swh.search", ql_rel_path)
    25 if os.path.exists(ql_path):
    26 break
    27 else:
    28 assert False, "swh_ql.so was not found in any of the expected paths"
  • Merge request was accepted

  • vlorentz approved this merge request

    approved this merge request

  • Author Contributor
    • Changes suggested by vlorentz
      • Improve keyword filter test using raw string
      • Improve swh_ql.so not found error message
  • Build is green

    Patch application report for D6025 (id=21870)

    Rebasing onto c34011b7...

    Current branch diff-target is up to date.
    Changes applied before test
    commit 3428e71a042b6e8b78de5c34c2fcc8cb1cae68ea
    Author: KShivendu <shivendu@iitbhilai.ac.in>
    Date:   Tue Jul 27 00:18:11 2021 +0530
    
        translator.py: Translate search query language to ES DSL
        
        Translate swh search query language queries into Elasticsearch DSL

    See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/246/ for more details.

  • Author Contributor

    Merge request was merged

  • Please register or sign in to reply
    Loading