utils.split_range: Make computed ranges not overlap
Existing listers use the is_within_bound [1] method from the base lister.
This method uses inclusive boundaries in all cases.
As some "range" task listers [2] [3] are using split_range function to create
"overlapping" ranges, this can be the concurrent insert issue cause we found [4].
This commit adapts the function split_range to make the generated ranges no
longer overlap.
-
[2] https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/gitlab/tasks.py$37-41
-
[3] https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/gitea/tasks.py$36-41
-
[4] https://sentry.softwareheritage.org/share/issue/aec9c2af347e47ea84f51ace3bfe2f25/
Related to #2577 (closed)
Test Plan
tox
Migrated from D3899 (view on Phabricator)
Merge request reports
Activity
Build has FAILED
Patch application report for D3899 (id=13743)
Could not rebase; Attempt merge onto 66a61f3d...
Updating 66a61f3..e407feb Fast-forward swh/lister/tests/test_utils.py | 37 +++++++++++++++++++++---------------- swh/lister/utils.py | 8 +++++--- 2 files changed, 26 insertions(+), 19 deletions(-)Changes applied before test
commit e407feb8c28b805db8ba220080d20f56d3287c50 Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Wed Sep 9 18:50:46 2020 +0200 utils.split_range: Make computed ranges not overlap Existing listers use the `is_within_bound` [1] method from the base lister. This method uses inclusive boundaries in all cases. As some "range" task listers [2] [3] are using `split_range` function to create "overlapping" ranges. So, when those range overlap, this can create concurrent insert issues down the line. This commit adapts the function `split_range` to make the generated ranges no longer overlap. - [1] https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/core/lister_base.py$194-199 - [2] https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/gitlab/tasks.py$37-41 - [3] https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/gitea/tasks.py$36-41 Related to #2577 commit 725c1fe4ad076c71d51d0c2998dcb4aaedd4b6bb Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Wed Sep 9 18:48:07 2020 +0200 test_utils: Migrate to pytestLink to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/41/ See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/41/console
-
Build is green
Patch application report for D3899 (id=13745)
Could not rebase; Attempt merge onto 66a61f3d...
Updating 66a61f3..6b880bc Fast-forward swh/lister/gitea/tests/test_tasks.py | 33 +++++++++++++------------------ swh/lister/gitlab/tests/test_tasks.py | 33 +++++++++++++------------------ swh/lister/tests/test_utils.py | 37 ++++++++++++++++++++--------------- swh/lister/utils.py | 21 +++++++++++++++++--- 4 files changed, 67 insertions(+), 57 deletions(-)Changes applied before test
commit 6b880bc00225ac05a94f001d6591ee394823a039 Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Wed Sep 9 18:50:46 2020 +0200 utils.split_range: Split into not overlapping ranges Existing listers use the `is_within_bound` [1] method from the base lister. This method uses inclusive boundaries in all cases. As some "range" task listers [2] [3] are using `split_range` function to create "overlapping" ranges. So, when those range overlap, this can create concurrent insert issues down the line. This commit adapts the function `split_range` to make the generated ranges no longer overlap. - [1] https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/core/lister_base.py$194-199 - [2] https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/gitlab/tasks.py$37-41 - [3] https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/gitea/tasks.py$36-41 Related to #2577 commit 725c1fe4ad076c71d51d0c2998dcb4aaedd4b6bb Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Wed Sep 9 18:48:07 2020 +0200 test_utils: Migrate to pytestSee https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/42/ for more details.
mentioned in merge request !158 (closed)
mentioned in merge request !331 (closed)
Some references in the commit message have been migrated:
- T2577 is now #2577 (closed)
-
Build is green
Patch application report for D3899 (id=13753)
Rebasing onto 725c1fe4...
Current branch diff-target is up to date.Changes applied before test
commit e3c856b5eef574427533fdb682163087337b2d8c Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org> Date: Wed Sep 9 18:50:46 2020 +0200 utils.split_range: Split into not overlapping ranges Existing listers use the `is_within_bound` [1] method from the base lister. This method uses inclusive boundaries in all cases. As some "range" task listers [2] [3] are using `split_range` function to create "overlapping" ranges, this can cause concurrent insert issues down the line [4]. This commit adapts the function `split_range` to make the generated ranges no longer overlap. - [1] https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/core/lister_base.py$194-199 - [2] https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/gitlab/tasks.py$37-41 - [3] https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/gitea/tasks.py$36-41 Related to #2577See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/43/ for more details.
mentioned in commit d1ce9b09
