Skip to content
GitLab
Explore
Sign in
Register
Primary navigation
Search or go to…
Project
S
swh-model
Manage
Activity
Members
Labels
Plan
Issues
Issue boards
Milestones
Wiki
Code
Merge requests
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Model registry
Operate
Environments
Monitor
Incidents
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
Platform
Development
swh-model
Commits
836198c4
Verified
Commit
836198c4
authored
6 years ago
by
Antoine R. Dumont
Browse files
Options
Downloads
Patches
Plain Diff
swh.model.hashutil: Remove unnecessary endpoints
parent
3b9e8e91
No related branches found
No related tags found
1 merge request
!12
model.hashutil: Open new endpoint to allow to hash stream
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
swh/model/hashutil.py
+9
-47
9 additions, 47 deletions
swh/model/hashutil.py
swh/model/tests/test_hashutil.py
+0
-21
0 additions, 21 deletions
swh/model/tests/test_hashutil.py
with
9 additions
and
68 deletions
swh/model/hashutil.py
+
9
−
47
View file @
836198c4
...
...
@@ -12,10 +12,6 @@ in a ValueError explaining the error.
This modules defines the following hashing functions:
- hash_stream: Hash the contents of something iterable (file, stream,
...
)
with
the
given
algorithms
(
defaulting
to
DEFAULT_ALGORITHMS
if
none
provided
).
- hash_file: Hash the contents of the given file object with the given
algorithms (defaulting to DEFAULT_ALGORITHMS if none provided).
...
...
@@ -233,24 +229,16 @@ def _new_hash(algo, length=None):
return
_new_hashlib_hash
(
algo
)
def
_read
(
fobj
):
"""
Wrapper function around reading a chunk from fobj.
"""
return
fobj
.
read
(
HASH_BLOCK_SIZE
)
def
hash_stream
(
s
,
readfn
=
_read
,
length
=
None
,
algorithms
=
DEFAULT_ALGORITHMS
,
chunk_cb
=
None
,
hash_format
=
'
bytes
'
):
"""
Hash the contents of a stream
def
hash_file
(
fobj
,
length
=
None
,
algorithms
=
DEFAULT_ALGORITHMS
,
chunk_cb
=
None
,
hash_format
=
'
bytes
'
):
"""
Hash the contents of the given file object with the given algorithms.
Args:
s: stream or object we can consume by successive call using `readfn`
readfn (fn): Function to read chunk data from s
length (int): the length of the contents of the object (for the
git-specific algorithms)
algorithms (set): the hashing algorithms to be used, as an
iterable over strings
fobj: a file-like object
length: the length of the contents of the file-like object (for the
git-specific algorithms)
algorithms: the hashing algorithms to be used, as an iterable over
strings
hash_format (str): Format required for the output of the
computed hashes (cf. HASH_FORMATS)
...
...
@@ -269,7 +257,7 @@ def hash_stream(s, readfn=_read, length=None, algorithms=DEFAULT_ALGORITHMS,
h
=
MultiHash
(
algorithms
,
length
)
while
True
:
chunk
=
readfn
(
s
)
chunk
=
fobj
.
read
(
HASH_BLOCK_SIZE
)
if
not
chunk
:
break
h
.
update
(
chunk
)
...
...
@@ -283,32 +271,6 @@ def hash_stream(s, readfn=_read, length=None, algorithms=DEFAULT_ALGORITHMS,
return
h
.
hexdigest
()
def
hash_file
(
fobj
,
length
=
None
,
algorithms
=
DEFAULT_ALGORITHMS
,
chunk_cb
=
None
,
hash_format
=
'
bytes
'
):
"""
Hash the contents of the given file object with the given algorithms.
Args:
fobj: a file-like object
length: the length of the contents of the file-like object (for the
git-specific algorithms)
algorithms: the hashing algorithms to be used, as an iterable over
strings
hash_format (str): Format required for the output of the
computed hashes (cf. HASH_FORMATS)
Returns: a dict mapping each algorithm to a digest (bytes by default).
Raises:
ValueError if:
algorithms contains an unknown hash algorithm.
hash_format is an unknown hash format
"""
return
hash_stream
(
fobj
,
length
=
length
,
algorithms
=
algorithms
,
chunk_cb
=
chunk_cb
,
hash_format
=
hash_format
)
def
hash_path
(
path
,
algorithms
=
DEFAULT_ALGORITHMS
,
chunk_cb
=
None
,
hash_format
=
'
bytes
'
,
track_length
=
True
):
"""
Hash the contents of the file at the given path with the given
...
...
This diff is collapsed.
Click to expand it.
swh/model/tests/test_hashutil.py
+
0
−
21
View file @
836198c4
...
...
@@ -125,27 +125,6 @@ class Hashutil(unittest.TestCase):
fobj
,
length
=
len
(
self
.
data
),
hash_format
=
'
bytehex
'
)
self
.
assertEqual
(
checksums
,
self
.
bytehex_checksums
)
@istest
def
hash_stream
(
self
):
class
StreamStub
:
def
__init__
(
self
,
data
):
self
.
data
=
data
def
iter_content
(
self
):
yield
from
io
.
BytesIO
(
self
.
data
)
s
=
StreamStub
(
self
.
data
).
iter_content
()
def
_readfn
(
s
):
try
:
return
next
(
s
)
except
StopIteration
:
return
None
checksums
=
hashutil
.
hash_stream
(
s
,
readfn
=
_readfn
,
length
=
len
(
self
.
data
),
hash_format
=
'
hex
'
)
self
.
assertEqual
(
checksums
,
self
.
hex_checksums
)
@istest
def
hash_file_missing_length
(
self
):
fobj
=
io
.
BytesIO
(
self
.
data
)
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment