Skip to content
Snippets Groups Projects
New look: Off

Support disordered directory entries in git

    • Closed Issue created by vlorentz @vlorentz

    Some git directories don't have their entries in the right order, which is an issue for us because:

    • swh-model reorders them before checksuming
    • the postgresql storage does not preserve the relative order of file and dir entries
    • the cassandra storage does not preserve order at all (they are sorted by null-padded name)

    Relevant issue to consider to regarding Cassandra: swh/devel/swh-storage#3582 (closed)

    Sentry: SWH-LOADER-GIT-QF

    Migrated from T3595 (view on Phabricator)

    Edited
    To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information

    Child items 0

    • No child items are currently assigned. Use child items to break down this issue into smaller parts.

    Activity

    • vlorentz mentioned in issue #3586 (closed)

      mentioned in issue #3586 (closed)

    • Phabricator Migration user marked this issue as related to #3594 (closed)

      marked this issue as related to #3594 (closed)

    • vlorentz added Data Model Storage manager meta-task priority:Normal labels

      added Data Model Storage manager meta-task priority:Normal labels

    • vlorentz changed the description

      changed the description

    • vlorentz
      vlorentz @vlorentz ·
      Author Maintainer

      Possible solution: store a rank along with each directory entry, but ignore it unless we are reconstructing a git object or computing a SWHID (v1?)

    • vlorentz
      vlorentz @vlorentz ·
      Author Maintainer

      Complete proposal to implement the above solution:

      1. add this to swh.model.model.DirectoryEntry (Directory.from_dict would set the rank if it's missing):
      rank = attr.ib(type=int, validators=type_validator())
"""Zero-based index of this entry in a directory."""

      1. In postgresql, change the type of dir_entries/file_entries/rev_entries from bigint[] to bigint[2][]: pairs of (directory_entry object id, rank). The migration would be: 1. duplicate columns and make python write to both but read only the old 2. fill the new columns (looooong) 3. drop the old columns, rename and make python use the new ones

      2. in cassandra, add column rank to directory_entry table. We can initialize them at 0 and make the Python code fill it when reading. (we can also have a script fill it, but it's not mandatory)

    • vlorentz changed the description

      changed the description

    • Antoine R. Dumont
      Antoine R. Dumont @ardumont ·
      Maintainer

      I came across a rather small repository [1] which i believe raise the same issue. So it may help to keep its reference to ease the testing of the improvment discussed here. Feel free to dismiss if not that useful.

      swh-loader_1                     | [2021-10-22 11:47:39,586: INFO/MainProcess] Task swh.loader.git.tasks.UpdateGitRepository[3b8b9037-f344-44e5-ab97-a3a310d9214f] received
swh-loader_1                     | [2021-10-22 11:47:39,632: INFO/ForkPoolWorker-1] Load origin 'https://github.com/technoweenie/attachment_fu' with type 'git'
swh-loader_1                     | Enumerating objects: 3549, done.
swh-loader_1                     | Total 3549 (delta 0), reused 0 (delta 0), pack-reused 3549
swh-loader_1                     | [2021-10-22 11:47:41,486: INFO/ForkPoolWorker-1] Listed 23 refs for repo https://github.com/technoweenie/attachment_fu
swh-loader_1                     | [2021-10-22 11:47:42,063: ERROR/ForkPoolWorker-1] Loading failure, updating to `failed` status
swh-loader_1                     | Traceback (most recent call last):
swh-loader_1                     |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/core/loader.py", line 339, in load
swh-loader_1                     |     self.store_data()
swh-loader_1                     |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/core/loader.py", line 457, in store_data
swh-loader_1                     |     for directory in self.get_directories():
swh-loader_1                     |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/git/loader.py", line 376, in get_directories
swh-loader_1                     |     yield converters.dulwich_tree_to_directory(raw_obj)
swh-loader_1                     |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/git/converters.py", line 104, in dulwich_tree_to_directory
swh-loader_1                     |     check_id(dir_)
swh-loader_1                     |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/git/converters.py", line 39, in check_id
swh-loader_1                     |     f"Expected {type(obj).__name__} hash to be {obj.id.hex()}, "
swh-loader_1                     | swh.loader.git.converters.HashMismatch: Expected Directory hash to be 6ac72a0858a5d5028d7f502de8777fbd5bdb8cae, got e23127f28dd0e1cf6e92a7b81cb9dbc53b44aa37
      $ time git clone https://github.com/technoweenie/attachment_fu
Cloning into 'attachment_fu'...
remote: Enumerating objects: 1737, done.
remote: Total 1737 (delta 0), reused 0 (delta 0), pack-reused 1737
Receiving objects: 100% (1737/1737), 377.44 KiB | 3.85 MiB/s, done.
Resolving deltas: 100% (740/740), done.
git clone https://github.com/technoweenie/attachment_fu  0.05s user 0.02s system 8% cpu 0.837 total
$ du -sh attachment_fu
964K    attachment_fu
    • vlorentz
      vlorentz @vlorentz ·
      Author Maintainer

      We decided to store manifests instead. #3594 (closed)

    • vlorentz assigned to @vlorentz

      assigned to @vlorentz

    • vlorentz added state:wontfix label

      added state:wontfix label

    • vlorentz closed

      closed

    Please register or sign in to reply