Postgresql backend unable to write a file directory entry with a huge name

The postgresql backend is not able to store a directory_entry_file which seems to have a big filename.
It's not detected by the model validator so it's stored in cassandra.
The mirrors will also be impacted when they reach this directory.
...
  File "/opt/swh/venv/lib/python3.11/site-packages/swh/journal/client.py", line 405, in process
    batch_processed, at_eof = self.handle_messages(messages, worker_fn)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/swh/venv/lib/python3.11/site-packages/swh/journal/client.py", line 444, in handle_messages
    worker_fn(dict(objects))
  File "/opt/swh/venv/lib/python3.11/site-packages/swh/storage/replay.py", line 196, in process_replay_objects
    _insert_objects(object_type, objects, storage)
  File "/opt/swh/venv/lib/python3.11/site-packages/swh/storage/replay.py", line 276, in _insert_objects
    method(objects)
  File "/opt/swh/venv/lib/python3.11/site-packages/swh/storage/proxies/buffer.py", line 237, in directory_add
    return self.flush(["raw_extrinsic_metadata", "content", "directory"])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/swh/venv/lib/python3.11/site-packages/swh/storage/proxies/buffer.py", line 365, in flush
    stats = add_fn(list(batch))
            ^^^^^^^^^^^^^^^^^^^
  File "/opt/swh/venv/lib/python3.11/site-packages/swh/storage/proxies/filter.py", line 79, in directory_add
    return self.storage.directory_add(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/swh/venv/lib/python3.11/site-packages/tenacity/__init__.py", line 336, in wrapped_f
    return copy(f, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^
  File "/opt/swh/venv/lib/python3.11/site-packages/tenacity/__init__.py", line 475, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/swh/venv/lib/python3.11/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/opt/swh/venv/lib/python3.11/site-packages/tenacity/__init__.py", line 418, in exc_check
    raise retry_exc.reraise()
          ^^^^^^^^^^^^^^^^^^^
  File "/opt/swh/venv/lib/python3.11/site-packages/tenacity/__init__.py", line 185, in reraise
    raise self.last_attempt.result()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/opt/swh/venv/lib/python3.11/site-packages/tenacity/__init__.py", line 478, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/opt/swh/venv/lib/python3.11/site-packages/swh/storage/proxies/retry.py", line 81, in newf
    return getattr(storage, attribute_name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/swh/venv/lib/python3.11/site-packages/swh/storage/postgresql/storage.py", line 158, in _meth
    return meth(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/swh/venv/lib/python3.11/site-packages/swh/core/db/common.py", line 73, in _meth
    return meth(self, *args, db=db, cur=cur, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/swh/venv/lib/python3.11/site-packages/swh/storage/postgresql/storage.py", line 730, in directory_add
    db.directory_add_from_temp(cur)
  File "/opt/swh/venv/lib/python3.11/site-packages/swh/core/db/db_utils.py", line 48, in _meth
    self._cursor(cur).execute("SELECT %s()" % stored_proc)
psycopg2.errors.ProgramLimitExceeded: index row size 6184 exceeds btree version 4 maximum 2704 for index "directory_entry_file_target_name_perms_idx"
DETAIL:  Index row references tuple (258717234,39) in relation "directory_entry_file".
HINT:  Values larger than 1/3 of a buffer page cannot be indexed.
Consider a function index of an MD5 hash of the value, or use full text indexing.
CONTEXT:  SQL statement "
    insert into directory_entry_file (target, name, perms)
    select distinct t.target, t.name, t.perms
    from tmp_directory_entry_file t
    where not exists (
      select 1
      from directory_entry_file i
      where t.target = i.target and t.name = i.name and t.perms = i.perms
    )
    on conflict
    do nothing
    ;
   "
PL/pgSQL function swh_directory_entry_add(directory_entry_type) line 3 at EXECUTE
SQL statement "SELECT swh_directory_entry_add('file')"
PL/pgSQL function swh_directory_add() line 3 at PERFORM
Postgresql backend unable to write a file directory entry with a huge name

Designs

Child items ...

Activity
