Skip to content

Swh components: make a default configuration file available for each of them

I understand that removing implicit configuration is the right way to go but giving easy access to default YAML configuration files synchronized with each configuration change will be a great helper here.

Indeed, I got back working on the NPM loader but I got stuck on the following swh-objstorage error that suddenly appeared:

Traceback (most recent call last):
  File "/home/antoine/swh/swh-environment/swh-loader-core/swh/loader/core/loader.py", line 894, in load
    self.store_data()
  File "/home/antoine/swh/swh-environment/swh-loader-npm/swh/loader/npm/loader.py", line 245, in store_data
    self.flush()
  File "/home/antoine/swh/swh-environment/swh-loader-core/swh/loader/core/loader.py", line 704, in flush
    self.send_batch_contents(contents)
  File "/home/antoine/swh/swh-environment/swh-loader-core/swh/loader/core/loader.py", line 647, in send_batch_contents
    packet_size_bytes=packet_size_bytes)
  File "/home/antoine/swh/swh-environment/swh-loader-core/swh/loader/core/loader.py", line 46, in send_in_packets
    sender(formatted_objects)
  File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/usr/lib/python3/dist-packages/retrying.py", line 206, in call
    return attempt.get(self._wrap_exception)
  File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/home/antoine/swh/swh-environment/swh-loader-core/swh/loader/core/loader.py", line 400, in send_contents
    self.storage.content_add(content_list)
  File "/home/antoine/swh/swh-environment/swh-storage/swh/storage/api/client.py", line 23, in content_add
    return self.post('content/add', {'content': content})
  File "/home/antoine/swh/swh-environment/swh-core/swh/core/api/__init__.py", line 185, in post
    return self._decode_response(response)
  File "/home/antoine/swh/swh-environment/swh-core/swh/core/api/__init__.py", line 220, in _decode_response
    raise pickle.loads(decode_response(response))
swh.core.api.RemoteException: Unexpected status code for API request: 413 (b'413: Request Entity Too Large')

It took me some time to understand that some changes were made in configuration and that my configuration should have been updated to the following (I was missing the client_max_size entry):

objstorage:
  cls: 'pathslicing'
  args: 
    root: '/home/antoine/swh/objects/'
    slicing: '0:2/2:4/4:6'

client_max_size: 8388608

Having the expected configuration clearly exposed somewhere would have greatly help me here.

So I think exposing default configuration for each swh component, maybe in a separate repository, could be useful.

Migrated from T1553 (view on Phabricator)

Edited by Phabricator Migration user