create-object-reference-partitions cronjob not working
The cronjob is failing because it seems it uses an objstorage configuration and volumes are not declared in the job configuration.
prepare-configuration Following variables will be replaced:
...
prepare-configuration HOSTNAME
Stream closed EOF for swh/create-object-reference-partitions-cronjob-manual-7hz-btzdm (prepare-configuration)
...
create-object-reference-partitions Running swh storage create-object-reference-partitions 2024-02-26 2024-03-26
create-object-reference-partitions Traceback (most recent call last):
create-object-reference-partitions File "/opt/swh/.local/bin/swh", line 8, in <module>
create-object-reference-partitions sys.exit(main())
create-object-reference-partitions File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/__init__.py", line 175, in main
create-object-reference-partitions return swh(auto_envvar_prefix="SWH")
create-object-reference-partitions File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
create-object-reference-partitions return self.main(*args, **kwargs)
create-object-reference-partitions File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1078, in main
create-object-reference-partitions rv = self.invoke(ctx)
create-object-reference-partitions File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
create-object-reference-partitions return _process_result(sub_ctx.command.invoke(sub_ctx))
create-object-reference-partitions File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
create-object-reference-partitions return _process_result(sub_ctx.command.invoke(sub_ctx))
create-object-reference-partitions File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
create-object-reference-partitions return ctx.invoke(self.callback, **ctx.params)
create-object-reference-partitions File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
create-object-reference-partitions return __callback(*args, **kwargs)
create-object-reference-partitions File "/opt/swh/.local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
create-object-reference-partitions return f(get_current_context(), *args, **kwargs)
create-object-reference-partitions File "/opt/swh/.local/lib/python3.10/site-packages/swh/storage/cli.py", line 355, in create_object_reference_partitions
create-object-reference-partitions storage = get_storage(**ctx.obj["config"]["storage"])
create-object-reference-partitions File "/opt/swh/.local/lib/python3.10/site-packages/swh/storage/__init__.py", line 62, in get_storage
create-object-reference-partitions return get_storage_pipeline(**kwargs)
create-object-reference-partitions File "/opt/swh/.local/lib/python3.10/site-packages/swh/storage/__init__.py", line 126, in get_storage_pipeline
create-object-reference-partitions return get_storage(**storage_config)
create-object-reference-partitions File "/opt/swh/.local/lib/python3.10/site-packages/swh/storage/__init__.py", line 81, in get_storage
create-object-reference-partitions storage = Storage(**kwargs)
create-object-reference-partitions File "/opt/swh/.local/lib/python3.10/site-packages/swh/storage/proxies/record_references.py", line 18, in __init__
create-object-reference-partitions self.storage: StorageInterface = get_storage(**storage)
create-object-reference-partitions File "/opt/swh/.local/lib/python3.10/site-packages/swh/storage/__init__.py", line 81, in get_storage
create-object-reference-partitions storage = Storage(**kwargs)
create-object-reference-partitions File "/opt/swh/.local/lib/python3.10/site-packages/swh/storage/postgresql/storage.py", line 275, in __init__
create-object-reference-partitions self.objstorage = ObjStorage(self, objstorage)
create-object-reference-partitions File "/opt/swh/.local/lib/python3.10/site-packages/swh/storage/objstorage.py", line 31, in __init__
create-object-reference-partitions self.objstorage = get_objstorage(**objstorage_config)
create-object-reference-partitions File "/opt/swh/.local/lib/python3.10/site-packages/swh/objstorage/factory.py", line 64, in get_objstorage
create-object-reference-partitions return ObjStorage(**kwargs)
create-object-reference-partitions File "/opt/swh/.local/lib/python3.10/site-packages/swh/objstorage/factory.py", line 75, in _construct_multiplexer_objstorage
create-object-reference-partitions storages = [get_objstorage(**conf) for conf in objstorages]
create-object-reference-partitions File "/opt/swh/.local/lib/python3.10/site-packages/swh/objstorage/factory.py", line 75, in <listcomp>
create-object-reference-partitions storages = [get_objstorage(**conf) for conf in objstorages]
create-object-reference-partitions File "/opt/swh/.local/lib/python3.10/site-packages/swh/objstorage/factory.py", line 64, in get_objstorage
create-object-reference-partitions return ObjStorage(**kwargs)
create-object-reference-partitions File "/opt/swh/.local/lib/python3.10/site-packages/swh/objstorage/backends/pathslicing.py", line 167, in __init__
create-object-reference-partitions self.check_config(check_write=False)
create-object-reference-partitions File "/opt/swh/.local/lib/python3.10/site-packages/swh/objstorage/backends/pathslicing.py", line 175, in check_config
create-object-reference-partitions raise ValueError(
create-object-reference-partitions ValueError: PathSlicingObjStorage root "/srv/softwareheritage/objects" is not a directory
Stream closed EOF for swh/create-object-reference-partitions-cronjob-manual-7hz-btzdm (create-object-reference-partitions)
The tables were manually created this morning via the toolbox
export SWH_CONFIG_FILENAME=etc/swh/config-storage.yml
swh storage create-object-reference-partitions "$(date +%Y-%m-%d)" "$(date -d '+1 month' +%Y-%m-%d)"
Created object references table for dates between 2024-02-26 and 2024-03-04
Created object references table for dates between 2024-03-04 and 2024-03-11
Created object references table for dates between 2024-03-11 and 2024-03-18
Created object references table for dates between 2024-03-18 and 2024-03-25
Created object references table for dates between 2024-03-25 and 2024-04-01
The loading failed this week between 2024-02-26 00:00 UTC to 2024-02-26 08:30 UTC (All loaders including save-code-now)
We should have some monitoring on the job failure or partition table presence
Edited by Vincent Sellier