Reset staging-next-version environment
It's been months since it's been cleaned up. So some data have accumulated there.
It'd nice to clean it up.
It's the occasion to also ensure db1 (dbs) and storage1 (objstorage) don't run extra kube services (adding missing affinities to chart they need to, e.g. kafka...).
[1] https://docs.softwareheritage.org/sysadm/deployment/howto-reset-next-version-environment.html
Activity
mentioned in issue #5505 (closed)
Closing mr [1]
mentioned in commit swh/infra/ci-cd/swh-charts@56059c45
mentioned in commit swh/infra/ci-cd/swh-charts@8ca7b531
mentioned in commit swh/infra/ci-cd/swh-charts@d3614ac2
-
The reset mostly went well, 2 issues to look up to:
- the reaper pod is crashlooping for some unknown reason
- the indexer storage db tool failed to initialize the db (i've unstuck it manually)
Reposted from [1]
-
the reaper pod is crashlooping for some unknown reason
Hit by this issue apparently [1]
[1] https://github.com/thelastpickle/cassandra-reaper/issues/1436
-
I tried to unstuck it but it's not working.
Retrieving the cql function to apply for the schema initialization of reaper from [1]
[1] https://github.com/thelastpickle/cassandra-reaper/issues/1436#issuecomment-2427411422
And the copy them on a writable folder in /config/patch (user: cassandra).
Connect via cqlsh in the cassandra cluster and apply the cql functions in the reaper ns [1] [2] [3]
And it did not help... The reaper pod are still crashing with the same error (about the function that got removed in cass 5).
There is one mr merged which allows to skip the migration step through a configuration key [4]. But it's not released just yet in the k8ssandra-operator repository... So i'm out of workaround.
[4] Now that the migration got done manually, we no longer need the init-container to do. I thought the init-container would be idempotent, so it would ignore the migration already done but no...
[1] To ease manipulation of cassandra tools
user=redacted pass=redacted alias nd='nodetool --username $user --password $pass' alias cqlsh='cqlsh --username $user --password $pass'[2] Copy the scripts to the pods
tony@kessel:~/work/swh/sysadm-environment/swh-charts (staging) $ namespace=swh-cassandra-next-version; pod=cassandra-cluster-next-version-default-sts-0; swh-kubectl stg cp ~/downloads/reaper-3.6.1/reaper-3.6.1-cql.txt $namespace/$pod:/config/patch/ + case "$1" in + context=archive-staging-rke2 + shift + kubectl --context archive-staging-rke2 cp /home/tony/downloads/reaper-3.6.1/reaper-3.6.1-cql.txt swh-cassandra-next-version/cassandra-cluster-next-version-default-sts-0:/config/patch/ Defaulted container "cassandra" out of: cassandra, server-system-logger, server-config-init-base (init), server-config-init (init) tony@kessel:~/work/swh/sysadm-environment/swh-charts (staging) $ namespace=swh-cassandra-next-version; pod=cassandra-cluster-next-version-default-sts-0; swh-kubectl stg cp ~/downloads/reaper-3.6.1/reaper-3.6.1-schema_migration.csv $namespace/$pod:/config/patch/ + case "$1" in + context=archive-staging-rke2 + shift + kubectl --context archive-staging-rke2 cp /home/tony/downloads/reaper-3.6.1/reaper-3.6.1-schema_migration.csv swh-cassandra-next-version/cassandra-cluster-next-version-default-sts-0:/config/patch/ Defaulted container "cassandra" out of: cassandra, server-system-logger, server-config-init-base (init), server-config-init (init)[3] Manage and execute the migration scripts ourselves
CREATE KEYSPACE IF NOT EXISTS cassandra_reaper WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 } AND DURABLE_WRITES = true; # Create the keyspace as the patched migration cqlsh assumes it's there and empty [cassandra@cassandra-cluster-next-version-default-sts-0 ~]$ cqlsh cassandra-cluster-superuser@cqlsh> CREATE KEYSPACE IF NOT EXISTS cassandra_reaper ... WITH REPLICATION = { ... 'class' : 'SimpleStrategy', 'replication_factor' : 1 ... } ... AND DURABLE_WRITES = true; cassandra-cluster-superuser@cqlsh> describe cassandra_reaper; CREATE KEYSPACE cassandra_reaper WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'} AND durable_writes = true; [cassandra@cassandra-cluster-next-version-default-sts-0 ~]$ cd /config/patch [cassandra@cassandra-cluster-next-version-default-sts-0 patch]$ cqlsh -f reaper-3.6.1-cql.txt Warning: Using a password on the command line interface can be insecure. Recommendation: use the credentials file to securely provide the password. # It's now populated cassandra-cluster-superuser@cqlsh> describe cassandra_reaper; CREATE KEYSPACE cassandra_reaper WITH replication = {'class': 'SimpleStrategy', 'replication_factor': '1'} AND durable_writes = true; CREATE TABLE cassandra_reaper.cluster ( name text PRIMARY KEY, last_contact timestamp, # Finally populate the schema migration (error message is about the first header line # trying to be ingested and failed) cassandra-cluster-superuser@cqlsh> COPY cassandra_reaper.schema_migration FROM 'reaper-3.6.1-schema_migration.csv'; Using 16 child processes Starting copy of cassandra_reaper.schema_migration with columns [applied_successful, version, executed_at, script, script_name]. Failed to import 1 rows: ParseError - Failed to parse version : invalid literal for int() with base 10: 'version', given up without retries Failed to process 1 rows; failed rows written to import_cassandra_reaper_schema_migration.err Processed: 18 rows; Rate: 10 rows/s; Avg. rate: 14 rows/s 17 rows imported from 1 files in 0 day, 0 hour, 0 minute, and 1.306 seconds (0 skipped). cassandra-cluster-superuser@cqlsh> select * from cassandra_reaper.schema_migration ... ; cassandra-cluster-superuser@cqlsh> select count(*) from cassandra_reaper.schema_migration limit 1; count ------- 17 (1 rows) Warnings : Aggregation query used without partition key
mentioned in commit swh/infra/ci-cd/swh-charts@5d968a30
-
the indexer storage db tool failed to initialize the db (i've unstuck it manually)
Well, i've destroyed it so i can have a stacktrace. And restarting the indexer-storage-rw (rpc) pod so it initializes the db, i've got this (that's what i had yesterday).
From that message, i gathered the message is right because the sql/ folder it's stored in the swh/indexer/storage part. But trying out various combinations of the swh db subcommands, i've got other kind of errors, none of which helped in initializing the schema.
[1]
2024-12-17T09:39:26.684882945Z version: None 2024-12-17T09:39:26.971801221Z ++ awk -F: '/code / {print $2}' /tmp/db-version.txt 2024-12-17T09:39:26.976546731Z + CODE_VERSION=' 137' 2024-12-17T09:39:26.977006011Z + CODE_VERSION=137 2024-12-17T09:39:26.977990661Z ++ awk -F: '/^version: / {print $2}' /tmp/db-version.txt 2024-12-17T09:39:26.980676760Z + DB_VERSION=' None' 2024-12-17T09:39:26.980804543Z + DB_VERSION=None 2024-12-17T09:39:26.980950567Z + '[' None = None ']' 2024-12-17T09:39:26.981012666Z + uri=postgresql://postgres:redacted@next-version-swh-indexer-rw:5432/swh-indexer 2024-12-17T09:39:26.981081725Z + swh db init-admin --db-name=postgresql://redacted@next-version-swh-indexer-rw:5432/swh-indexer indexer 2024-12-17T09:39:29.062369844Z WARNING:swh.core.sentry:Sentry DSN not provided, events will not be sent. 2024-12-17T09:39:30.161125556Z WARNING:swh.core.config:swh package does not yet declare the swh.indexer.classes endpoint. Make sure all your swh dependencies are up to date. 2024-12-17T09:39:30.182352882Z Traceback (most recent call last): 2024-12-17T09:39:30.182421156Z File "/opt/swh/.local/bin/swh", line 8, in <module> 2024-12-17T09:39:30.182434622Z sys.exit(main()) 2024-12-17T09:39:30.182444022Z File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/__init__.py", line 175, in main 2024-12-17T09:39:30.182453097Z return swh(auto_envvar_prefix="SWH") 2024-12-17T09:39:30.182461993Z File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__ 2024-12-17T09:39:30.182471589Z return self.main(*args, **kwargs) 2024-12-17T09:39:30.182478286Z File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1078, in main 2024-12-17T09:39:30.182485985Z rv = self.invoke(ctx) 2024-12-17T09:39:30.182493102Z File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke 2024-12-17T09:39:30.182499959Z return _process_result(sub_ctx.command.invoke(sub_ctx)) 2024-12-17T09:39:30.182506692Z File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke 2024-12-17T09:39:30.182513399Z return _process_result(sub_ctx.command.invoke(sub_ctx)) 2024-12-17T09:39:30.182519928Z File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke 2024-12-17T09:39:30.182526777Z return ctx.invoke(self.callback, **ctx.params) 2024-12-17T09:39:30.182533267Z File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 783, in invoke 2024-12-17T09:39:30.182540658Z return __callback(*args, **kwargs) 2024-12-17T09:39:30.182547482Z File "/opt/swh/.local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func 2024-12-17T09:39:30.182554098Z return f(get_current_context(), *args, **kwargs) 2024-12-17T09:39:30.182560975Z File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/db.py", line 226, in db_init_admin 2024-12-17T09:39:30.182567752Z init_admin_extensions(f"{package}:{cfg['cls']}", dbname) 2024-12-17T09:39:30.182575173Z File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/db/db_utils.py", line 639, in init_admin_extensions 2024-12-17T09:39:30.182605484Z sqlfiles = get_sql_for_package(modname) 2024-12-17T09:39:30.182613063Z File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/db/db_utils.py", line 533, in get_sql_for_package 2024-12-17T09:39:30.182619805Z raise ValueError( 2024-12-17T09:39:30.182627213Z ValueError: Module indexer:postgresql does not provide a db schema (no sql/ dir)[2] The toolbox is what we used to connect remotely to various backends, that's one location from which we can use the 'swh db' backend.
swh@swh-toolbox-774c789558-qwg27:~$ export SWH_CONFIG_FILENAME=/etc/swh/config-indexer-storage.yml swh@swh-toolbox-774c789558-qwg27:~$ cat $SWH_CONFIG_FILENAME indexer_storage: cls: postgresql db: host=next-version-swh-indexer-rw.swh-cassandra-next-version port=5432 user=swh-indexer dbname=swh-indexer password=redacted swh@swh-toolbox-774c789558-qwg27:~$ swh db shell indexer_storage INFO:swh.core.cli.db:Opening database shell for 'host=next-version-swh-indexer-rw.swh-cassandra-next-version port=5432 user=swh-indexer dbname=swh-indexer' psql (13.18 (Debian 13.18-0+deb11u1), server 16.2 (Debian 16.2-1.pgdg110+2)) WARNING: psql major version 13, server major version 16. Some psql features might not work. SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off) Type "help" for help. swh-indexer=> \d Did not find any relations. swh-indexer=> exit swh-indexer=> \df List of functions Schema | Name | Result data type | Argument data types | Type --------+------+------------------+---------------------+------ (0 rows) swh@swh-toolbox-774c789558-qwg27:~$ swh db version indexer Traceback (most recent call last): File "/opt/swh/.local/bin/swh", line 8, in <module> sys.exit(main()) File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/__init__.py", line 175, in main return swh(auto_envvar_prefix="SWH") File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/opt/swh/.local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/db.py", line 515, in db_version backends = handle_cmd_args( File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/db.py", line 873, in handle_cmd_args dbname, dbcfg = get_dburl_from_config(cfg.get(config_key or module, {})) File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/db.py", line 722, in get_dburl_from_config if cfg["cls"] == "pipeline": KeyError: 'cls' swh@swh-toolbox-774c789558-qwg27:~$ swh db version indexer.storage Traceback (most recent call last): File "/opt/swh/.local/bin/swh", line 8, in <module> sys.exit(main()) File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/__init__.py", line 175, in main return swh(auto_envvar_prefix="SWH") File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/opt/swh/.local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/db.py", line 515, in db_version backends = handle_cmd_args( File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/db.py", line 873, in handle_cmd_args dbname, dbcfg = get_dburl_from_config(cfg.get(config_key or module, {})) File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/db.py", line 722, in get_dburl_from_config if cfg["cls"] == "pipeline": KeyError: 'cls' swh@swh-toolbox-774c789558-qwg27:~$ swh db version -p indexer.storage Traceback (most recent call last): File "/opt/swh/.local/bin/swh", line 8, in <module> sys.exit(main()) File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/__init__.py", line 175, in main return swh(auto_envvar_prefix="SWH") File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/opt/swh/.local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/db.py", line 515, in db_version backends = handle_cmd_args( File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/db.py", line 865, in handle_cmd_args package, dbcfg, dbname = get_dburl_from_config_key(cfg, module) File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/db.py", line 749, in get_dburl_from_config_key cfg = cfg[swhmod] UnboundLocalError: local variable 'swhmod' referenced before assignment swh@swh-toolbox-774c789558-qwg27:~$ swh db version indexer_storage WARNING the database does not have a dbmodule table. module: indexer_storage:postgresql current code version: 137 version: None swh@swh-toolbox-774c789558-qwg27:~$ swh db version -a indexer_storage WARNING the database does not have a dbmodule table. module: indexer_storage:postgresql current code version: 137 version: None swh@swh-toolbox-774c789558-qwg27:~$ swh db version -a indexer swh@swh-toolbox-774c789558-qwg27:~$ swh db version -a indexer.storage -
Iterating over the previous errors, i unstuck the functions installation through [1] [2]
But then trying to just init the schema following the some pattern as the init-admin call, it plainly fails [3]
[1] fail
swh@swh-toolbox-774c789558-qwg27:~$ swh db init-admin --db-name=postgresql://postgres:redacted@next-version-swh-indexer-rw:5432/swh-indexer -p indexer.storage # did not work Traceback (most recent call last): File "/opt/swh/.local/bin/swh", line 8, in <module> sys.exit(main()) File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/__init__.py", line 175, in main return swh(auto_envvar_prefix="SWH") File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/opt/swh/.local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/db.py", line 216, in db_init_admin args = handle_cmd_args( File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/db.py", line 831, in handle_cmd_args raise ValueError("Cannot use both 'dbaname' and a specific config target") ValueError: Cannot use both 'dbaname' and a specific config target[2] Ok!
swh@swh-toolbox-774c789558-qwg27:~$ swh db init-admin --db-name=postgresql://postgres:redacted@next-version-swh-indexer-rw:5432/swh-indexer indexer_storage # worked swh@swh-toolbox-774c789558-qwg27:~$ swh db shell indexer_storage INFO:swh.core.cli.db:Opening database shell for 'host=next-version-swh-indexer-rw.swh-cassandra-next-version port=5432 user=swh-indexer dbname=swh-indexer' psql (13.18 (Debian 13.18-0+deb11u1), server 16.2 (Debian 16.2-1.pgdg110+2)) WARNING: psql major version 13, server major version 16. Some psql features might not work. SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off) Type "help" for help. swh-indexer=> \df List of functions Schema | Name | Result data type | Argument data types | Type --------+------------------------+------------------+----------------------------------------------------------------+------ public | armor | text | bytea | func public | armor | text | bytea, text[], text[] | func public | cash_dist | money | money, money | func public | crypt | text | text, text | func public | date_dist | integer | date, date | func public | dearmor | bytea | text | func public | decrypt | bytea | bytea, bytea, text | func public | decrypt_iv | bytea | bytea, bytea, bytea, text | func public | digest | bytea | bytea, text | func public | digest | bytea | text, text | func public | encrypt | bytea | bytea, bytea, text | func public | encrypt_iv | bytea | bytea, bytea, bytea, text | func public | float4_dist | real | real, real | func public | float8_dist | double precision | double precision, double precision | func public | gbt_bit_compress | internal | internal | func public | gbt_bit_consistent | boolean | internal, bit, smallint, oid, internal | func public | gbt_bit_penalty | internal | internal, internal, internal | func public | gbt_bit_picksplit | internal | internal, internal | func public | gbt_bit_same | internal | gbtreekey_var, gbtreekey_var, internal | func public | gbt_bit_union | gbtreekey_var | internal, internal | func public | gbt_bool_compress | internal | internal | func public | gbt_bool_consistent | boolean | internal, boolean, smallint, oid, internal | func public | gbt_bool_fetch | internal | internal | func public | gbt_bool_penalty | internal | internal, internal, internal | func public | gbt_bool_picksplit | internal | internal, internal | func public | gbt_bool_same | internal | gbtreekey2, gbtreekey2, internal | func public | gbt_bool_union | gbtreekey2 | internal, internal | func public | gbt_bpchar_compress | internal | internal | func public | gbt_bpchar_consistent | boolean | internal, character, smallint, oid, internal | func public | gbt_bytea_compress | internal | internal | func public | gbt_bytea_consistent | boolean | internal, bytea, smallint, oid, internal | func public | gbt_bytea_penalty | internal | internal, internal, internal | func public | gbt_bytea_picksplit | internal | internal, internal | func public | gbt_bytea_same | internal | gbtreekey_var, gbtreekey_var, internal | func public | gbt_bytea_union | gbtreekey_var | internal, internal | func public | gbt_cash_compress | internal | internal | func[2'] I've also scratched and started back the db to check this combination. It did not crash but it did nothing.
swh@swh-toolbox-774c789558-qwg27:~$ swh db init-admin --db-name=postgresql://postgres:redacted@next-version-swh-indexer-rw:5432/swh-indexer -a indexer # worked swh@swh-toolbox-774c789558-qwg27:~$ swh db shell indexer_storage INFO:swh.core.cli.db:Opening database shell for 'host=next-version-swh-indexer-rw.swh-cassandra-next-version port=5432 user=swh-indexer dbname=swh-indexer' psql (13.18 (Debian 13.18-0+deb11u1), server 16.2 (Debian 16.2-1.pgdg110+2)) WARNING: psql major version 13, server major version 16. Some psql features might not work. SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off) Type "help" for help. swh-indexer=> \df List of functions Schema | Name | Result data type | Argument data types | Type --------+------+------------------+---------------------+------ (0 rows)[3]
swh@swh-toolbox-774c789558-qwg27:~$ swh db init indexer_storage INFO:swh.core.cli.db:Initializing database version to 137 from the swh.indexer.storage datastore Error during database setup Command '['psql', '--quiet', '--no-psqlrc', '-v', 'ON_ERROR_STOP=1', '-d', 'host=next-version-swh-indexer-rw.swh-cassandra-next-version port=5432 user=swh-indexer dbname=swh-indexer password=gating-wharf-disown', '-f', '/opt/swh/.local/lib/python3.10/site-packages/swh/indexer/storage/sql/30-schema.sql']' returned non-zero exit status 3. Process output: psql:/opt/swh/.local/lib/python3.10/site-packages/swh/indexer/storage/sql/30-schema.sql:8: ERROR: permission denied for schema public Aborted! -
And that's how we were doing it and that's no longer working.
swh@swh-toolbox-774c789558-qwg27:~$ swh db init indexer --module-config-key indexer_storage Traceback (most recent call last): File "/opt/swh/.local/bin/swh", line 8, in <module> sys.exit(main()) File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/__init__.py", line 175, in main return swh(auto_envvar_prefix="SWH") File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/opt/swh/.local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/db.py", line 322, in db_init args = handle_cmd_args( File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/db.py", line 873, in handle_cmd_args dbname, dbcfg = get_dburl_from_config(cfg.get(config_key or module, {})) File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/db.py", line 722, in get_dburl_from_config if cfg["cls"] == "pipeline": KeyError: 'cls' swh@swh-toolbox-774c789558-qwg27:~$ swh db upgrade indexer --module-config-key indexer_storage WARNING:swh.core.config:swh package does not yet declare the swh.indexer.classes endpoint. Make sure all your swh dependencies are up to date. Usage: swh db upgrade [OPTIONS] MODULE Try 'swh db upgrade -h' for help. Error: Invalid value: Error: the given module (indexer) does not match the value stored in the database (indexer_storage:postgresql). -
So to unstuck this mess, i'm inserting the version since the schema got installed somehow.
swh@swh-toolbox-774c789558-qwg27:~$ swh db upgrade indexer --module-config-key indexer_storage WARNING:swh.core.config:swh package does not yet declare the swh.indexer.classes endpoint. Make sure all your swh dependencies are up to date. Usage: swh db upgrade [OPTIONS] MODULE Try 'swh db upgrade -h' for help. Error: Invalid value: Error: the given module (indexer) does not match the value stored in the database (indexer_storage:postgresql). swh@swh-toolbox-774c789558-qwg27:~$ swh db upgrade indexer_storage Traceback (most recent call last): File "/opt/swh/.local/bin/swh", line 8, in <module> sys.exit(main()) File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/__init__.py", line 175, in main return swh(auto_envvar_prefix="SWH") File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/opt/swh/.local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/db.py", line 711, in db_upgrade new_db_version = swh_db_upgrade(dbname, backend, go_to_version) File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/db/db_utils.py", line 173, in swh_db_upgrade raise ValueError("Unable to retrieve the current version of the database") ValueError: Unable to retrieve the current version of the database swh@swh-toolbox-774c789558-qwg27:~$ swh db init indexer_storage INFO:swh.core.cli.db:Initializing database version to 137 from the swh.indexer.storage datastore Error during database setup Command '['psql', '--quiet', '--no-psqlrc', '-v', 'ON_ERROR_STOP=1', '-d', 'host=next-version-swh-indexer-rw.swh-cassandra-next-version port=5432 user=swh-indexer dbname=swh-indexer password=gating-wharf-disown', '-f', '/opt/swh/.local/lib/python3.10/site-packages/swh/indexer/storage/sql/30-schema.sql']' returned non-zero exit status 3. Process output: psql:/opt/swh/.local/lib/python3.10/site-packages/swh/indexer/storage/sql/30-schema.sql:8: ERROR: type "sha1" already exists Aborted![2]
swh@swh-toolbox-774c789558-qwg27:~$ swh db shell indexer_storage INFO:swh.core.cli.db:Opening database shell for 'host=next-version-swh-indexer-rw.swh-cassandra-next-version port=5432 user=swh-indexer dbname=swh-indexer' psql (13.18 (Debian 13.18-0+deb11u1), server 16.2 (Debian 16.2-1.pgdg110+2)) WARNING: psql major version 13, server major version 16. Some psql features might not work. SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off) Type "help" for help. swh-indexer=> \d List of relations Schema | Name | Type | Owner --------+------------------------------------------+----------+------------- public | content_fossology_license | table | swh-indexer public | content_fossology_license_license_id_seq | sequence | swh-indexer public | content_metadata | table | swh-indexer public | content_mimetype | table | swh-indexer public | dbmodule | table | swh-indexer public | dbversion | table | swh-indexer public | directory_intrinsic_metadata | table | swh-indexer public | fossology_license | table | swh-indexer public | fossology_license_id_seq | sequence | swh-indexer public | indexer_configuration | table | swh-indexer public | indexer_configuration_id_seq | sequence | swh-indexer public | origin_extrinsic_metadata | table | swh-indexer public | origin_intrinsic_metadata | table | swh-indexer (13 rows) swh-indexer=> insert into dbversion values(137, now(), 'unstuck bootstrap'); INSERT 0 1 swh-indexer=> exit swh@swh-toolbox-774c789558-qwg27:~$ swh db upgrade indexer_storage No migration needed for 'indexer_storage:postgresql': the current version is 137 swh@swh-toolbox-774c789558-qwg27:~$ swh db version indexer_storage module: indexer_storage:postgresql current code version: 137 version: 137 swh@swh-toolbox-774c789558-qwg27:~$ swh db version indexer --module-config-key indexer_storage WARNING:swh.core.config:swh package does not yet declare the swh.indexer.classes endpoint. Make sure all your swh dependencies are up to date. module: indexer_storage:postgresql version: 137 # But some stuff are still messy swh@swh-toolbox-774c789558-qwg27:~$ swh db version indexer Traceback (most recent call last): File "/opt/swh/.local/bin/swh", line 8, in <module> sys.exit(main()) File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/__init__.py", line 175, in main return swh(auto_envvar_prefix="SWH") File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/opt/swh/.local/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/opt/swh/.local/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/db.py", line 515, in db_version backends = handle_cmd_args( File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/db.py", line 873, in handle_cmd_args dbname, dbcfg = get_dburl_from_config(cfg.get(config_key or module, {})) File "/opt/swh/.local/lib/python3.10/site-packages/swh/core/cli/db.py", line 722, in get_dburl_from_config if cfg["cls"] == "pipeline": KeyError: 'cls' swh@swh-toolbox-774c789558-qwg27:~$ swh db version -a indexer swh@swh-toolbox-774c789558-qwg27:~$ # does not crash but prints nothing ^ -
And for the indexer-storage-rw pod to start, i still have to fix an inconsistency in the dbmodule table [1]
Given it used to work, there is something that changed in the swh db submodule (since that worked as is before the changes).
I'm happy to change whatever needs to change in the charts but I don't really see a clear path. What might be the solution is renaming the module from 'indexer' to 'indexer_storage' but i'm not entirely sure.
[1]
swh@swh-toolbox-774c789558-qwg27:~$ swh db shell indexer_storage INFO:swh.core.cli.db:Opening database shell for 'host=next-version-swh-indexer-rw.swh-cassandra-next-version port=5432 user=swh-indexer dbname=swh-indexer' psql (13.18 (Debian 13.18-0+deb11u1), server 16.2 (Debian 16.2-1.pgdg110+2)) WARNING: psql major version 13, server major version 16. Some psql features might not work. SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off) Type "help" for help. swh-indexer=> select * from dbmodule ; dbmodule | single_row ----------------------------+------------ indexer_storage:postgresql | x (1 row) swh-indexer=> update dbmodule set dbmodule='indexer:postgresql'; UPDATE 1 swh-indexer=> exit -
I'm happy to change whatever needs to change in the charts but I don't really see a clear path. What might be the solution is renaming the module from 'indexer' to 'indexer_storage' but i'm not entirely sure.
I've destroyed again the db and started with the adaptation [1], it worked out of the box [2]
So i'll settle on this for now.
[1] swh/infra/ci-cd/swh-charts@93b420f3
[2]
swh@swh-toolbox-774c789558-qwg27:~$ export SWH_CONFIG_FILENAME=/etc/swh/config-indexer-storage.yml swh@swh-toolbox-774c789558-qwg27:~$ swh db shell indexer_storage INFO:swh.core.cli.db:Opening database shell for 'host=next-version-swh-indexer-rw.swh-cassandra-next-version port=5432 user=swh-indexer dbname=swh-indexer' psql (13.18 (Debian 13.18-0+deb11u1), server 16.2 (Debian 16.2-1.pgdg110+2)) WARNING: psql major version 13, server major version 16. Some psql features might not work. SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off) Type "help" for help. swh-indexer=> select * from dbmodule; dbmodule | single_row ----------------------------+------------ indexer_storage:postgresql | x (1 row) swh-indexer=> select * from dbversion; version | release | description ---------+-------------------------------+------------------- 137 | 2024-12-17 10:33:20.691177+00 | DB initialization (1 row)
mentioned in commit swh/infra/ci-cd/swh-charts@5ea14d08
mentioned in commit swh/infra/ci-cd/swh-charts@c473d073
mentioned in commit swh/infra/ci-cd/swh-charts@93b420f3
mentioned in commit swh/devel/swh-core@c9a23524
mentioned in commit swh/devel/swh-core@6368526c
mentioned in merge request swh/devel/swh-core!403 (merged)
Please register or sign in to reply