Deploy swh-scheduler v1.9.1

Changes since v1.9.0:

commit 006d60c1c6152f41a2be9f57a125fc9af32fae18 (HEAD -> master, tag: v1.9.1, origin/master, origin/HEAD)
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Tue Jun 13 11:07:58 2023 +0000

    sql: Fix task creation when providing a custom next_run value

    Previously, it was not possible to create sequentially two oneshot tasks
    whose only differ by their next_run value.

    Related to swh/devel/swh-web#4548

commit 04a22075478142552382630e6956c8f179495e75
Author: vlorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 7 14:43:47 2023 +0000

    cli: Use ctx.fail instead of raising an exception

That version is required to deploy next version of swh-web adding cooldown for webhook requests (swh/devel/swh-web!1141 (merged)).

staging:

• scheduler0: Update package
• scheduler-db: Apply migration script (34)
• systemctl restart gunicorn-swh-scheduler.service
• w/ anlambert: Checks (deployed the new swh-web and trigger check from a webhook)

production:

• Deactivate puppet (so the previous packaging of swh-web is not installed automatically)
• scheduler0: Update package
• Drop old function whose signature got changed after a rename (a while back)
• scheduler-db: Apply migration script (34)
• systemctl restart gunicorn-swh-scheduler.service
• Reactivate puppet
• Checks

added Scheduling utilities activity::Deployment priority:Normal labels

changed the description

assigned to @ardumont

Current version:

swhscheduler@scheduler0:~$ swh db --config-file /etc/softwareheritage/scheduler/backend.yml version scheduler
module: scheduler
current code version: 34
version: 33

Of course, that won't work. There is an extra file which the db cli does not support. [1]

[1]

swhscheduler@scheduler0:~$ swh db --config-file /etc/softwareheritage/scheduler/backend.yml upgrade scheduler
Traceback (most recent call last):
  File "/usr/bin/swh", line 33, in <module>
    sys.exit(load_entry_point('swh.core==2.22.2', 'console_scripts', 'swh')())
  File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 144, in main
    return swh(auto_envvar_prefix="SWH")
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3/dist-packages/swh/core/cli/db.py", line 418, in db_upgrade
    new_db_version = swh_db_upgrade(dbname, module, to_version)
  File "/usr/lib/python3/dist-packages/swh/core/db/db_utils.py", line 181, in swh_db_upgrade
    for fname in get_sql_for_package(modname, upgrade=True)
  File "/usr/lib/python3/dist-packages/swh/core/db/db_utils.py", line 182, in <listcomp>
    if db_version < int(fname.stem) <= to_version
ValueError: invalid literal for int() with base 10: '30-bis'

arf

I'm fixing it ;p

Although i can bypass it too but might as well fixing it for later. [1]

[1] using the good ol' way (psql ;p)

Patching so the db core utility works, i now have a real issue with the sql migration script packaged though. [1] I did not investigate it yet (i'm fixing the other shenanigan).

[1]

swhscheduler@scheduler0:~$ swh db --config-file /etc/softwareheritage/scheduler/backend.yml upgrade scheduler
No migration needed: the current version is 34
swhscheduler@scheduler0:~$ swh db --config-file /etc/softwareheritage/scheduler/backend.yml upgrade scheduler
INFO:swh.core.db.db_utils:Executing migration 34 with script '/usr/lib/python3/dist-packages/swh/scheduler/sql/upgrades/34.sql'
psql:/usr/lib/python3/dist-packages/swh/scheduler/sql/upgrades/34.sql:50: ERROR:  cannot change return type of existing function
HINT:  Use DROP FUNCTION swh_scheduler_create_tasks_from_temp() first.
Traceback (most recent call last):
  File "/usr/bin/swh", line 33, in <module>
    sys.exit(load_entry_point('swh.core==2.22.2', 'console_scripts', 'swh')())
  File "/usr/lib/python3/dist-packages/swh/core/cli/__init__.py", line 144, in main
    return swh(auto_envvar_prefix="SWH")
  File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3/dist-packages/swh/core/cli/db.py", line 418, in db_upgrade
    new_db_version = swh_db_upgrade(dbname, module, to_version)
  File "/usr/lib/python3/dist-packages/swh/core/db/db_utils.py", line 207, in swh_db_upgrade
    execute_sqlfiles([sqlfile], conninfo, db_flavor)
  File "/usr/lib/python3/dist-packages/swh/core/db/db_utils.py", line 685, in execute_sqlfiles
    subprocess.check_call(psql_command + ["-f", str(sqlfile)])
  File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['psql', '--quiet', '--no-psqlrc', '-v', 'ON_ERROR_STOP=1', '-d', 'host=db1.internal.staging.swh.network port=5432 dbname=swh-scheduler user=swh-scheduler password=4QQBg4jzjs', '-f', '/usr/lib/python3/dist-packages/swh/scheduler/sql/upgrades/34.sql']' returned non-zero exit status 3.

Weird, I cannot reproduce locally:

(swh) ✔ ~/swh/swh-environment/swh-scheduler [master|✔] 
15:01 $ dropdb softwareheritage-dev 

(swh) ✔ ~/swh/swh-environment/swh-scheduler [v1.9.0|✔] 
15:01 $ swh db -C ~/.config/swh/scheduler/scheduler.yml create scheduler
WARNING:swh.core.db.db_utils:Grant read-only access to guest user failed. Skipping.

(swh) ✔ ~/swh/swh-environment/swh-scheduler [v1.9.0|✔] 
15:01 $ swh db -C ~/.config/swh/scheduler/scheduler.yml init scheduler
WARNING:swh.core.db.db_utils:Grant read-only access to guest user failed. Skipping.
psql:/home/anlambert/swh/swh-environment/swh-core/swh/core/db/sql/36-dbmodule.sql:12: NOTICE:  relation "dbmodule" already exists, skipping
WARNING:swh.core.db.db_utils:Grant read-only access to guest user failed. Skipping.
INFO:swh.core.cli.db:Initializing database version to 33 from the scheduler datastore
DONE database for scheduler initialized at version 33

(swh) ✔ ~/swh/swh-environment/swh-scheduler [v1.9.0|✔] 
15:01 $ git checkout master 
Previous HEAD position was cf2ca93 cli.add_forge_now: Allow queue name prefix override
Switched to branch 'master'
Your branch is up to date with 'origin/master'.

(swh) ✔ ~/swh/swh-environment/swh-scheduler [master|✔] 
15:02 $ swh db -C ~/.config/swh/scheduler/scheduler.yml upgrade scheduler
INFO:swh.core.db.db_utils:Executing migration script '/home/anlambert/swh/swh-environment/swh-scheduler/swh/scheduler/sql/upgrades/34.sql'
WARNING:swh.core.db.db_utils:Grant read-only access to guest user failed. Skipping.
Migration to version 34 done

(swh) ✔ ~/swh/swh-environment/swh-scheduler [master|✔] 
15:02 $ swh db -C ~/.config/swh/scheduler/scheduler.yml upgrade scheduler
No migration needed: the current version is 34

(swh) ✔ ~/swh/swh-environment/swh-scheduler [master|✔] 
15:02 $ swh db -C ~/.config/swh/scheduler/scheduler.yml upgrade scheduler
No migration needed: the current version is 34

For the record, I patched the db_utils.py file the following way:

diff --git a/swh/core/db/db_utils.py b/swh/core/db/db_utils.py
index b738fb8..643b7af 100644
--- a/swh/core/db/db_utils.py
+++ b/swh/core/db/db_utils.py
@@ -179,7 +179,7 @@ def swh_db_upgrade(
     sqlfiles = [
         fname
         for fname in get_sql_for_package(modname, upgrade=True)
-        if db_version < int(fname.stem) <= to_version
+        if db_version < int(fname.stem.split("-")[0]) <= to_version
     ]
 
     if not sqlfiles:

Yes, i had something similar to your patch (it was taking a bit longer 'cause i added some tests) [3].

Anyway, it's a live environment only related problem ;p In the staging db, the result data type is indeed different than the new function does [1] (and it will be the same on production)

It must be because of an old migration when we stopped using oneshot tasks (and we "archived" the table tasks) to make some room.

Anyway, I'll fix that [1].

[1]

2023-06-13 15:06:00 swh-scheduler@db1:5432 λ \df+ swh_scheduler_create_tasks_from_temp
List of functions
+-[ RECORD 1 ]--------+----------------------------------------------------------------------------------+
| Schema              | public                                                                           |
| Name                | swh_scheduler_create_tasks_from_temp                                             |
| Result data type    | SETOF archive_task                                                               | <---- here 
| Argument data types |                                                                                  |
| Type                | func                                                                             |
| Volatility          | volatile                                                                         |
| Parallel            | unsafe                                                                           |
| Owner               | swh-scheduler                                                                    |
| Security            | invoker                                                                          |
| Access privileges   |                                                                                  |
| Language            | plpgsql                                                                          |
| Source code         |                                                                                 +|
|                     | begin                                                                           +|
|                     |   -- update the default values in one go                                        +|
|                     |   -- this is separated from the insert/select to avoid too much                 +|
|                     |   -- juggling                                                                   +|
|                     |   update tmp_task t                                                             +|
|                     |   set current_interval = tt.default_interval,                                   +|
|                     |       retries_left = coalesce(retries_left, tt.num_retries, 0)                  +|
|                     |   from task_type tt                                                             +|
|                     |   where tt.type=t.type;                                                         +|
|                     |                                                                                 +|
|                     |   insert into task (type, arguments, next_run, status, current_interval, policy,+|
|                     |                     retries_left, priority)                                     +|
|                     |     select type, arguments, next_run, status, current_interval, policy,         +|
|                     |            retries_left, priority                                               +|
|                     |     from tmp_task t                                                             +|
|                     |     where not exists(select 1                                                   +|
|                     |                      from task                                                  +|
|                     |                      where type = t.type and                                    +|
|                     |                            md5(arguments::text) = md5(t.arguments::text) and    +|
|                     |                            arguments = t.arguments and                          +|
|                     |                            policy = t.policy and                                +|
|                     |                            priority is not distinct from t.priority and         +|
|                     |                            status = t.status);                                  +|
|                     |                                                                                 +|
|                     |   return query                                                                  +|
|                     |     select distinct t.*                                                         +|
|                     |     from tmp_task tt inner join task t on (                                     +|
|                     |       tt.type = t.type and                                                      +|
|                     |       md5(tt.arguments::text) = md5(t.arguments::text) and                      +|
|                     |       tt.arguments = t.arguments and                                            +|
|                     |       tt.policy = t.policy and                                                  +|
|                     |       tt.priority is not distinct from t.priority and                           +|
|                     |       tt.status = t.status                                                      +|
|                     |     );                                                                          +|
|                     | end;                                                                            +|
|                     |                                                                                  |
| Description         | Create tasks in bulk from the temporary table                                    |
+---------------------+----------------------------------------------------------------------------------+

[2]

# use super powa' to drop the function in a psql session
# and run the patched migrated cli [3]
swhscheduler@scheduler0:~$ swh db --config-file /etc/softwareheritage/scheduler/backend.yml upgrade scheduler
INFO:swh.core.db.db_utils:Executing migration 34 with script '/usr/lib/python3/dist-packages/swh/scheduler/sql/upgrades/34.sql'
Migration to version 34 done
swhscheduler@scheduler0:~$ swh db --config-file /etc/softwareheritage/scheduler/backend.yml version scheduler
module: scheduler
current code version: 34
version: 34

[3] swh/devel/swh-core!356 (merged)

mentioned in commit swh/devel/swh-core@b8159c6d

mentioned in merge request swh/devel/swh-core!356 (merged)

marked the checklist item scheduler0: Update package as completed

marked the checklist item scheduler-db: Apply migration script (34) as completed

marked the checklist item Checks as completed

mentioned in commit swh/devel/swh-core@a45eb0ac

mentioned in commit swh/devel/swh-core@0a143f5b

mentioned in commit swh/devel/swh-core@7d830658

marked the checklist item scheduler0: Update package as completed

changed the description

I've reactivated puppet and trigger a run. So it should be available for production as well now.

root@moma:~# puppet agent --test
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for moma.softwareheritage.org
Info: Applying configuration version '1686671721'
Notice: /Stage[main]/Profile::Swh::Deploy::Webapp/Profile::Swh::Deploy::Install_web_deps[swh-web]/Package[python3-swh.web]/ensure: ensure changed '0.2.27-1~swh1~bpo10+1' to '0.2.28-1~swh1~bpo10+1'
Info: /Stage[main]/Profile::Swh::Deploy::Webapp/Profile::Swh::Deploy::Install_web_deps[swh-web]/Package[python3-swh.web]: Scheduling refresh of Service[gunicorn-swh-webapp]
Notice: /Stage[main]/Profile::Swh::Deploy::Webapp/Gunicorn::Instance[swh-webapp]/Service[gunicorn-swh-webapp]: Triggered 'refresh' from 1 event
Notice: Applied catalog in 26.11 seconds

closed