Skip to content

backend: Insert cooking task info in database after task creation

It might happen that the scheduler RPC service is failing when attempting to submit a vault cooking task.

So prefer to insert cooking task info in vault database once the task successfully created by the scheduler. Previously, the non created cooking task was inserted in database with a null task id and a new status which prevented to create a new one once the scheduler service rpc issue fixed.

I did some tests in the docker environment by first stopping scheduler service and attempting to create a cooking task using the public Web API, an expected status code of 500 is well returned.

$ curl -i -X POST http://localhost:5004/api/1/vault/flat/swh:1:dir:86170c4a719bc655b893cd5b061c98ab0cadc860/
HTTP/1.1 500 Internal Server Error
Server: gunicorn
Date: Thu, 28 Sep 2023 11:56:10 GMT
Connection: keep-alive
Content-Type: application/json
Expires: Thu, 28 Sep 2023 11:56:10 GMT
Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private
Vary: Accept, origin, Cookie, Accept-Encoding
Allow: GET, POST, OPTIONS
X-Frame-Options: DENY
Content-Length: 3085
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin

{"exception":"RemoteException","reason":"<RemoteException 500 APIError: [ConnectionError(Exception(\"HTTPConnectionPool(host='swh-scheduler', port=5008): Max retries exceeded with url: /task/create (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f298426f4d0>: Failed to establish a new connection: [Errno -2] Name or service not known'))\"))]>"}

Then after restarting the scheduler service and sending the same request, the cooking task could be successfully created.

$ curl -i -X POST http://localhost:5004/api/1/vault/flat/swh:1:dir:86170c4a719bc655b893cd5b061c98ab0cadc860/
HTTP/1.1 200 OK
Server: gunicorn
Date: Thu, 28 Sep 2023 11:57:19 GMT
Connection: keep-alive
Content-Type: application/json
Expires: Thu, 28 Sep 2023 11:57:19 GMT
Cache-Control: max-age=0, no-cache, no-store, must-revalidate, private
Vary: Accept, origin, Cookie, Accept-Encoding
Allow: GET, POST, OPTIONS
X-Frame-Options: DENY
Content-Length: 217
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin

{"fetch_url":"http://localhost:5004/api/1/vault/flat/swh:1:dir:86170c4a719bc655b893cd5b061c98ab0cadc860/raw/","progress_message":null,"id":2,"status":"new","swhid":"swh:1:dir:86170c4a719bc655b893cd5b061c98ab0cadc860"}

Related to swh/infra/sysadm-environment#5055 (closed).

Edited by Antoine Lambert

Merge request reports