Add end to end save code now check

This defines an end to end check which trigger for an dedicated origin with a given type a save code now requests.

This then regularly polls the status out of the save code now api. Depending on the time it took to actually ingest and the ingestion status, it will return the status of said ingestion and roughly the time it took.

The end goal being to deploy such checks into icinga and being notified when a save code now request takes too long.

Example use:

$ swh icinga_plugins \
  check-savecodenow --swh-web-url https://webapp.staging.swh.network \
  origin <url> \
  --visit-type git

SAVECODENOW WARNING - Save code now request for ('git', '<url>') took 30.58s and succeeded.
| 'total_time' = 30.58s

Related to swh/infra/sysadm-environment#2117 (closed)

Test Plan

Using the staging webapp api, then checking up until the code worked or being rate limited, whatever came first:

$ swh icinga_plugins check-savecodenow --swh-web-url https://webapp.staging.swh.network origin https://github.com/BRGM/map2loop-2 --visit-type git

SAVECODENOW WARNING - Save code now request for (git, https://github.com/BRGM/map2loop-2) took 51.18s and succeeded.
| 'total_time' = 51.18s

$ swh icinga_plugins check-savecodenow --swh-web-url https://webapp.staging.swh.network origin https://github.com/divnix/devos --visit-type git
SAVECODENOW OK - Save code now request for (git, https://github.com/divnix/devos) took 32.28s and succeeded.
| 'total_time' = 32.28s

$ swh icinga_plugins check-savecodenow --swh-web-url https://webapp.staging.swh.network origin https://github.com/divnix/devos --visit-type git
SAVECODENOW CRITICAL - Save code now request for (git, https://github.com/divnix/devos) took more than 349.80s and has status: succeeded
| 'total_time' = 349.80s    # <-------------------------- pdb session and letting it hang a bit then continue after threshold

$ swh icinga_plugins check-savecodenow --swh-web-url https://webapp.staging.swh.network origin https://github.com/divnix/devos --visit-type git
Traceback (most recent call last):
  File "/home/tony/.virtualenvs/swh/bin/swh", line 33, in <module>
    sys.exit(load_entry_point('swh.core', 'console_scripts', 'swh')())
  File "/home/tony/work/inria/repo/swh/swh-environment/swh-core/swh/core/cli/__init__.py", line 185, in main
    return swh(auto_envvar_prefix="SWH")
  File "/home/tony/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/tony/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/tony/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/tony/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/tony/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/tony/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/tony/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/tony/.virtualenvs/swh/lib/python3.7/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/tony/work/inria/repo/swh/swh-environment/swh-icinga-plugins/swh/icinga_plugins/cli.py", line 86, in check_scn_origin
    sys.exit(SaveCodeNowCheck(ctx.obj).main())
  File "/home/tony/work/inria/repo/swh/swh-environment/swh-icinga-plugins/swh/icinga_plugins/save_code_now.py", line 80, in main
    assert response.status_code == 200, (response, response.text)
AssertionError: (<Response [429]>, '{"exception":"Throttled","reason":"Request was throttled. Expected available in 2058 seconds."}')
$ swh icinga_plugins check-savecodenow --swh-web-url https://archive.softwareheritage.org origin https://github.com/SebaUbuntu/TWRP-device-tree-generator --visit-type git
SAVECODENOW OK - Save code now request for (git, https://github.com/SebaUbuntu/TWRP-device-tree-generator) took 20.66s and succeeded.
| 'total_time' = 20.66s

---

Migrated from D5541 (view on Phabricator)