add-forge-now: Allow max_pages choice with staging preset
Compare changes
When staging preset is used, the max_pages argument is always overwritten:
swh@95e644cdaf0e:/$ swh scheduler -C $SWH_CONFIG_FILENAME \
add-forge-now --preset staging \
register-lister gitlab \
instance=git.iem.at \
max_pages=1
{'instance': 'git.iem.at', 'max_pages': 1}
{'instance': 'git.iem.at', 'max_pages': 3, 'max_origins_per_page': 10, 'enable_origins': False}
Created 1 tasks
Task 1
Next run: today (2023-11-22T09:32:26.226019+00:00)
Interval: 90 days, 0:00:00
Type: list-gitlab-full
Policy: oneshot
Args:
Keyword args:
enable_origins: False
instance: 'git.iem.at'
max_origins_per_page: 10
max_pages: 3
swh@95e644cdaf0e:/$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-listed-origins gitlab git.iem.at -l
url last_seen last_update
------------------------------------------------------------------------ -------------------------------- --------------------------------
https://git.iem.at/cm/diyremin.git 2023-11-22 09:32:33.388019+00:00 2021-12-14 19:08:00.541000+00:00
[...]
https://git.iem.at/zmoelnig/VeSTige.git 2023-11-22 09:32:32.101888+00:00 2023-09-21 05:51:13.928000+00:00
Forge git.iem.at (gitlab) has 30 listed origins in the scheduler database.These modifications will be very useful in the add-forge-now processing requests in term of time and resources.
Check 10 origins ingestion in staging will be enough , IMHO.
swh@b7559d725a59:/$ swh scheduler -C $SWH_CONFIG_FILENAME \
add-forge-now --preset staging \
register-lister gitlab \
instance=git.iem.at \
max_pages=1
{'instance': 'git.iem.at', 'max_pages': 1}
{'instance': 'git.iem.at', 'max_pages': 1, 'max_origins_per_page': 10, 'enable_origins': False}
Created 1 tasks
Task 2
Next run: today (2023-11-22T09:22:06.800213+00:00)
Interval: 90 days, 0:00:00
Type: list-gitlab-full
Policy: oneshot
Args:
Keyword args:
enable_origins: False
instance: 'git.iem.at'
max_origins_per_page: 10
max_pages: 1
swh@b7559d725a59:/$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-listed-origins gitlab git.iem.at -l
url last_seen last_update
---------------------------------------------- -------------------------------- --------------------------------
https://git.iem.at/pd/iemguts.git 2023-11-22 09:22:27.088210+00:00 2023-07-07 15:11:26.293000+00:00
https://git.iem.at/zmoelnig/DVImatrix848.git 2023-11-22 09:22:27.088210+00:00 2023-06-22 19:45:50.147000+00:00
Forge git.iem.at (gitlab) has 10 listed origins in the scheduler database.