Skip to content

add-forge-now: Allow max_pages choice with staging preset

Guillaume Samson requested to merge afn_staging_max_pages into master

When staging preset is used, the max_pages argument is always overwritten:

Test before modifications 
swh@95e644cdaf0e:/$ swh scheduler -C $SWH_CONFIG_FILENAME \
  add-forge-now --preset staging \
  register-lister gitlab \
    instance=git.iem.at \
    max_pages=1
{'instance': 'git.iem.at', 'max_pages': 1}
{'instance': 'git.iem.at', 'max_pages': 3, 'max_origins_per_page': 10, 'enable_origins': False}
Created 1 tasks

Task 1
  Next run: today (2023-11-22T09:32:26.226019+00:00)
  Interval: 90 days, 0:00:00
  Type: list-gitlab-full
  Policy: oneshot
  Args:
  Keyword args:
    enable_origins: False
    instance: 'git.iem.at'
    max_origins_per_page: 10
    max_pages: 3

swh@95e644cdaf0e:/$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-listed-origins gitlab git.iem.at -l
url                                                                       last_seen                         last_update
------------------------------------------------------------------------  --------------------------------  --------------------------------
https://git.iem.at/cm/diyremin.git                                        2023-11-22 09:32:33.388019+00:00  2021-12-14 19:08:00.541000+00:00
[...]
https://git.iem.at/zmoelnig/VeSTige.git                                   2023-11-22 09:32:32.101888+00:00  2023-09-21 05:51:13.928000+00:00

Forge git.iem.at (gitlab) has 30 listed origins in the scheduler database.

These modifications will be very useful in the add-forge-now processing requests in term of time and resources.
Check 10 origins ingestion in staging will be enough , IMHO.

Test with modifications 
swh@b7559d725a59:/$ swh scheduler -C $SWH_CONFIG_FILENAME \
  add-forge-now --preset staging \
  register-lister gitlab \
    instance=git.iem.at \
    max_pages=1
{'instance': 'git.iem.at', 'max_pages': 1}
{'instance': 'git.iem.at', 'max_pages': 1, 'max_origins_per_page': 10, 'enable_origins': False}
Created 1 tasks

Task 2
  Next run: today (2023-11-22T09:22:06.800213+00:00)
  Interval: 90 days, 0:00:00
  Type: list-gitlab-full
  Policy: oneshot
  Args:
  Keyword args:
    enable_origins: False
    instance: 'git.iem.at'
    max_origins_per_page: 10
    max_pages: 1

swh@b7559d725a59:/$ swh scheduler -C $SWH_CONFIG_FILENAME origin check-listed-origins gitlab git.iem.at -l
url                                             last_seen                         last_update
----------------------------------------------  --------------------------------  --------------------------------
https://git.iem.at/pd/iemguts.git               2023-11-22 09:22:27.088210+00:00  2023-07-07 15:11:26.293000+00:00

https://git.iem.at/zmoelnig/DVImatrix848.git    2023-11-22 09:22:27.088210+00:00  2023-06-22 19:45:50.147000+00:00

Forge git.iem.at (gitlab) has 10 listed origins in the scheduler database.

Merge request reports