Skip to content
Snippets Groups Projects

add-forge-now: Allow max_pages choice with staging preset

Merged add-forge-now: Allow max_pages choice with staging preset
afn_staging_max_pages into master
3 unresolved threads
Merged Guillaume Samson requested to merge afn_staging_max_pages into master
3 unresolved threads
Compare
and
1 file
+ 12
− 3
Compare changes
  • Side-by-side
  • Inline
swh/scheduler/cli/add_forge_now.py
+ 12
− 3
@@ -58,8 +58,8 @@ def register_lister_cli(
\b
- staging preset: a single oneshot full listing task is scheduled. This "full"
listing is limited to 3 pages and 10 origins per page. The origins are recorded as
disabled (to avoid their recurrent loading).
listing is limited to 3 pages (default is one) and 10 origins per page.
The origins are recorded as disabled (to avoid their recurrent loading).
- production preset: a recurrent full and incremental (if the loader has such a
task) listing task are scheduled. The first run of the full lister is scheduled
immediately, and the first run of the incremental lister is delayed by a day.
@@ -86,12 +86,21 @@ def register_lister_cli(
(args, kw) = parse_options(options)
max_pages = kw.get("max_pages", 2)
max_origins_per_page = kw.get("max_origins_per_page", 5)
    • Comment on lines +89 to +94
      Antoine R. Dumont
      Antoine R. Dumont @ardumont ·
      Maintainer

      why not simply?

      Suggested change
      86 if "max_pages" in kw:
      87 if kw["max_pages"] <= 3:
      88 max_pages = kw["max_pages"]
      89 else:
      90 max_pages = 1
      91
      86 max_pages = kw.get("max_pages", 1)
      87
      Edited by Antoine R. Dumont
      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Maintainer

        And I'd suggest that max_pages declaration be moved within the else condition (staging).

        More generally, let's declare variables near to where it's used ;) That's less confusing.

      • Antoine R. Dumont
        Antoine R. Dumont @ardumont ·
        Maintainer

        And i'd change the default max_pages to 2 for staging (decreasing the total number of origins per page to 5 to keep testing the pagination ;)

      • Please register or sign in to reply
Please register or sign in to reply
# Recurring policy on production
if preset == "production":
policy = "recurring"
else: # staging, "full" but limited listing as a oneshot
policy = "oneshot"
kw.update({"max_pages": 3, "max_origins_per_page": 10, "enable_origins": False})
kw.update(
{
"max_pages": max_pages,
"max_origins_per_page": max_origins_per_page,
"enable_origins": False,
}
)
# We want a "full" listing in production if both incremental and full exists
if "full" in task_types:
task_types.pop("incremental", None)