Skip to content

Stop sending next-gen scheduled task results to scheduler listener

Antoine R. Dumont requested to merge generated-differential-D6408-source into generated-differential-D6408-target

This is 3-fold, this should:

  • ease the listener's work to only deal with required tasks (the ones that are still scheduled through the initial scheduler-runner, e.g. save code now, deposit, vault, ...). The other tasks scheduled with the next-gen scheduler no longer needs to report such status as a dedicated journal client is in charge of that part.

  • decrease the load induced by those update in the scheduler db

  • help the save code now have better status report (it's still using the scheduler db to access some of those status as fallback).

This is the other side of the configuration adaptation done in the swh.scheduler.task module [1]

Related to T3458

Test Plan

Run staging scheduler0 with some loader npm patched with this configuration.

The "task-result" even is still received but no longer processed, so no more writes happen to the scheduler backend (the part updating the result in db, the scheduler journal client is in charge of that).

bin/octocatalog-diff

$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging worker01
I, [2021-10-05T09:16:49.356428 #909179]  INFO -- : Catalogs compiled for worker01.softwareheritage.org
I, [2021-10-05T09:16:49.691323 #909179]  INFO -- : Diffs computed for worker01.softwareheritage.org
diff origin/production/worker01.softwareheritage.org current/worker01.softwareheritage.org
  File[/etc/softwareheritage/loader_archive.yml] =>
   parameters =>
     content =>
      @@ -20,3 +20,4 @@
         task_queues:
         - swh.loader.package.archive.tasks.LoadArchive
      +  task_ignore_result: true
      _
*******************************************
  File[/etc/softwareheritage/loader_cran.yml] =>
   parameters =>
     content =>
      @@ -20,3 +20,4 @@
         task_queues:
         - swh.loader.package.cran.tasks.LoadCRAN
      +  task_ignore_result: true
      _
*******************************************
  File[/etc/softwareheritage/loader_debian.yml] =>
   parameters =>
     content =>
      @@ -20,3 +20,4 @@
         task_queues:
         - swh.loader.package.debian.tasks.LoadDebian
      +  task_ignore_result: true
      _
*******************************************
  File[/etc/softwareheritage/loader_deposit.yml] =>
   parameters =>
     content =>
      @@ -20,4 +20,5 @@
         task_queues:
         - swh.loader.package.deposit.tasks.LoadDeposit
      +  task_ignore_result: false
       deposit:
         url: https://deposit.softwareheritage.org/1/private/
*******************************************
  File[/etc/softwareheritage/loader_git.yml] =>
   parameters =>
     content =>
      @@ -23,3 +23,4 @@
         - swh.loader.git.tasks.LoadDiskGitRepository
         - swh.loader.git.tasks.UncompressAndLoadDiskGitRepository
      +  task_ignore_result: true
      _
*******************************************
  File[/etc/softwareheritage/loader_high_priority.yml] =>
   parameters =>
     content =>
      @@ -26,3 +26,4 @@
         - save_code_now:swh.loader.svn.tasks.DumpMountAndLoadSvnRepository
         - save_code_now:swh.loader.package.archive.tasks.LoadArchive
      +  task_ignore_result: false
      _
*******************************************
  File[/etc/softwareheritage/loader_mercurial.yml] =>
   parameters =>
     content =>
      @@ -22,3 +22,4 @@
         - swh.loader.mercurial.tasks.LoadMercurial
         - swh.loader.mercurial.tasks.LoadArchiveMercurial
      +  task_ignore_result: true
      _
*******************************************
  File[/etc/softwareheritage/loader_nixguix.yml] =>
   parameters =>
     content =>
      @@ -20,4 +20,5 @@
         task_queues:
         - swh.loader.package.nixguix.tasks.LoadNixguix
      +  task_ignore_result: false
       unsupported_file_extensions:
       - patch
*******************************************
  File[/etc/softwareheritage/loader_npm.yml] =>
   parameters =>
     content =>
      @@ -20,3 +20,4 @@
         task_queues:
         - swh.loader.package.npm.tasks.LoadNpm
      +  task_ignore_result: true
      _
*******************************************
  File[/etc/softwareheritage/loader_opam.yml] =>
   parameters =>
     content =>
      @@ -20,3 +20,4 @@
         task_queues:
         - swh.loader.package.opam.tasks.LoadOpam
      +  task_ignore_result: true
      _
*******************************************
  File[/etc/softwareheritage/loader_pypi.yml] =>
   parameters =>
     content =>
      @@ -20,3 +20,4 @@
         task_queues:
         - swh.loader.package.pypi.tasks.LoadPyPI
      +  task_ignore_result: true
      _
*******************************************
  File[/etc/softwareheritage/loader_svn.yml] =>
   parameters =>
     content =>
      @@ -22,3 +22,4 @@
         - swh.loader.svn.tasks.MountAndLoadSvnRepository
         - swh.loader.svn.tasks.DumpMountAndLoadSvnRepository
      +  task_ignore_result: true
      _
*******************************************
*** End octocatalog-diff on worker01.softwareheritage.org

Migrated from D6408 (view on Phabricator)

Merge request reports