Skip to content

swh-docker-dev: `load-tar` tasks are never executed

With the swh-docker-dev stack, the load-tar tasks generated by the GNU listers are never executed. Note tasks generated by the gitlab lister don't have this issue.

It's quite easy to reproduce, with a fresh swh-docker-dev environment, run the GNU lister from the lister container:

docker-compose exec swh-lister python3 -c 'import logging; from swh.lister.gnu.tasks import gnu_lister; logging.basicConfig(level=logging.DEBUG); gnu_lister()'

You can see more than 300 tasks are created. Here is a fragment:

docker-compose exec  swh-scheduler-api  swh scheduler task list 
...
Task 384
  Next run: 7 minutes ago (2019-09-04 17:35:39+00:00)
  Interval: 64 days, 0:00:00
  Type: load-tar
  Policy: recurring
  Status: next_run_scheduled
  Priority: 
  Args:
    'glibc'
    'https://ftp.gnu.org/gnu/glibc/'
  Keyword args:
    tarballs: [{'date': '859017600', 'archive': 'https://ftp.gnu.org/old-gnu/glibc/glibc-2.0.2.tar.gz'}, {'date': '862470000', 'archive': 'https://ftp.gnu.org/old-gnu/glibc/glibc-2.0.3-m68k-linux.bin.tar.gz'}, {'date': '861692400', 'archive': 'https://ftp.gnu.org/old-gnu/glibc/glibc-2.0.3.tar.gz'}, {'date': '865321200', 'archive': 'https://ftp.gnu.org/old-gnu/glibc/glibc-2.0.4-m68k-linux.bin.tar.gz'}, {'date': '865148400', 'archive': 'https://ftp.gnu.org/old-gnu/glibc/glibc-2.0.4.bin.i386.tar.gz'}, {'date': '864716400', 'archive': 'https://ftp.gnu.org/old-gnu/glibc/glibc-2.0.4.tar.gz'}, {'date': '872492400', 'archive': 'https://ftp.gnu.org/old-gnu/glibc/glibc-2.0.5.tar.gz'}, {'date': '854352000', 'archive': 'https://ftp.gnu.org/old-gnu/glibc/glibc-2.0.bin.i386.tar.gz'}, {'date': '854352000', 'archive': 'https://ftp.gnu.org/old-gnu/glibc/glibc-2.0.tar.gz'}, {'date': '859017600', 'archive': 'https://ftp.gnu.org/old-gnu/glibc/glibc-linuxthreads-2.0.2.tar.gz'}, {'date': '861951600', 'archive': 'https://ftp.gnu.org/old-gnu/glibc/glibc-linuxthreads-2.0.3a.tar.gz'}, {'date': '864716400', 'archive': 'https://ftp.gnu.org/old-gnu/glibc/glibc-linuxthreads-2.0.4.tar.gz'}, {'date': '872492400', 'archive': 'https://ftp.gnu.org/old-gnu/glibc/glibc-linuxthreads-2.0.5.tar.gz'}, {'date': '854352000', 'archive': 'https://ftp.gnu.org/old-gnu/glibc/glibc-linuxthreads-2.0.tar.gz'}, {'date': '859017600', 'archive': 'https://ftp.gnu.org/old-gnu/glibc/glibc-localedata-2.0.2.tar.gz'}, {'date': '861692400', 'archive': 'https://ftp.gnu.org/old-gnu/glibc/glibc-localedata-2.0.3.tar.gz'}, {'date': '864716400', 'archive': 'https://ftp.gnu.org/old-gnu/glibc/glibc-localedata-2.0.4.tar.gz'}, {'date': '872492400', 'archive': 'https://ftp.gnu.org/old-gnu/glibc/glibc-localedata-2.0.5.tar.gz'}, {'date': '854352000', 'archive': 'https://ftp.gnu.org/old-gnu/glibc/glibc-localedata-2.0.tar.gz'}]

As you can see, the status of the task is next_run_scheduled, but it is never executed.

Migrated from T1985 (view on Phabricator)