Skip to content

Reschedule pack file too big failing loading task to dedicated queue consumed by large enough workers

Current git loading tasks can fail to be ingested with a [1] 'pack file too big' message. They are then disabled by the scheduler after a number of tryouts (3). Those could be redirected to another queue [2] whose consumption can happen only with large enough workers [3]. (with proper setup like pack file limit set to 32g or something).

  • [1]
                raise IOError(
                    f"Pack file too big for repository {origin_url}, "
                    f"limit is {size_limit} bytes, current size is {cur_size}, "
                    f"would write {would_write}"
                )
  • [3] worker17-18 currently

  • [2] large-repository:swh.loader.git.tasks.UpdateGitRepository


Migrated from T4390 (view on Phabricator)

Edited by Antoine R. Dumont