Skip to content

Refactor and optimize the archiver

Quentin Campos requested to merge generated-differential-D81-source into generated-differential-D81-target

  • Minor optimizations of the code

  • Archiver's batches structure is now simplified in order to reduce the redundancy of the data and simplify the task creation.

  • Director's iterations over the contents are now correctly separated into simplified functions:

    • Batches from db are flattened into a continuous stream
    • Archiver's director iterate over it and yield filtered/transformed data
    • This continuous stream is batched separately from the previous step to avoid multiple functionality in a single function

  • Remove int() conversions around python's time API as the db contains float as well.

  • Perform a full refactoring of the archiver worker in order to simplify its code and behavior. Also, remove the redundant check of the potential modifications of the content status once the task have been scheduled.

Migrated from D81 (view on Phabricator)

Merge request reports