- Mar 13, 2017
-
-
Antoine R. Dumont authored
Related T494
-
- Mar 07, 2017
-
-
Nicolas Dandrimont authored
-
Nicolas Dandrimont authored
-
Nicolas Dandrimont authored
-
Nicolas Dandrimont authored
This will allow us to handle another storage backend for the storage of the archiver data.
-
Nicolas Dandrimont authored
-
- Mar 02, 2017
-
-
Nicolas Dandrimont authored
By default we would try to copy objects from all the archives, even those for which we didn't have a configuration.
-
Nicolas Dandrimont authored
The default value for content copies is "missing", so we don't need to make it explicit.
-
Nicolas Dandrimont authored
-
Nicolas Dandrimont authored
-
Antoine R. Dumont authored
Implementation wise, this uses COPY statement and drop duplicates if encountered for content_archive insertion.
-
Antoine R. Dumont authored
Use the same insertion pattern as swh.storage.content_add.
-
Antoine R. Dumont authored
This also has the benefit to hide some celery name (which is an implementation detail from swh.scheduler).
-
Antoine R. Dumont authored
Related: T494
-
Antoine R. Dumont authored
This use case cannot happen with ArchiverWithRetentionPolicyDirector: - If a row entry is referenced in the archiver db, it's present in the objstorage - And if a row entry is not referenced in the archiver db, it won't be listed as missing since it's the archiver db which is read for listing the contents we want to archive.
-
Antoine R. Dumont authored
Related T494 Related T569
-
Antoine R. Dumont authored
Related T494
-
- Feb 16, 2017
-
-
Nicolas Dandrimont authored
-
- Feb 14, 2017
-
-
Nicolas Dandrimont authored
To make sure corruptions such as T680 don't happen again, use the same normalization function as swh.model before inserting timestamps into our database. This makes swh.storage reject non-integer timestamp values as well. Update tests to reflect this change.
-
- Feb 09, 2017
-
-
Nicolas Dandrimont authored
Close T672
-
Nicolas Dandrimont authored
-
Antoine Pietri authored
-
Antoine Pietri authored
-
- Feb 07, 2017
-
-
Nicolas Dandrimont authored
The buckets are using the last two bytes of the object id, so that we spread the load across different lines on sequential archivings.
-
- Jan 26, 2017
-
-
Antoine R. Dumont authored
Closes T646
-
- Jan 03, 2017
-
-
Nicolas Dandrimont authored
-
- Dec 31, 2016
-
-
Antoine R. Dumont authored
-
- Dec 20, 2016
-
-
Antoine R. Dumont authored
-
- Nov 15, 2016
-
-
Antoine R. Dumont authored
-
- Nov 03, 2016
-
-
Nicolas Dandrimont authored
The check_config method allows a dynamic check of the configuration for a running storage. We can make sure that we have proper permissions on the object storage as well as the database before running things.
-
- Oct 13, 2016
-
-
Antoine R. Dumont authored
Related T575
-
- Sep 29, 2016
-
-
Antoine R. Dumont authored
-
Antoine R. Dumont authored
-
Antoine R. Dumont authored
-
- Sep 23, 2016
-
-
Antoine R. Dumont authored
-
Antoine R. Dumont authored
-
Antoine R. Dumont authored
-
Antoine R. Dumont authored
-
Antoine R. Dumont authored
The 'unknown sha1 path' cannot happen in the default archiver since it reads from the archive db (so the fallback code is not necessary in the worker). To the contrary, since 'archiver to backend' reads from stdin (for now), we could have unregistered sha1s from that source. This commit makes the director deal with that before sending sha1 to workers. It's also the director's job to set the state to 'missing' when the force_copy is true before sending sha1 to worker.
-
Antoine R. Dumont authored
-