Related T589

Related T589

Related T575

Related T582

This adds the optional conflict_update parameter which specifies what to
do when conflicts on sha1 occurs.

conflict_update by default is false which ignores duplicates.
Otherwise, conflich_update to true, overwrite existing data.

Related T582

Related T578

Related T578

Related T577

Related T577

Towards starting computing information on contents

Related T577

it includes the cache_* tables that are currently being populated

This should reduce lock contention when parallelizing the operation

The 'unknown sha1 path' cannot happen in the default archiver since it
reads from the archive db (so the fallback code is not necessary in the
worker).

To the contrary, since 'archiver to backend' reads from stdin (for now),
we could have unregistered sha1s from that source.

This commit makes the director deal with that before sending sha1 to
workers.

It's also the director's job to set the state to 'missing' when the
force_copy is true before sending sha1 to worker.

Before that, it could for example pushed copies to other mirrors where
the content was missing.