Skip to content

[production] Provision enough space for the search ES cluster to ingest all intrinsic metadata

Considering it looks like there will be a lot of new indexed object, we must be sure the chosen solution will be able to scale up in the future

Possible identified options:

  • Deploy a new elasticsearch instance on esnode* [1]

  • Build a new cluster with new bare metal servers [2]

  • extends current cluster storage [3]

  • [1] Some drawbacks:

  • there is 32Go of memory on the servers, 16G are allocated to the current elasticsearches, in the short term, we can use 8go per instance but we must be sure the log cluster is functional with only 8go

    • Possible solution: Increase memory (4 memory slots are still available on the servers)

  • the current cluster use 3To per node for a total of 7To, which could be too small to keep the replication factor in case of a failure of one node. There is no more slot available to add new disks, the possible solutions:

    • reducing the retention delay of the logs (estimated gain 2To)
    • Replace disk by bigger ones

  • [2] 3 new servers 1U 4x2.4To (4.8Toeffective) 32Go memory cost: ~8500e (possible to have up to 8 disks)

  • [3] The storage is on the ceph cluster can be increased as there are several free disk slots on beaubourg/hypervisor3/branly

Initial discussion on irc:

11:25 <+vsellier> we will have to check the elasticsearch behavior with production volume
11:26 <+vsellier> the size increases have been important on staging
11:26 <+vsellier> around x5
11:26 <+vsellier> (the index size)
11:27 <+vsellier> if the ratio is the same for production, the bump could be from 250g to 1.2T
11:28 <+vsellier> the search-esnode are not yet sized for this
11:31 <vlorentz> shouldn't we stop the metadata ingestion until they are?
11:31 <+vsellier> (they are vms on proxmox, we will have to think where we can find so much space)
11:32 <+vsellier> the search journal client is not deployed so no problem for the moment, nothing is sent to ES
11:33 <+vsellier> (I mean not deployed in production, it is only on staging)

Migrated from T3041 (view on Phabricator)

Edited by Antoine R. Dumont