Find a way to properly open the kafka brokers to the internet

The kafka brokers need to be accessible from the internet, so our mirrors can subscribe to the topics and process messages.

We need to figure out:

• frontend/proxying
• TLS
• authentication
• authorization

For reference:

• proxy-based deployment with TLS offloading : https://medium.com/code-tech/kafka-in-aws-with-ssl-offloading-using-load-balancer-c337da1435c3
• Authentication: https://docs.confluent.io/current/kafka/authentication_sasl/index.html
• Authorization: https://docs.confluent.io/current/kafka/authorization.html

There's a strong chance that the journal code will need to be adapted to allow passing the proper settings to the kafka libraries.

---

Migrated from T1829 (view on Phabricator)

added Mirror System administration priority:High labels

A new Kafka cluster has been spun up on azure virtual machines, with 6 machines each with 8TB of storage available.

A Kafka Mirror Maker has been setup by hand on getty to pull the data from the cluster in Rocquencourt to the cluster on Azure (only on the content topic for now).

My working theory for now is to use the Rocquencourt cluster as a low-latency buffer in front of the Azure cluster.

The next step is to lock this cluster down with SASL authentication, and to give it public ip addresses and TLS setup so it can be opened to the internet.

added state:wip label

The content topic has fully replicated to the new cluster over the weekend.

I've now added replication for the other topics.

Following documentation with the following links:

• https://kafka.apache.org/documentation/#security
• https://rmoff.net/2018/08/02/kafka-listeners-explained/
• https://docs.confluent.io/current/kafka/encryption.html#ssl-overview
• sort of https://medium.com/code-tech/kafka-in-aws-with-ssl-offloading-using-load-balancer-c337da1435c3
• kinda https://www.slideshare.net/ConfluentInc/how-to-lock-down-apache-kafka-and-keep-your-streams-safe

We've updated settings of the azure kafka cluster to:

• enable TLS
• enable SASL authentication
• open the TLS port to the world

Latest commit in puppet: rSPSITE250ea3a376cc Latest commit in terraform: rSPRE1acb92757799

With the current deployment, the inter-broker, internal communication still happens in plaintext; clients local to the swh infra can still access the cluster without authentication or encryption. However, an internal port has been opened for TLS communication (port 9094 on all brokers).

Adding a new user and credentials is done with the following command:

/opt/kafka/bin/kafka-configs.sh \
    --zookeeper kafka01.euwest.azure.internal.softwareheritage.org:2181,kafka02.euwest.azure.internal.softwareheritage.org:2181,kafka03.euwest.azure.internal.softwareheritage.org:2181,kafka04.euwest.azure.internal.softwareheritage.org:2181,kafka05.euwest.azure.internal.softwareheritage.org:2181,kafka06.euwest.azure.internal.softwareheritage.org:2181/kafka/softwareheritage \
    --alter \
    --add-config 'SCRAM-SHA-256=[iterations=8192,password=<password>],SCRAM-SHA-512=[password=<password>]' \
    --entity-type users \
    --entity-name <username>

Removing a user's credentials is done with the following:

/opt/kafka/bin/kafka-configs.sh \
    --zookeeper kafka01.euwest.azure.internal.softwareheritage.org:2181,kafka02.euwest.azure.internal.softwareheritage.org:2181,kafka03.euwest.azure.internal.softwareheritage.org:2181,kafka04.euwest.azure.internal.softwareheritage.org:2181,kafka05.euwest.azure.internal.softwareheritage.org:2181,kafka06.euwest.azure.internal.softwareheritage.org:2181/kafka/softwareheritage \
    --alter \
    --delete-config 'SCRAM-SHA-256,SCRAM-SHA-512' \
    --entity-type users \
    --entity-name <username>

(I'll have puppet generate wrapper scripts for this mess).

Running a client on my laptop at Inria pointed at the public IP addresses, using the following SASL config, works properly:

c = confluent_kafka.Consumer({
    'group.id': 'olasd-test-sasl-1',
    'bootstrap.servers': ','.join('kafka%02d.euwest.azure.softwareheritage.org:9093' % i for i in range(1,7)),
    'security.protocol': 'SASL_SSL',
    'sasl.mechanisms': 'SCRAM-SHA-512',
    'sasl.username': '<username>',
    'sasl.password': '<password>',
    'debug': 'consumer',
})

Unfortunately, port 9093 is firewalled between our infra at SESI and the outside world. For now, the consumer on uffizi has been re-configured to use port 9094 (internal tls) and a standalone credential.

The next step will be introducing Authorization.

Using swh.journal.client:

c = swh.journal.client.JournalClient(**{
    'group_id': 'olasd-test-sasl-1',
    'brokers': ['kafka%02d.euwest.azure.softwareheritage.org:9093' % i for i in range(1,7)],
    'security.protocol': 'SASL_SSL',
    'sasl.mechanisms': 'SCRAM-SHA-512',
    'sasl.username': '<username>',
    'sasl.password': '<password>',
    'debug': 'consumer',
})

(yes, passing dotted config parameters in kwargs is... not the cleanest)

The new cluster in rocquencourt is using the built-in Kafka ACLs now (9993a81ffc7a1c8bd519b33ae63ac1145105f624).

Connections to the internal plaintext port are mapped to a built-in ANONYMOUS user. To keep the status quo, this user has been configured as superuser (e1a942059f0081cfb1e69baeab9defb5d93d3776), until we've configured producers and consumers not to need that.

User management

Usernames follow the pattern : <namespace>-<username>. For instance: swh-olasd is a user for @olasd in the swh namespace.

User creation

To create a user, run the following command:

/opt/kafka/bin/kafka-configs.sh \
    --zookeeper kafka1.internal.softwareheritage.org,kafka2.internal.softwareheritage.org,kafka3.internal.softwareheritage.org,kafka4.internal.softwareheritage.org:2181/kafka/softwareheritage \
    --alter \
    --add-config 'SCRAM-SHA-256=[iterations=8192,password=<password>],SCRAM-SHA-512=[password=<password>]' \
    --entity-type users \
    --entity-name <full-username>

The password is stored, hashed, in zookeeper.

User deletion

To delete a user, run the following command:

/opt/kafka/bin/kafka-configs.sh \
    --zookeeper kafka1.internal.softwareheritage.org,kafka2.internal.softwareheritage.org,kafka3.internal.softwareheritage.org,kafka4.internal.softwareheritage.org:2181/kafka/softwareheritage \
    --alter \
    --delete-config 'SCRAM-SHA-256,SCRAM-SHA-512' \
    --entity-type users \
    --entity-name <full-username>

Superuser

To set a user as superuser (allowing it to create topics and perform other management tasks), create it with the previous command, and add it to the kafka configuration in puppet (swh-site/data/common/kafka.yaml). The cluster needs to be restarted for the config change to apply.

Kafka command configuration

To use the Kafka CLI utilities, the following settings need to be configured in a kafka.properties file, passed to the various command line utilities (either as --command-config or --consumer.config, depending on the command line utility):

sasl.mechanism=SCRAM-SHA-512
security.protocol=SASL_SSL
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="<username>" \
  password="<password>";

ACLs

By default, users don't have any read or write access to any data on the cluster.

Consumer ACLs

• We box consumers in a given consumer group prefix, to allow multiple consumers for a given user but avoiding users interfering with one another.
• We only allow reads on topics starting with a given prefix.

bootstrap_servers=kafka1.internal.softwareheritage.org:9092,kafka2.internal.softwareheritage.org:9092,kafka3.internal.softwareheritage.org:9092,kafka4.internal.softwareheritage.org:9092
username=<full-username>

# Allow READ and DESCRIBE on unprivileged topics
/opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --topic swh.journal.objects. --allow-principal User:$username --operation READ
/opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --topic swh.journal.objects. --allow-principal User:$username --operation DESCRIBE

# Allow READ on consumer groups prefixed with `$username-`
/opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --group ${username}- --allow-principal User:$username --operation READ

To allow a consumer to read privileged (non-anonymized) topics:

bootstrap_servers=kafka1.internal.softwareheritage.org:9092,kafka2.internal.softwareheritage.org:9092,kafka3.internal.softwareheritage.org:9092,kafka4.internal.softwareheritage.org:9092
username=<full-username>

# Allow READ and DESCRIBE on **privileged** topics
/opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --topic swh.journal.objects_privileged. --allow-principal User:$username --operation READ
/opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --topic swh.journal.objects_privileged. --allow-principal User:$username --operation DESCRIBE

Producer ACLs

TBD.

removed state:wip label

closed

mentioned in issue #3188 (closed)

mentioned in issue #1017 (closed)