LISA18 has ended
Back To Schedule
Tuesday, October 30 • 2:00pm - 2:30pm
Kafka on Kubernetes: From Evaluation to Production at Intuit

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Feedback form is now closed.
Kubernetes is fast becoming the platform of choice for running distributed, containerized applications in the cloud. It has great features for availability, scalability, monitoring, ease of deployment, a rich set of tools and an extremely fast-growing ecosystem that is making it ever more useful.

However, running stateful applications such as Kafka on Kubernetes is not a common practice today. At Intuit, we took an experimentation and data driven approach for evaluating Kafka on Kubernetes in AWS. In this talk, we provide details of our requirements, the experimental setup, and details of the tests. Tests included basic functional tests for produce/consume of messages, network isolation tests, cross-region produce/consume, performance and scale tests. We’ll also talk about the various tools used during this testing. We focus on the problems we ran into and how we addressed them.

This talk will demonstrate a Kubernetes cluster running Kafka along with the details of how each component is configured. Specifically, the kafka and zookeeper statefulsets, the configmaps used for storing the server.properties used by all brokers, the service objects for enabling access to the brokers and, last but not least, integration with Splunk and Wavefront for logging and monitoring respectively.

avatar for Shri Javadekar

Shri Javadekar

Software Engineer, Outerbounds, Inc.
Shri Javadekar leads the Kubernetes based efforts for a data-science and machine learning platform built at Outerbounds, based on the open-source Metaflow project. Previously, he led the engineering efforts of a modern SAAS platform at Intuit. It involved hundreds of K8s clusters... Read More →

Tuesday October 30, 2018 2:00pm - 2:30pm CDT
Legends Ballroom ABC