Blog
Blog
Stories, how-tos and talk recaps from people running data on Kubernetes.
- All posts
- DoK Day at KubeCon NA 2021
- Announcement
- DoK Day at KubeCon EU 2022
- How To
- Community
- End user
- Events
- Opinion
- Featured
- Data on Kubernetes Report
- DoK Days
- DoK Day at KubeCon EU 2024
- DoK Day at KubeCon NA 2022
Aug 5, 2024
Released From the Cage: Apache Kafka Without Its ZooKeeper
For years, Apache Kafka relied on Apache ZooKeeper for maintaining its metadata and coordination. But that is coming to an end. After a lot of work in the Apache Kafka community, ZooKeeper is going away from Apache Kafka and it will be replaced with its own Raft-inspired implementation called KRaft. This is a major architecture change for all Kafka users, including those running Kafka on Kubernetes. And it also affects projects such as Strimzi that provide tooling for running Apache Kafka on Kubernetes. So, how does it work? What are the advantages? What does this change mean for the existing ZooKeeper-based Kafka clusters? What are the main challenges and limitations when using Kraft on Kubernetes? What are the changes we had to make in the Strimzi project to make it ready for KRaft? All of this was answered in this talk including a short demo of what Strimzi support for KRaft looks like.
Jul 31, 2024
Lightning Talk: Enabling Hot Restart of Stateful Applications Including GPU-Accelerate AI/ML Workloads
This talk both described and demonstrated how stateful applications, including GPU-accelerated AI/ML workflows, can be automatically hot restarted after pod kill/eviction events that are augmented by transparent memory-snapshotting techniques. Applications with complex initializations/startup times, as well as long-running, batch-like workloads, are the best candidates for leveraging this type of operator, and representative benchmarks will be shared. This approach can be used in conjunction with Day 2 node maintenance, autoscaling, OOM issues, and Public Cloud Spot instances to obtain operational benefits, including reduced downtime, higher infrastructure utilization, and reduced computing costs. This presentation aimed to stimulate a broader discussion of additional use cases for memory snapshotting.
Jul 25, 2024
Lessons Learned from Building a Database Operator
In this talk, we shared our journey in developing Kubernetes Operators to run MongoDB in critical production Kubernetes environments. This presentation offered a deep dive into the lessons learned from both building a database operator and learnings from our customers from running a database on Kubernetes. We explored the nuanced challenges and innovative solutions encountered while orchestrating MongoDB within Kubernetes’ dynamic environment. We covered the architectural decisions, performance optimization strategies, and the critical role of stateful workload management in Kubernetes.
Jul 8, 2024
Cloud-Native Dataspaces: Experiences from the German Research Data Ecosystem
Data on Kubernetes and stateful applications have gained remarkable adoption across the community. But why stop there? Kubernetes and cloud-native tools can provide compelling core technologies for building sophisticated data ecosystems, from advanced metadata handling to workflows and events. Enter the realm of “dataspaces,” a transformative concept empowering organizations to seamlessly integrate and synchronize data sharing patterns for diverse existing data landscapes, that can even extend across organizational boundaries. Our session gave practical examples how Kubernetes and open source tools can be used to harmonize a heterogeneous data ecosystem. The resulting cloud-native dataspaces increase interoperability, reduce vendor lock-ins, and create an overall boost in operational efficiency by enabling the use of modern DataOps practices.
Jul 3, 2024
Lightning Talk: My Database Runs on Kubernetes. What's Next? Data Platforms!
There’s not much doubt that databases now run well on Kubernetes: operators have matured, storage management works, and there are lots of success stories. What do you do now? Build your own data platform to replace expensive, proprietary cloud services! Argo CD and Flux make it possible to integrate databases, data ingest, visualization, integration, and operations into a single platform that deploys from GitHub or GitLab. Our talk reviewed open source projects for data platforms configuration as well as standard design patterns for applying them in real systems.
Jul 1, 2024
Lightning Talk: Ditching Data Pipelines: why treating Data as Assets is the best thing you can do
Efficient data handling traditionally involves constructing robust pipelines to process information from diverse sources. However, recent open-source tools question this approach and propose an alternative: rather than detailing data processing steps, why not focus on the relationships between data objects?
Jul 1, 2024
Announcing Your New Data on Kubernetes Ambassadors
We put out the call for Data on Kubernetes community members to submit applications for the Ambassador program, and you responded. We had an overwhelming response of amazing candidates. Several existing ambassadors are returning and will be joined by a roster of new DoK experts.
Jun 27, 2024
From Zero to Hero: Scaling Postgres in Kubernetes Using the Power of CloudNativePG
Unleash PostgreSQL’s potential in Kubernetes with CloudNativePG, a community-driven control plane reshaping the database landscape. Join a dedicated CloudNativePG maintainer and active Postgres contributor on a captivating journey through managing highly available clusters in the Cloud Native era. Discover best practices for large-scale databases: architecture, deployment on bare metal or virtual machines in Kubernetes, storage optimization, robust backup, recovery strategies, vertical scalability, and performance tuning. Gain insights into real-world challenges and battle-tested solutions for seamless PostgreSQL cluster operation, catering to applications and AI use cases. Delve into CloudNativePG’s strengths, discussions on existing capabilities, limitations, and the future roadmap. Revolutionize PostgreSQL in Kubernetes—liberate data, embrace portability, and empower your organization with CloudNativePG.
Jun 4, 2024
How to Create Your Own Metadata-Driven ML Platform from Scratch
This talk demonstrated integrating a data lakehouse with a metadata-driven orchestration engine, utilizing open-source tools like Presto and Kubeflow. Other key learnings included: Learn about Presto and how it works with the open lakehouse concept for efficient data analytics. Understand the basics of setting up Presto, connecting it to your data sources, and executing your first queries. Explore Kubeflow, a Kubernetes-based platform for ML workflows. We walked through setting up Kubeflow, creating metadata-driven pipelines, and deploying notebook servers. Learn about distributed training operators, hyper-parameter tuning, and model serving within the Kubeflow ecosystem. Automating your ML lifecycle, from data ingestion to model deployment. Understand how to implement traceability and manage your workflows efficiently using the tools discussed. Learn how to integrate real-time data streams with your ML platform, ensuring your models stay relevant and effective. Speakers:
May 28, 2024
Data on Kubernetes Ambassador Interview with Alex Lines
This interview with Data on Kubernetes Ambassador, Alex Lines, is part of series where we learn where each of our DoK Ambassadors got their start in open source and the Data on Kubernetes Community. Alex also tells his thoughts on the future of Data on Kubernetes.
May 20, 2024
Advanced CSI-FUSE Filesystem for AI/ML Data Management in Kubernetes
AI/ML workloads, known for their data intensity, often depend on cloud storage. However, Kubernetes faces significant challenges in accessing this cloud storage data efficiently, primarily due to the lack of a Kubernetes volume interface and a heavy reliance on object storage-specific APIs. In this session, Lu Qiu discussed how to use an advanced CSI-FUSE filesystem to address the above challenges. She also analyzed various industry-standard CSI-FUSE products to guide users in selecting the most suitable CSI-FUSE solution for their specific Kubernetes AI/ML workloads. Other key learnings:
May 2, 2024
Does containerization affect the performance of databases?
This post has been provided by*DoK community sponsor* ApeCloud and authored by Cai Songlu The wave of database containerization is on the rise, as clearly shown in Fig.1. Databases and analytics are now a major part of the tech scene. Yet, a common dilemma persists: Does containerization impact database performance? If yes, what factors come into play? How can we tackle performance and stability issues brought about by containerization?