Blog on Data on Kubernetes Community

Data on Kubernetes Community Transitions to Community-Led Stewardship

Thu, 08 Oct 2026 12:00:00 +0000

The Data on Kubernetes (DoK) Community is entering a new chapter. Since 2020 it has been a global hub for practitioners running databases, streaming, analytics and AI data on Kubernetes. Constantia LLC is transferring the community’s assets and stewardship to the community itself, with Sergey Pronin acting as steward during the transition.

Beyond Adoption: The Age of Operational Excellence

Mon, 10 Nov 2025 16:00:59 +0000

The 2025 Data on Kubernetes (DoK) Report Is Here

Five years ago, the idea of running data workloads on Kubernetes still raised eyebrows. Today, it’s the norm — and the question has changed from “Can we?” to “How do we do it best?”

CFP open for Data on Kubernetes Day at KubeCon EU 2025

Fri, 25 Oct 2024 21:08:11 +0000

We’re excited to announce that the Data on Kubernetes (DoK) Community will once again host a co-located event at KubeCon EU 2025 in London. As the leading gathering for DoK practitioners and enthusiasts, DoK Day showcases cutting-edge innovations and fosters invaluable knowledge sharing within our vibrant community.

Vitess Joins Data on Kubernetes Community Collaborator Program

Fri, 25 Oct 2024 21:01:22 +0000

The Data on Kubernetes Community is excited to announce the addition of Vitess as its newest Community Collaborator. Vitess is an open-source database clustering system that helps scale MySQL databases horizontally through sharding while maintaining MySQL compatibility and handling connection pooling, query routing, and automated failover. It was originally developed by YouTube/Google to solve their MySQL scaling challenges, and it has become particularly well-suited for Kubernetes environments because it was designed to run in containerized environments

Empowering developers w/ easy, scalable stream processing on K8s

Mon, 14 Oct 2024 16:00:44 +0000

In this Town Hall, Intuit shared insights on Numaflow, an open-source Kubernetes-native stream processing framework developed to address the challenges of real-time data analytics. The presentation highlighted how Numaflow enables easy, cost-efficient, and resilient stream processing without the steep learning curve or operational intensity of existing solutions. It showed Intuit’s successful implementation of Numaflow, which processes billions of messages daily across hundreds of Kubernetes clusters, supporting various applications from analytics to machine learning predictions and anomaly detection.

Vitess: A cloud-native distributed system for scaling MySQL

Mon, 07 Oct 2024 16:00:35 +0000

In this DoK Talk, we learned about Vitess, a powerful database solution for deploying and managing large clusters of MySQL and Percona Server instances. The talk highlighted Vitess’s versatility across cloud and dedicated hardware environments, its ability to extend SQL features with impressive scalability, and its architecture with Kubernetes-friendly features, demonstrating why many enterprises have adopted this solution for their production needs.

Running Batch Data Workloads in Kubernetes at Dish Network

Mon, 30 Sep 2024 16:00:41 +0000

This DoK Town Hall explored optimizing batch data processing at scale on Kubernetes, addressing the challenges of fast, reliable data processing in the face of exponential data growth. It showcased an architecture leveraging Apache Spark Operator, Apache YuniKorn, Karpenter, and Argo Workflows. This approach aimed to enhance resource allocation, storage performance, and job scheduling, demonstrating how businesses can efficiently handle large-scale data processing for improved decision-making.

Women in Cloud Native / Airflow on K8s

Mon, 23 Sep 2024 16:00:15 +0000

Women in Cloud Native was presented by Nancy Chauhan, Developer Advocate at LocalStack and founder of Women in Cloud Native. In this session, we heard about Nancy’s journey in cloud native and why she founded the group Women in Cloud Native.

Calling on Contributors for the Data on Kubernetes SIG Get Started Project

Thu, 19 Sep 2024 22:54:22 +0000

Since its inception in 2020, the Data on Kubernetes Community (DoKC) has been on a mission to promote the use of Kubernetes for stateful and data workloads. While we’ve made significant strides, running stateful applications on Kubernetes remains a challenge for many.

Anatomy of a DBaaS: Bringing self-serve databases to Kubernetes with open source

Mon, 16 Sep 2024 23:07:53 +0000

This talk was an under-the-hood look at developing an open source DBaaS. We looked at the components that are used for its backend services, frontend, and the inner workings of a DBaaS running on top of Kubernetes.

Data on Kubernetes Survey: Contribute to the Collective Knowledge of our Community

Tue, 20 Aug 2024 23:31:02 +0000

The Data on Kubernetes (DoK) Community is conducting its annual survey to understand the technology issues and trends for running data and stateful workloads on Kubernetes. This year’s survey will focus on DoK ecosystem technologies and AI/ML workloads. We invite you to be part of this year’s survey and contribute to the collective knowledge of our community.

Announcing Lori Lorusso as Co-Chair for the Data on Kubernetes SIG

Mon, 05 Aug 2024 18:42:14 +0000

We are thrilled to announce that Lori Lorusso, Head of Community at Percona, has been appointed as co-chair for the Data on Kubernetes Special Interest Group (SIG). This appointment marks an exciting new chapter for our community, and we’re confident that Lori’s extensive experience and passion will drive significant advancements in our mission.

Released From the Cage: Apache Kafka Without Its ZooKeeper

Mon, 05 Aug 2024 17:00:57 +0000

For years, Apache Kafka relied on Apache ZooKeeper for maintaining its metadata and coordination. But that is coming to an end. After a lot of work in the Apache Kafka community, ZooKeeper is going away from Apache Kafka and it will be replaced with its own Raft-inspired implementation called KRaft. This is a major architecture change for all Kafka users, including those running Kafka on Kubernetes. And it also affects projects such as Strimzi that provide tooling for running Apache Kafka on Kubernetes. So, how does it work? What are the advantages? What does this change mean for the existing ZooKeeper-based Kafka clusters? What are the main challenges and limitations when using Kraft on Kubernetes? What are the changes we had to make in the Strimzi project to make it ready for KRaft? All of this was answered in this talk including a short demo of what Strimzi support for KRaft looks like.

Lightning Talk: Enabling Hot Restart of Stateful Applications Including GPU-Accelerate AI/ML Workloads

Wed, 31 Jul 2024 17:00:44 +0000

This talk both described and demonstrated how stateful applications, including GPU-accelerated AI/ML workflows, can be automatically hot restarted after pod kill/eviction events that are augmented by transparent memory-snapshotting techniques. Applications with complex initializations/startup times, as well as long-running, batch-like workloads, are the best candidates for leveraging this type of operator, and representative benchmarks will be shared. This approach can be used in conjunction with Day 2 node maintenance, autoscaling, OOM issues, and Public Cloud Spot instances to obtain operational benefits, including reduced downtime, higher infrastructure utilization, and reduced computing costs. This presentation aimed to stimulate a broader discussion of additional use cases for memory snapshotting.

Lessons Learned from Building a Database Operator

Thu, 25 Jul 2024 21:15:41 +0000

In this talk, we shared our journey in developing Kubernetes Operators to run MongoDB in critical production Kubernetes environments. This presentation offered a deep dive into the lessons learned from both building a database operator and learnings from our customers from running a database on Kubernetes. We explored the nuanced challenges and innovative solutions encountered while orchestrating MongoDB within Kubernetes’ dynamic environment. We covered the architectural decisions, performance optimization strategies, and the critical role of stateful workload management in Kubernetes.

Cloud-Native Dataspaces: Experiences from the German Research Data Ecosystem

Mon, 08 Jul 2024 16:00:19 +0000

Data on Kubernetes and stateful applications have gained remarkable adoption across the community. But why stop there? Kubernetes and cloud-native tools can provide compelling core technologies for building sophisticated data ecosystems, from advanced metadata handling to workflows and events. Enter the realm of “dataspaces,” a transformative concept empowering organizations to seamlessly integrate and synchronize data sharing patterns for diverse existing data landscapes, that can even extend across organizational boundaries. Our session gave practical examples how Kubernetes and open source tools can be used to harmonize a heterogeneous data ecosystem. The resulting cloud-native dataspaces increase interoperability, reduce vendor lock-ins, and create an overall boost in operational efficiency by enabling the use of modern DataOps practices.

Lightning Talk: My Database Runs on Kubernetes. What's Next? Data Platforms!

Wed, 03 Jul 2024 16:00:49 +0000

There’s not much doubt that databases now run well on Kubernetes: operators have matured, storage management works, and there are lots of success stories. What do you do now? Build your own data platform to replace expensive, proprietary cloud services! Argo CD and Flux make it possible to integrate databases, data ingest, visualization, integration, and operations into a single platform that deploys from GitHub or GitLab. Our talk reviewed open source projects for data platforms configuration as well as standard design patterns for applying them in real systems.

Lightning Talk: Ditching Data Pipelines: why treating Data as Assets is the best thing you can do

Mon, 01 Jul 2024 16:00:40 +0000

Efficient data handling traditionally involves constructing robust pipelines to process information from diverse sources. However, recent open-source tools question this approach and propose an alternative: rather than detailing data processing steps, why not focus on the relationships between data objects?

Announcing Your New Data on Kubernetes Ambassadors

Mon, 01 Jul 2024 13:00:17 +0000

We put out the call for Data on Kubernetes community members to submit applications for the Ambassador program, and you responded. We had an overwhelming response of amazing candidates. Several existing ambassadors are returning and will be joined by a roster of new DoK experts.

From Zero to Hero: Scaling Postgres in Kubernetes Using the Power of CloudNativePG

Thu, 27 Jun 2024 16:00:44 +0000

Unleash PostgreSQL’s potential in Kubernetes with CloudNativePG, a community-driven control plane reshaping the database landscape. Join a dedicated CloudNativePG maintainer and active Postgres contributor on a captivating journey through managing highly available clusters in the Cloud Native era. Discover best practices for large-scale databases: architecture, deployment on bare metal or virtual machines in Kubernetes, storage optimization, robust backup, recovery strategies, vertical scalability, and performance tuning. Gain insights into real-world challenges and battle-tested solutions for seamless PostgreSQL cluster operation, catering to applications and AI use cases. Delve into CloudNativePG’s strengths, discussions on existing capabilities, limitations, and the future roadmap. Revolutionize PostgreSQL in Kubernetes—liberate data, embrace portability, and empower your organization with CloudNativePG.

How to Create Your Own Metadata-Driven ML Platform from Scratch

Tue, 04 Jun 2024 16:00:23 +0000

This talk demonstrated integrating a data lakehouse with a metadata-driven orchestration engine, utilizing open-source tools like Presto and Kubeflow. Other key learnings included:

Learn about Presto and how it works with the open lakehouse concept for efficient data analytics. Understand the basics of setting up Presto, connecting it to your data sources, and executing your first queries.
Explore Kubeflow, a Kubernetes-based platform for ML workflows. We walked through setting up Kubeflow, creating metadata-driven pipelines, and deploying notebook servers. Learn about distributed training operators, hyper-parameter tuning, and model serving within the Kubeflow ecosystem.
Automating your ML lifecycle, from data ingestion to model deployment. Understand how to implement traceability and manage your workflows efficiently using the tools discussed.
Learn how to integrate real-time data streams with your ML platform, ensuring your models stay relevant and effective.

Speakers:

Data on Kubernetes Ambassador Interview with Alex Lines

Tue, 28 May 2024 23:23:57 +0000

This interview with Data on Kubernetes Ambassador, Alex Lines, is part of series where we learn where each of our DoK Ambassadors got their start in open source and the Data on Kubernetes Community. Alex also tells his thoughts on the future of Data on Kubernetes.

Advanced CSI-FUSE Filesystem for AI/ML Data Management in Kubernetes

Mon, 20 May 2024 16:00:27 +0000

AI/ML workloads, known for their data intensity, often depend on cloud storage. However, Kubernetes faces significant challenges in accessing this cloud storage data efficiently, primarily due to the lack of a Kubernetes volume interface and a heavy reliance on object storage-specific APIs. In this session, Lu Qiu discussed how to use an advanced CSI-FUSE filesystem to address the above challenges. She also analyzed various industry-standard CSI-FUSE products to guide users in selecting the most suitable CSI-FUSE solution for their specific Kubernetes AI/ML workloads. Other key learnings:

Does containerization affect the performance of databases?

Thu, 02 May 2024 18:10:14 +0000

This post has been provided by*DoK community sponsor* ApeCloud and authored by Cai Songlu

The wave of database containerization is on the rise, as clearly shown in Fig.1. Databases and analytics are now a major part of the tech scene. Yet, a common dilemma persists: Does containerization impact database performance? If yes, what factors come into play? How can we tackle performance and stability issues brought about by containerization?

Become a Data on Kubernetes Ambassador in 2024

Tue, 30 Apr 2024 20:49:40 +0000

The Data on Kubernetes Community (DoKC) is thrilled to announce that we will be accepting applications for the DoKC Ambassadors Program starting May 1st. The DoK Ambassadors program was created to acknowledge and empower members of the community that have demonstrated expertise in DoK ecosystem and a level of enthusiasm for participating in the community.

Kubernetes as a Data Platform, DoK Panel at KubeCon

Mon, 29 Apr 2024 15:00:11 +0000

Data on Kubernetes is well-positioned to become the operational default in a world where data and AI/ML applications are expected to grow. Scalability, flexibility, resilience, openness, and costs are among the reasons; and it has a transformative impact on organizations who benefit from increased productivity, revenue growth, market share and margin, according to the 2022 Data on Kubernetes Report.

Adding Zonal Resiliency to Etsy's Kafka Cluster

Thu, 18 Apr 2024 15:00:14 +0000

Kafka is an important part of Etsy’s data ecosystem, moving data that powers a number of things like analytics, A/B testing, and search indexing. Etsy runs its Kafka cluster on Kubernetes and in 2022 the data team made an effort to ensure that the cluster could withstand a GCP zonal outage. This created an interesting opportunity for the team to change the way they approached rolling out changes to the Kafka brokers. They were able to cut the time for these updates from 7 hours to a little over 2.

DoK Day Meets Paris at KubeCon EU 2024

Thu, 04 Apr 2024 16:57:43 +0000

For the second time, the Data on Kubernetes Community’s DoK Day event was hosted by the CNCF as a co-located event at KubeCon Europe 2024 in Paris, France – and it was our biggest and most well attended event yet!

DoK Community Sponsor Spotlight: Apache YuniKorn

Tue, 13 Feb 2024 23:07:02 +0000

The Data on Kubernetes Community (DoKC) is the global home for thousands of end users sharing best practices for running data workloads on Kubernetes including 4k Slack users, nearly 3k Meetup followers and 7k newsletter subscribers. We deliver a range of community programs to support practitioners on their DoK journey including meetups, conferences, end user roundtables, research reports, an ambassador program, and other activities to help practitioners connect and learn. Our work would not be possible without the support of sponsors who enable us to bring you these programs while sharing their knowledge and experience to foster growth and innovation with Kubernetes.

Data On Kubernetes Ecosystem Day

Mon, 29 Jan 2024 19:57:31 +0000

A summary of our discussion and presentations during our first-ever DoK Ecosystem Day.

DoK is a vendor-neutral space, but we also know organizations rely on vendors for different parts of their stack. In January, the Data on Kubernetes Community gave vendors an opportunity to be shameless for its inaugural Ecosystem Day event!

Data Compliance Across Multiple Regions with KubeSlice

Tue, 23 Jan 2024 19:53:04 +0000

A summary of our discussion and presentation during our first DoK Talk in January.

Data on Kubernetes (DoK) Talks are virtual events presented by Data on Kubernetes Community members sharing best practices, use cases, and other practical advice to help end users on their DoK journey.

K8s scheduling: Batch workloads in multi-tenant environment with Apache YuniKorn

Tue, 12 Dec 2023 19:50:59 +0000

A summary of our discussion during our DoKC Town Hall held in November.

Data on Kubernetes Community (DoKC) Town Halls are an opportunity to bring the community together to meet one another, share end-user journey stories, discuss DoK-related projects and technologies, and stay up-to-date on community events and ways to participate.

Get started with AI on AWS with MLFlow and Notebooks on K8s

Wed, 25 Oct 2023 17:49:38 +0000

A summary of our discussion during our DoKC Town Hall held in October.

Data on Kubernetes Community (DoKC) Town Halls allow our community to meet one another, share end user journey stories, discuss DoK-related projects and technologies, and learn more about community events and ways to participate.

Rook - Helping the Kubernetes Storage Community Thrive

Thu, 28 Sep 2023 16:54:00 +0000

A summary of our discussion during our DoKC Town Hall held in September.

How Chick-fil-A runs Kubernetes at the Edge

Tue, 29 Aug 2023 00:17:44 +0000

Brian Chambers, Chick-fil-A’s Chief Architect, shared insights into how the company strategically harnessed Kubernetes to tackle intricate challenges posed by high sales volumes and the need for real-time localized solutions. The company’s drive for technology innovation stemmed from improving kitchen operations. When faced with kitchen efficiency challenges, they embarked on an initiative to deploy technology solutions that enhanced real-time visibility and streamlined staff operations.

Dutch Government: Implementing Data & Databases on K8s

Wed, 26 Jul 2023 16:15:12 +0000

A summary of our discussion during our DoKC Town Hall held in July

Join the DoKC Ambassador Program Today

Wed, 19 Jul 2023 23:08:52 +0000

We are thrilled to announce the launch of the Data on Kubernetes Community (DoKC) Ambassador Program! The DoKC Ambassador Program is designed to recognize individuals in the community who are actively participating in activities and engaging with other community members. It provides an opportunity for community members to take their involvement to the next level; play a more significant role in shaping the future of our community; and help educate other users who are on their journey running data on Kubernetes.

Kubernetes Operators and Data Pipelines

Tue, 27 Jun 2023 16:00:51 +0000

A summary of our discussion during our DoKC Town Hall held in June.

DoK Operators: Workload Reliability and Availability

Wed, 21 Jun 2023 22:39:24 +0000

On May 24, Akshay Ram and Michelle Au from Google led the DoK Operator Roundtable, representing both the Kubernetes project and Google Kubernetes Engine (GKE) as a managed Kubernetes provider. DoK roundtables unite operator developers for an informal deep dive into challenges that come up when writing operators and using them in production. The goal of the roundtable was to explore challenges projects are facing when writing operators, and to find opportunities where the Kubernetes project could address those gaps.

May Data on Kubernetes Bay Area Meetup Report

Wed, 31 May 2023 16:51:31 +0000

The DoK Bay Area meetup held its first official meeting on May 17 at Bootup World in Menlo Park. Thanks to the hard work and support from Olyvia Rakshit and the team at Avesha, we were able to obtain a venue and meet for pizza, pop, and presentations on Kubernetes.

Recap: Data on Kubernetes First Town Hall

Tue, 30 May 2023 14:30:10 +0000

A summary of our discussion during our first DoKC Town Hall held in May: Improved DevX with DoK and best practices for deploying Spark on Kubernetes

We held our first-ever community Town Hall on Thursday, May 18. Hosted by the Data on Kubernetes Community (DoKC), Town Halls allow our community to meet one another, share end user journey stories, discuss DoK-related projects and technologies, and learn more about community events and ways to participate.

Data on Kubernetes Community @ KubeCon Europe 2023

Mon, 01 May 2023 15:46:01 +0000

By Melissa Logan

We had an eventful and rewarding week with the Data on Kubernetes Community at KubeCon Europe in Amsterdam. From meetups to panels to hallway conversations, it was energizing and fulfilling to see how the DoKC came together to learn, share, and grow together.

Upcoming Data on Kubernetes Community Events

Thu, 13 Apr 2023 16:24:06 +0000

As we welcome a new season - for some of us spring has sprung and others are buckling down for fall. No matter which season you’re welcoming in, seasonal changes bring fresh opportunities. We are excited to announce new ways for us to gather as a community. We have some exciting DoK community events, including some big firsts for us.

Using Kubernetes for Healthcare Data

Thu, 02 Feb 2023 19:14:54 +0000

Healthcare organizations are transforming their applications and embracing digital platforms for efficient patient care. Edge computing plays a critical role in deploying innovative care that would not be possible without it.

Why we decided to migrate our Jaeger storage to ClickHouse

Wed, 04 Jan 2023 11:05:38 +0000

In this talk, OpsVerse co-founder Arul Jegadish explains why they decided to switch from using Jaeger storage backend to ClickHouse, and why they choose to run it on Kubernetes. He also shares tips and tricks to migrate and run it without problems.

Building a Digital Factory for the Sheet Metal Industry

Tue, 03 Jan 2023 18:15:34 +0000

CamData 3D, one of the newest projects of Kurago in partnership with Bystronic, provides a multi-tenant cloud service aiming to improve the digitization of the sheet metal industry. In this talk, Senior DevOps Engineer Elie Assi elaborates on why and how they choose to host their service in Kubernetes instead of virtual machines.

Graph in Kubernetes Panel

Mon, 12 Dec 2022 10:21:44 +0000

Graph databases are the fastest-growing data store in the world. According to Gartner, the application of graph processing and graph DBMSs will grow at 100 percent annually through 2022 to continuously accelerate data preparation and enable more complex and adaptive data science.

How to build a cloud-native event-driven app and deploy to Kubernetes

Mon, 28 Nov 2022 18:39:17 +0000

In the following tutorial, we will explore and learn the foundations of building an event-driven app based on a cloud-native approach. We will develop several cloud-native microservices, deploy them over a minikube cluster, and install Memphis, which is a cloud-native real-time data processing platform that will enable communication between the services. Finally, we will run the entire application on a Kubernetes cluster.

From Laptop to Cloud: Developing Cloud-Native Applications with Containerized Databases

Mon, 28 Nov 2022 17:35:09 +0000

With the advent of microservices in Kubernetes, individual developer teams now manage their own data, middleware, and databases. Automated tests and CI/CD pipelines must be revisited to include these new requirements.

Kubernetes Operator Lifecycle Management

Mon, 28 Nov 2022 15:43:46 +0000

In this Talk, Anynines CEO Julian Fisher shares about the challenges of managing Kubernetes operator’s lifecycle. Julian and his team benchmarked multiple tools that could help with that. The talk covers Carvel, Helm, OLM, Operator SDK, where the pros and cons are shared for each tool.

Protection against accidental deletion of PVs or Namespaces

Wed, 16 Nov 2022 17:40:18 +0000

Accidental PVC delete or namespace delete can cause the Persistent Volume to get deleted. Such volumes lose their data, and the stateful applications lose their state. By using Persistent Volume TrashCan, users can get a grace period to undo such unintended delete operation.

Leveraging Running Stateful Workloads on Kubernetes for the Benefit of Developers

Mon, 14 Nov 2022 21:06:52 +0000

Kubernetes comes with a lot of useful features like Volumes and StatefulSets, which make running stateful workloads simple. Interestingly, when combined with the right tools, these features can make Kubernetes very valuable for developers wanting to run massive production databases in development! This is exactly what was seen at Extendi.

Data on Kubernetes 2022 Report

Tue, 18 Oct 2022 16:02:06 +0000

Data applications empower organizations to create new business models and transform industries, and Kubernetes is increasing the velocity with which they can be deployed. Last year’s report found that the appetite for Data on Kubernetes (DoK) is strong, with 90% of respondents believing that Kubernetes is ready for stateful workloads and a large majority of respondents (70%) running them in production.

Resilient Redis

Tue, 11 Oct 2022 15:19:46 +0000

Redis is a widely used open-source in-memory data store and cache that has become a key component in the development of scalable microservice systems. While all of the main cloud providers provide fully managed Redis services (Amazon ElastiCache, Azure Cache for Redis, and GCP Memorystore), it may also be simply implemented in Kubernetes if you require additional control over the Redis configurations.

DoK Day @ KubeCon NA 2022 Schedule is Here!

Tue, 11 Oct 2022 12:38:10 +0000

DoK Day – happening next October 24th – is where the industry meets to forge relationships, share best practices, and use cases about running data on Kubernetes. We have a great lineup of 15+ talks, check out the schedule here 👈

Build your own social media analytics with Apache Kafka

Wed, 21 Sep 2022 14:25:50 +0000

Apache Kafka is more than just a messaging broker. It has a rich ecosystem of different components. There are connectors for importing and exporting data, different stream processing libraries, schema registries, and a lot more. In this talk, Senior Principal Software Engineer at Red Hat Jakub Schultz, shows how to use Kafka to read data from social networks such as Twitter, process them and use machine learning to analyze them. With everything running on top of Kubernetes.

One Click to Run Apache Spark as a Service on Kubernetes

Sun, 18 Sep 2022 20:01:42 +0000

It is still challenging to run Apache Spark on Kubernetes, especially on a large scale. People must address issues like resource isolation, queuing, and cost efficiency. In this talk, Bo Yang discusses these challenges and how to address them. He also presents a one-click way to deploy Apache Spark on Kubernetes.

Kanister & Kopia - An Open-Source Data Protection Match Made in Heaven

Fri, 09 Sep 2022 19:07:19 +0000

During this talk, Pavan Navarathna demonstrates how two open-source tools, Kanister and Kopia, work together to optimize backup and recovery for Kubernetes applications. Kanister allows domain experts to capture application-specific workflows in Blueprints to extend and share. Kopia uses state-of-the-art encryption algorithms (AES-256 or ChaCha20) and compresses data to save bandwidth and storage.

Kubernetes and Real-Time Analytic Applications

Mon, 29 Aug 2022 20:24:57 +0000

Kubernetes is turning into a preferred platform for real-time analytic apps that crunches billions of events per day and returns insights in seconds. In this talk, Robert Hodges, the CEO of Altinity introduces the standard analytic app design pattern of fast event streams coupled with low-latency data warehouses, using open source projects. He walks us through deploying the pipeline on Kubernetes from ingest to end-user access. Touching on the use of operators, scaling, monitoring, upgrade, security, and approaches to adding custom components. He shares concrete lessons about how to stand up low-latency analytics quickly on Kubernetes.

Protecting data with CSI Volume Snapshots on Kubernetes

Mon, 22 Aug 2022 18:53:16 +0000

The container storage interface (CSI) is a contract between different container orchestrators (Kubernetes, Nomad, etc.) and storage plugins. This contract is a set of gRPC services for provisioning, utilizing, and snapshotting storage volumes. In this talk, Software Engineer at Portworx by Pure Storage Grant Griffiths focuses on one aspect of the CSI spec: Volume Snapshots.

Automating Azure Files Restore in AKS

Mon, 22 Aug 2022 14:31:11 +0000

If you are considering Azure Files as the persistent storage for your Azure Kubernetes Service (AKS) applications, there are important considerations for backup and recovery with implications on how you can migrate data for Dev, Test, and Staging. This article outlines these data management considerations in detail and how to work around Azure Files CSI driver limitations to achieve feature parity with using Azure Managed Disks.

Operating FoundationDB on Kubernetes

Tue, 16 Aug 2022 21:50:54 +0000

FoundationDB is a free and open-source multi-model distributed NoSQL database developed by Apple, with a shared-nothing architecture. In this talk, Apple Site Reliability Engineer Johannes M. Scheuermann shares how to run the FoundationDB on Kubernetes. Which is something the company does on a large scale. Johannes talks about what the FoundationDB operator can do and how it does it up to the upcoming features on the roadmap.

Disaggregated Container Attached Storage - Yet Another Topology with What Purpose

Sun, 31 Jul 2022 23:47:18 +0000

Container native storage is designed for Kubernetes and managed by Kubernetes. Ideally, it’s based on disaggregated NVMe storage for scalability and performance.

DataCore Chief Scientist Nick Connelly, shares the basics of disaggregated storage and how it offers scalability advantages for Kubernetes. This talk was given for DoK Day EU 2022, you can access all the talks here.

1000 node Cassandra cluster on Amazon's EKS

Mon, 25 Jul 2022 23:30:08 +0000

Apache Cassandra is a distributed database that does a great job with geo-distributed workloads. K8ssandra, on the other hand, makes it possible to run it on Kubernetes, equipped with backups and dashboarding.

Bringing Apache Cassandra closer to Kubernetes

Sat, 23 Jul 2022 23:43:18 +0000

Aside from geo-distributed workloads, Apache Cassandra also works great as a stand-alone service. However, since Cassandra was built before Kubernetes existed, there’s a mismatch in how they do things.

In line with the series of takeaways we got from DoK Day 2022, Jake Luciani, Chief Architect at the DataStax, discusses how they made Apache Cassandra work more natively on Kubernetes.

Leaning on Kubernetes Portability to Manage Databases Anywhere

Mon, 11 Jul 2022 07:37:58 +0000

Kubernetes has the potential to change the game by enabling users to run complex databases anywhere they want. Its portability coupled with a SaaS user experience makes it a powerful tool.

DoK isn't just Database on Kubernetes

Mon, 27 Jun 2022 07:07:50 +0000

There are crucial parts one should consider in their infrastructure when going all-in on Kubernetes, as well as existing procedures that could deliver success once you go beyond deploying databases on it.

Databases Operations and the Cloud

Sat, 11 Jun 2022 13:00:59 +0000

Database is one of the first-class citizens to run data on Kubernetes, and there are multi-faceted operations to run it. It further requires a data layer to improve availability, scale, and bursting.

Getting Started with the Kubernetes Secrets Store CSI Driver

Wed, 01 Jun 2022 13:00:24 +0000

In Kubernetes, you might encounter difficulties keeping application API keys, access tokens, and passwords safe. There are several approaches to solve this problem.

In this DoKC session, DigitalOcean’s developer advocate, Kim Schlesinger, demonstrates how to install Hashicorp Vault and the Secrets Store CSI Driver to ensure your application’s accessibility to secrets stored in ephemeral volumes.

DoK Day at KubeCon EU 2022 Summary

Tue, 24 May 2022 14:15:36 +0000

What a blast we had a DoK Day @ KubeCon EU 2022. Running stateful workloads on Kubernetes is becoming an industry standard, but there are always challenges! Thank you to our 47 speakers for sharing their knowledge on how to DoK well!

ETL/ELT on Kubernetes with Airbyte: K8s development insights

Mon, 23 May 2022 15:00:47 +0000

There are a lot of different approaches working for a spot in an attempt to solve ETL/ELT on Kubernetes. Considering that the cloud-native landscape is built for deploying Dockerized, open-source software, many of the closed-source solutions don’t align with the trajectory of the community.

Announcing the DoK Landscape

Mon, 16 May 2022 07:02:48 +0000

Data on Kubernetes is becoming an industry standard, the first Data on Kubernetes Report revealed that 90% of respondents believe Kubernetes is ready for stateful workloads. In the past year alone, we have seen interest in the topic of DoK explode with over 400% growth across our core channels.

Benchmarking for PostgreSQL workloads in Kubernetes (Part 2)

Tue, 10 May 2022 14:44:52 +0000

A few months have passed since the last DoK webinar on benchmarking PostgreSQL workloads in a Kubernetes environment. While many things have happened at EDB, the team continues to elevate its discourse and introduces an open-source set of tools in bringing PostgreSQL to Kubernetes.

Help us find the horn

Tue, 03 May 2022 13:29:50 +0000

As Bart was preparing to record videos for DoK Day @ KubeCon EU 2022 –where you can learn all about running stateful workloads on Kubernetes – he realized that the community horn was missing. 😱

Benchmarking for PostgreSQL workloads in Kubernetes

Thu, 28 Apr 2022 14:14:31 +0000

Notions such as “databases like PostgreSQL cannot run on Kubernetes” are something that echoes in this industry all the time. However, Gabriele Bartolini, the Vice President of Cloud Native for EDB, joined by his colleague, Francesco Canovai, a System Engineer for 2ndQuadrant, proves us otherwise.

DoK Day @ KubeCon EU 2022 Schedule is Here!

Mon, 18 Apr 2022 19:35:45 +0000

The schedule for Data on Kubernetes Day at KubeCon EU 2022 is finally here! Make sure to register to learn from experts at the forefront of running data on Kubernetes. This online event will feature 31 talks, 3 panels, and tons of entertainment in-between sessions.

Can Data Become a Declarative Resource?

Fri, 15 Apr 2022 22:53:32 +0000

The declarative, portable, self-healing nature that Kubernetes users are accustomed to in stateless deployments is replaced with complexity, resiliency issues, and portability restrictions when it comes to stateful workloads.

In this synergizing DoKC session, Statehub head of marketing Roey Libfield, head of product Michael Greenberg , and DevOps lead Uri Zaidenwerg, discuss the challenges facing stateful Kubernetes and how to enjoy the same level of portability.

Designing Stateful Apps for the Cloud and Kubernetes

Mon, 11 Apr 2022 13:59:59 +0000

Almost all applications have some kind of stateful part. How do we navigate a cloud-based world of containers where stateless and functions-as-a-service are all the rage? As a long-time architect, designer, and developer of very stateful apps, Evan takes us on a journey through the modern cloud world and Kubernetes, offering helpful design patterns, considerations, tips, and where things are going.

Production Postgres Made Easy on Kubernetes

Mon, 04 Apr 2022 13:28:08 +0000

It’s one thing to get a database up and running on Kubernetes; it’s another to run it in production.

There are many “Day 2” operations to manage when running any database in production, including high availability, backups, monitoring, security, and disaster recovery. Kubernetes operators can take care of that and also manage such as advanced features like self-healing, simple scaling, secure credential management, and more.

How to manage Apache Druid on Kubernetes

Sun, 27 Mar 2022 12:02:31 +0000

This talk introduces Druid operator and how Kubernetes and Operator framework can be used to write an operator that enables provisioning, management, and scaling of a complex cluster of Apache Druid to 1,000 of nodes.

Kubernetes Standardization - What About the Data?

Tue, 15 Mar 2022 13:24:30 +0000

The DoK recent survey of 500 IT professionals found that most organizations are now running stateful workloads on Kubernetes. The benefits are clear: increased agility, scalability, and resilience. But to reap those benefits, organizations need to get a handle on managing all the resources first. And that includes data.

Kubernetes Needs Deep System Integrations

Mon, 14 Mar 2022 14:28:38 +0000

Cloud juggernaut Kelsey Hightower recently shared that the Google team had, against all odds, created a Kubernetes operator for Oracle database. “Here we are, we have officially crossed the chasm,” said Kelsey. But have we really?

Developing a Multi-cluster Operator with the K8ssandra Operator

Mon, 07 Mar 2022 14:18:13 +0000

Cassandra is a highly scalable database with an architecture that makes it well suited for multi-region workloads. A Kubernetes cluster often spans across multiple zones within a single region. However, multi-region Kubernetes clusters are less common though due to the technical challenges that they present. This has led to a growing number of multi-cluster solutions.

Resources for getting started with Kubernetes

Wed, 02 Mar 2022 18:48:19 +0000

Stuti Mishra, DoK Contributor and current CSE undergrad from VIT - Bhopal, India is sharing with us her favorite resources to get started with Kubernetes.

This talk was part of DoK Day at KubeCon NA 2021, watch it below. You can access the other talks here.

DBAs need lovin' too!

Tue, 22 Feb 2022 14:48:03 +0000

An honest discussion about the challenges of running stateful, distributed applications inside of Kubernetes. This talk features Umair Mufti, former Manager of Data Services at DreamWorks where he led the development of a Kubernetes-based, Database-as-a-Service platform. Currently building products for Portworx that incorporate all the lessons learned from managing enterprise-grade databases on Kubernetes.

CFP DoK Day EU 2022

Tue, 15 Feb 2022 17:46:42 +0000

We are back at it again! As Kubecon Europe 2022 is quickly approaching, we are looking for speakers who can share their knowledge about the art of running stateful workloads on Kubernetes. Happening on Monday, May 16, from 9:00 AM - 5:00 PM CET, the event will be 100% virtual and will feature speakers from all over the world.

Scheduled Scaling with Dask and Argo workflows

Tue, 15 Feb 2022 15:18:12 +0000

Complex computational workloads in Python are a common sight these days, especially in the context of processing large and complex datasets. Battle-hardened modules such as Numpy, Pandas, and Scikit-Learn can perform low-level tasks, while tools like Dask make it easy to parallelize these workloads across distributed computational environments.

Announcing DoK London!

Thu, 10 Feb 2022 19:19:54 +0000

DoKC is coming to London! Our community is international, and we have been operating fully online. As the community is growing and running stateful workload on Kubernetes is on its way to becoming an industry-standards, we are increasing our impact by launching local chapters with in-person events, starting with London!

Persistent Disk or StatefulSet?

Tue, 08 Feb 2022 14:28:15 +0000

There is confusion amongst Kubernetes users as to when one should make a Deployment with a PVC and when they should use a StatefulSet with a PVC. There is also a general lack of understanding when it comes to disk access policies, what RWO/RWX means, and what they allow you to do. These concepts are complicated and require a deep level of understanding in order to avoid users making bad decisions that they come to regret later.

From Enemy to Evangelist of Stateful Workloads on Kubernetes

Tue, 01 Feb 2022 16:28:24 +0000

Kubernetes was originally designed to run stateless workloads. Today, it is increasingly used to run databases and other stateful workloads.

Rick Vasquez used to be a huge opponent of putting any data in Kubernetes, now he is one of the most vocal proponents of doing so. Learn why he changed his mind.

Deploying Enterprise Databases on Kubernetes

Sun, 30 Jan 2022 17:25:26 +0000

“Is Kubernetes ready for stateful workloads?” is the first question that pops up when decision-makers consider deploying databases on Kubernetes. For years the answer was “don’t do it”, and for good reasons. Kubernetes was initially designed to handle the orchestration of stateless workload. But the technology has matured, and it is time to reconsider running data on Kubernetes.

Building CI/CD Pipelines Using Kubernetes

Mon, 24 Jan 2022 14:18:03 +0000

According to Bob Ballantyne, traditional CI/CD pipelines are not suitable for native Kubernetes. K8s brings lots of desired features for the enterprise, however, migrations from static virtual machines bring lots of new challenges.

Smart Cities data infrastructure in a cloud-native containerized environment

Thu, 20 Jan 2022 15:28:37 +0000

Smart Cities represent complex and challenging environments that generate an overwhelming amount of data to ingest, transfer, prepare and store.

In this presentation – which is based on a fictitious but potential real world scenario in the city of London – we will see how you can leverage different data engineering patterns to create a fully automated edge-to-core data pipeline for interesting use cases which include license plate recognition and facial recognition.

Machine learning on Kubernetes with Metaflow

Fri, 14 Jan 2022 14:26:44 +0000

Metaflow is an open source machine learning platform, originally built and broadly battle-tested at Netflix to help with ML, analytics, and other data-intensive use cases.

It was initially built to heavily rely on AWS, but it’s been rebuilt to run large-scale data crunching workloads on Kubernetes. In this talk, Oleg shares the surprises and challenges they’ve run into in our journey to get to a consistent, friendly experience on Kubernetes for data scientists.

How to deploy Elasticsearch on Kubernetes

Thu, 06 Jan 2022 13:15:00 +0000

99.co is Singapore’s fastest-growing real estate portal. We power our listings search feature with Elasticsearch (ES), a distributed search engine that can perform complicated search queries at a fast speed. Our backend is a microservices architecture running in Google Kubernetes Engine (GKE), which includes the search service.

DoKC 2021 year in review

Mon, 20 Dec 2021 16:47:53 +0000

As 2021 is coming to an end, we wanted to take a step back to appreciate what the Data on Kubernetes Community achieved. Spoiler alert: a lot!

The DoKC organized not one Dok Day, but two! For KubeCon EU and KubeCon NA, we learned a ton from our community. If you were not able to attend, fear not! You can access all the talks here (EU) and here (NA). Our Dok Days include music interludes, an excellent way to relax in between talks!

Application Level Data Operations on Kubernetes

Thu, 16 Dec 2021 15:16:11 +0000

For stateful, cloud-native applications, data operations must often be performed by tools with a semantic understanding of the data. The volume-level primitives provided by orchestrators are not sufficient to support data workflows like backup/recovery of complex, distributed databases.

Cloud Native Database as a Service using Kubernetes

Mon, 13 Dec 2021 15:57:34 +0000

In the recent Cloud Native Computing Foundation (CNCF) survey, managing complexity is the number one issue faced by Kubernetes architects and practitioners.

Deploying stateful applications in Kubernetes further adds to the cognitive load of practitioners. MayaData and Percona have partnered together to build a unified Helm chart aimed at providing an easy onramp to a highly performant Database as a Service offering.

Migrating MongoDB to Kubernetes

Fri, 10 Dec 2021 16:05:01 +0000

Kubernetes needs help to create and manage stateful applications like databases. It needs a specific network configuration, persistent storage, and dedicated computing capacity. This applies to MongoDB.

This talk goes over an overview of Kubernetes operators for MongoDB, why operators are needed to run MongoDB on Kubernetes. Sergey also explains how to migrate MongoDB to Kubernetes ending with a demo.

Postgres on Kubernetes Applied at Scale

Tue, 07 Dec 2021 15:22:12 +0000

DoKC staff Sylvain and Bart interviewed OnGres CEO Álvaro, engaging in an entertaining, technical discussion about a very real use case: running Postgres on Kubernetes, at scale, in production. We covered open-source tools to use, how to properly handle day-2 operations and why you’d be missing out by not running your database on K8s!

Migrating and running Elasticsearch on Kubernetes

Sat, 04 Dec 2021 23:08:35 +0000

Singapore-based property search engine startup 99.co provides renters, buyers, and agents fast and efficient property search experience. At the core of their platform used by 20 million users/month across Southeast Asia is their Elasticsearch cluster. The engineering team decided to move it to Kubernetes after being frustrated with the lack of agility of their former infrastructure causing outages. Learn about the why, how, and learning of running a production Elasticsearch cluster on Kubernetes.

Using stateful services to index the world of open source

Tue, 30 Nov 2021 23:26:15 +0000

Sourcegraph is a universal code search engine that allows developers to search across their entire codebases. It can be deployed on-prem or used directly from the cloud. The service currently indexes over 1 million open source repositories and provides code search across them. Sourcegraph is planning on indexing every open source repository with more than 1 star on GitHub, GitLab, and other code hosts. To achieve this scale, it relies on Kubernetes to scale several stateful services.

What's Coming in Kubernetes Storage

Fri, 26 Nov 2021 14:51:52 +0000

Kubernetes provides a way for containers in the pods to consume block or file storage. There’s also a COSI sub-project in SIG Storage that is trying to add object storage support in Kubernetes. In this session, Xing will talk about some features that SIG Storage is currently developing and designing, and highlight a few items that the Data Protection WG has been working on.

Deploying Cassandra to Multiple Kubernetes Clusters

Mon, 22 Nov 2021 16:00:05 +0000

Over the last few months, the K8ssandra team has been talking about building a new operator for K8ssandra. The first release of that work has officially arrived! We’re not at the finish line just yet, but we’ve hit the first major milestone: making it easier than ever to run Apache Cassandra® across multiple Kubernetes (K8s) clusters in multiple regions.

Principles for building Kubernetes Operators

Wed, 17 Nov 2021 16:29:34 +0000

The automation of data services on Kubernetes is enjoying increasing popularity. Often engineers are surprised by the complexity when writing their first operator. In this talk the distilled experience collected from years of automating several data services is presented as a collection of principles. These principles will be presented with vivid examples using PostgreSQL as an example. Learn from the PostgreSQL operator example which is being developed in conjunction with Kubernetes end-users. See how principles guide the automation while it is the end-user’s requirements leading to the prioritization of use cases to be automated next.

Macquarie Bank’s Journey to Running Databases on Kubernetes

Wed, 10 Nov 2021 01:23:38 +0000

Macquarie Bank migrated stateful workloads including databases to the cloud using Kubernetes. The talk goes over how they progressively matured automation while also going from self-managed Kubernetes on AWS to Google’s GKE. As early adopters of the beta version of StatefulSets, they discuss how it matured and how they added additional customizations and database awareness with kOps, Helm charts, and operators. They also discuss backup/restore, secure config management, and a roadmap for the future.

The future of Data on Kubernetes

Wed, 10 Nov 2021 00:55:07 +0000

The Kubernetes community has made major strides in supporting stateful workloads, but it’s just the beginning of how Kubernetes could be leveraged as a standard to revolutionize data: to make data declarative, just like Kubernetes. The Data on Kubernetes Community (DoKC) is an openly governed community of practitioners who share an interest in making this a reality.

Secure PostgreSQL on Kubernetes

Mon, 01 Nov 2021 18:39:21 +0000

What are the most important security requirements that we expect from a PostgreSQL deployment on Kubernetes? What challenges arise for stateful workloads? Discover how EDB has addressed them while building Cloud Native PostgreSQL to make it secure by default. The talk focuses on the 4C security model, with emphasis on cluster, container, and code. Then cover client and server CAs, certificates management, logging, auditing, and more.

Databases on Kubernetes panel discussion

Mon, 01 Nov 2021 17:28:47 +0000

As part of DoK Day at KubeCon NA 2021, we brought together Western Digital Senior Director, Strategic Initiatives Rick Vasquez, Oracle Product Manager Mickey Boxell and PlanetScale Software Engineer and Vitess maintainer Deepthi Sigireddi to discuss what the future may hold for database providers, moderated by Alex Williams, founder of The New Stack. You can access the DoK Day at KubeCon NA 2021 talks here.

The state of Kubernetes stateful workloads at DreamWorks

Tue, 26 Oct 2021 21:22:39 +0000

Interviewed by DoKC Head of Content Sylvain Kalache, Dreamworks’ Data Service Lead Ara Zarifian discusses how the company is managing stateful workloads on Kubernetes, key benefits, and challenges. DreamWorks currently runs 370 databases on over Kubernetes 1200 pods. Ara shares about the innovative standard they developed for Kubernetes operators. This innovation allowed them to grow the number of databases hosted on Kubernetes without needing to exponentially increase their DBAs headcount.

First Data on Kubernetes Report Published

Thu, 07 Oct 2021 15:04:50 +0000

Regardless of which survey you read, the use of Kubernetes is on the rise in organizations of all sizes. While Kubernetes was initially designed for stateless workloads, the community has made major strides in supporting stateful workloads. Today, we are excited to announce DoKC first research report about the state of data on Kubernetes!

New Panels Added to Data on Kubernetes Day Event @ KubeCon NA 2021

Wed, 06 Oct 2021 09:02:12 +0000

The Data on Kubernetes Day virtual event is quickly approaching. Register to learn from experts at the forefront of running data on Kubernetes. Today, we are excited to announce the addition of two new panels to our schedule:

A new chapter for DoKC

Thu, 30 Sep 2021 19:20:50 +0000

Today we’re beginning a new chapter in the Data on Kubernetes Community (DoKC) by announcing a new governance structure – with over 20 sponsors joining – to accelerate the emergence and development of techniques for the use of Kubernetes for data.

Data on Kubernetes Students Day

Tue, 28 Sep 2021 23:42:23 +0000

It’s always fun to make history, so we’re bringing you the first ever DoK Students Day! We’ve got nearly 30 talks lined up, and the best one will get a spot in our co-located DoK Day event in KubeCon!

Deployments vs StatefulSets vs Daemonsets

Fri, 17 Sep 2021 22:59:57 +0000

Kubernetes provides different resources for deploying applications, we will be looking at them and the differences between them and how can we persist data using each of them. Kubernetes provides different resources for deploying applications, we will be looking at them and the differences between them and how can we persist data using each of them.

Talks Schedule for DoK Day at KubeCon North America 2021

Wed, 15 Sep 2021 17:06:16 +0000

Today, we are happy to announce the second Data on Kubernetes Day schedule. This event will be held at KubeCon North America 2021 on October 12th! Thank you to every community member who participated in our CFP. Join us by registering here!

Intro to Persistence in Kubernetes

Thu, 09 Sep 2021 18:30:59 +0000

So you want want to run stateful containers? Check out a brief intro to persistence in Kubernetes. Kubernetes and persistent storage go together like oil and water. Kubernetes is inherently an ephemeral system and persistent storage by definition must survive. After this talk, you should have a clear understanding of how to get started on the path to successfully manage a persistent data storage solution on your Kubernetes cluster

How to Deploy PostgreSQL On Kubernetes Using OpenEBS

Thu, 09 Sep 2021 10:07:58 +0000

An Object-Relational Database (ORD) combines both entity-relationship modeling, and object-oriented mapping techniques to enable the efficient organization of data from a limited set of data types. As a result, this database model is often referred to as an intermediate solution between object-oriented and relational databases. Using Object-Relational Database Management Systems (ORDBMS), organizations can implement inheritance, enable complex data types that enable useful data manipulation, and extend existing data models.

Why you should be deploying Postgres primarily on Kubernetes

Thu, 02 Sep 2021 16:19:31 +0000

Running a Postgres installation, with or without containers, is trivial. However, setting up a production environment is a whole different matter. Postgres is not by itself a production-ready software: it requires a set of side tools to complement its functionality: connection pooling, monitoring, backup tools, high availability software, you name it. This is called the “Stack Problem”. This brief talk discusses the Stack Problem, understanding how Kubernetes is the platform that best solves it, and what the main advantages (and disadvantages!) are of running Postgres on Kubernetes.

The Future of Kubernetes for Stateful Workloads @ DoK Day

Wed, 25 Aug 2021 13:57:59 +0000

As KubeCon North America 2021 is quickly approaching, so is the CFP deadline for DoK Day. The Data on Kubernetes Community is an openly governed group of technologists who champion the emergence and development of techniques for the use of Kubernetes for data.

Explanation of NVMe for Everyone

Wed, 19 May 2021 19:00:17 +0000

I was talking to a friend recently about his computer build and one thing struck me. He had bought an NVMe drive not because he knew what it was or how it differed from anything else but because he felt overwhelmed by information when trying to research it. Due to this he just made his purchase based on the advice of a popular youtube channel. Working with storage professionally I was a bit taken aback by the lack of simple, plain english, easily accessible information on what the actual differences are. This blog is going to be my effort to clear the air and provide the layman’s guide to understanding what NVMe is as a technology and why it matters to you. This should be applicable all the way from DIY computer builders people setting up enterprise architectures.

Deconstructing Postgres into a cloud native platform

Tue, 09 Feb 2021 17:27:28 +0000

Postgres has been around for a long time in the DB space. Known to be a very reliable and sturdy technology even considered boring to some. But we were lucky enough to have Álvaro Hernández on to talk to us about Postgres innovations that are coming our way. If you want to re-watch the original click here. If you don’t have time, give this article a read.

The Architecture of Distributed Databases with / Cockroach Labs

Wed, 03 Feb 2021 21:45:28 +0000

Even though 2020 is over an 2021 brings a great deal of optimism and Hope not that much has changed apart from the changing of the calendar year. So many people still have Zoom meetings to attend to, children to take care of in the next room and a beautiful constellation of responsibilities that only a person who works from home has to deal with, so it’s totally understandable that you didn’t have time to check out one of the latest meetups that we held here at Data on Kubernetes. But fear not, in this article we will give you the lowdown on what was discussed, a summary with the key points of everything you need to know about the key points of the cockroach DB presentation. In case you have some time, and you like to watch up just follow the link.

Running Apache Spark on Kubernetes

Sun, 03 Jan 2021 13:30:26 +0000

Apache Spark is one of the leading frameworks for analyzing data in real-time with Spark Structured Streaming, this talk explains how to step by step run Spark structured streaming jobs on Kubernetes and access the Spark UI to see the jobs metrics and progress.

SRE vs Data Engineer — Who Owns What?

Sat, 05 Sep 2020 13:10:27 +0000

Photo by Lorenzo from Pexels

Demetrios

One question that I have been pondering lately is where do the requirements of an SRE end and a data engineer begin and visa versa?

bmath

A Call for Collaboration: Announcing the Data On Kubernetes Community

Wed, 26 Aug 2020 14:31:06 +0000

I’m happy to help announce the launch of a new community — the Data On Kubernetes Community. The Dok.community is an openly governed and self-organizing group of curious and experienced operators and engineers concerned with running data-intensive workloads on Kubernetes. DoKc takes inspiration from the CNCF and Apache foundations and aims like each of them to be open, vendor-neutral, and extremely inclusive.