Imagine, if you will, a series of tragic tales from dark dimensions where nightmares come to life and best practices go to die. The stories you’re about to witness are all true, pieced together from the shattered psyches of those who lived to tell the tales. Accounts have been anonymized to safeguard the unfortunate souls who were caught in the crosshairs of catastrophe.

As you follow this motley crew of unsuspecting engineers navigating the murky waters of automation, integration, and delivery, each alarming anecdote will become another foreboding reminder that it could all happen to you one day.

Prepare for downtime, disasters, and dilemmas. Your pulse will quicken and your hardware will cringe as you travel through the hair-raising vortex of DevOps Nightmares…

Automation is great because it lets you do good things quickly … But you can do terrible things quickly, too.

“It’s been five years and just thinking about it still makes me tired,” says the former VP of Infrastructure at the biometric security company where this DevOps Nightmare took place. “The whole fiasco was exhausting, physically and emotionally.”

At the time, the company was going through a growth spurt like a middle schooler with NBA dreams. Customers were signing up in record numbers to get various body parts scanned into the system, and the servers running the operation were redlining just to keep up.

No one had bothered to build auto-scaling infrastructure because it had always worked fine. But with the new demand it started wheezing like an old man trying to reach a fifth-floor walk-up with seven bags of groceries.

And then the plastic bags broke and servers started rolling down the stairwell. Systems were going down and customers were unable to access their biometric scan services. Let’s just say: the “stuff” was hitting the fan.

The classic scale-out story: build a new cluster and maybe automate it

Under pressure from higher-ups to get things working NOW, the DevOps team was called in to wave their magic wands. They first tried their go-to spell: slap together a new server cluster. But thanks to the platform's complexity and the manual configuration work required, the whole process took five hours plus another hour for the data to trickle in.

Even after all that effort, the new cluster only worked for a couple days, keeling over like a cow tipping contest gone wrong. Back to square one!

“There was a gremlin in the tech stack and we couldn't figure out what it was,” recalls the former Infrastructure VP. “Then someone said they could write the automation to build new clusters and it would ‘only’ take a week.”

The DevOps engineer turned half a day’s cluster provisioning work into a 20-minute automated job. He was treated like someone who stopped a runaway train with his bare hands. While the automation didn’t fix the performance issues entirely, the DevOps team could rapidly scale capacity on demand.

Every silver lining has a cloud (not that kind)

But then, after an all-hands-on-deck week or two of people working long hours and not sleeping much, Fate’s Fat Finger intervened. The engineer who solved the original problem created an even bigger one with a single mistaken keystroke: instead of adding a cluster, he deleted all the existing ones, making the entire platform inoperable. Behold, the incredible power of automation!

The former VP of Infrastructure recounts what happened next: “He came to me whiter than a ghost, trembling as he said, ‘So...I think I just took us offline. Like, completely offline.’ I had to break the news to the CTO and the CEO that we’d be out of order for about 12 hours. As you can imagine, they were not pleased.”

The exec team was emotionally in pieces. The same DevOps engineer who solved the automation problem also brought down the entire company. Heroics one week, company-crushing news the next. Do they promote the guy or fire him?

Despite the tragic accident, in the end most people came to see that automating cluster creation relieved more heartburn and ulcers than it caused. The managers who knew this stuff is difficult got past their disappointment, and the engineers gained a new respect for peer review and the importance of proceeding with caution, especially when exhausted and addled.

The engineer who came up with the solution that saved the company stayed with the company for a few years, but he was never totally forgiven for the mistake he made. Fortunately, he moved on to bigger and better things.

Have your own tales of automation woes or delivery disasters? We want to hear them! If you've endured a devastating DevOps debacle and are willing to anonymously share the cringe-worthy details, please reach out to us at 

Don’t hold back or hide the scars of your most frightening system scares. Together, we can immortalize the valuable lessons within your darkest DevOps hours. Your therapy is our treasured content, and we’ll gracefully craft your organizational mishap into a cautionary case study for the ages. And, in return for your candor, we'll ship some sweet swag your way as thanks.

Imagine, if you will, a series of tragic tales from dark dimensions where nightmares come to life and best practices go to die. 

The stories you’re about to witness are all true, pieced together from the shattered psyches of those who lived to tell the tales. Accounts have been anonymized to safeguard the unfortunate souls who were caught in the crosshairs of catastrophe.

As you follow this motley crew of unsuspecting engineers navigating the murky waters of automation, integration, and delivery, each alarming anecdote will become another foreboding reminder that it could all happen to you one day. 

DevOps Nightmares — Fate’s Fat Finger

How do you evaluate if Generative AI is something you should add to your solution?

In the fast-paced software ecosystem, Generative AI is a game-changer. But first, let’s explain what it actually is. GenAI is a subfield of Artificial Intelligence that focuses on developing systems that generate new content or data. It involves training machine learning models to learn patterns from existing data and use that knowledge to create unique, original output. This powerful technology promises to transform every industry by enabling machines to develop, learn, and generate new content.

 expertise in the healthcare industry, we hope to give readers new perspectives on the possibilities of generative AI; many apply to any field, and many others are assessed from the Healthcare industry's point of view. We'll examine how to determine whether GenAI is a good fit for a particular project and its advantages in digital product development.

When is Generative AI suitable for a digital product?

While generative AI offers numerous benefits, assessing its suitability for a specific industry, such as healthcare, is crucial. Here are three factors to consider:

For generative AI to produce valuable outputs, there must be a large amount of high-quality training data. Check to see if you have access to a big enough, diverse dataset to support generative AI model training.

Consider the complexity of the problem your solution aims to address. While conventional machine learning techniques can yield satisfactory results for simpler problems, the use of generative AI may be necessary when dealing with intricate and diverse datasets.

2. Clinical Natural Language Processing (NLP) - Extracting Insights from Medical Texts:

Regulations regarding using artificial intelligence and generative AI (

) in healthcare vary by country and region. It's important to note that the regulatory landscape is subject to change, and new guidelines are being introduced as these new technologies advance.

For a highly regulated domain such as digital health products, deploying generative AI in healthcare products may introduce additional regulatory considerations. Let’s break down the main 3.

 Health Insurance Portability and Accountability Act (

) in the United States; General Data Protection Regulation (GDPR) in the European Union.


These regulations focus on the protection of patients' sensitive health data. When using generative AI in healthcare, it's crucial to comply with data privacy regulations to ensure patient information's secure handling and storage.

 CE marking in the EU; US Food and Drug Administration (FDA) regulations in the United States.


If the generative AI system is considered a medical device, it may need to adhere to specific regulatory requirements to ensure its safety and effectiveness. This is particularly important for AI systems involved in diagnosis, treatment planning, or decision-making.

3. Clinical Validation and Certification:

Regulatory bodies may require evidence of the clinical validity and reliability of AI applications in healthcare. Pre-market certification or post-market surveillance processes may be in place to ensure the safety and efficacy of these applications.

Massive volumes of medical data, including electronic health records, medical imaging, and clinical notes, serve as the foundation for training generative AI algorithms. This enables them to generate synthetic data closely mirroring authentic patient information.

The power of generative AI lies in its ability to simulate and furnish precise data applicable across diverse tasks. This technology can augment limited datasets, generate synthetic patient profiles for simulation purposes, and formulate fresh patient records for training and validation. This empowers physicians and academics to delve into a broader spectrum of scenarios, experiment with concepts, and develop more refined models using generative AI.

Given its many advantages, generative AI is an attractive method for healthcare solutions and provides immense benefits.

Data must be anonymized and de-identified to adhere to privacy laws and safeguard patient confidentiality. With the assistance of generative AI, synthetic data can be produced that preserves the characteristics of the original patient data while excluding personally identifiable information. This artificial data can be used for research, training, and development without violating patient privacy.

Because each patient is different, effective treatments for one person may not be effective for another. Generative AI can use patient-specific data to create individualized treatment plans and suggestions. By considering specific patient features, generative AI algorithms can aid in optimizing treatment choices and enhance patient outcomes.

The breadth and diversity of healthcare datasets are frequently constrained, which makes it difficult to develop reliable models. With the help of generative AI, current datasets can be supplemented with new data, expanding and diversifying it. This enlarged dataset can improve the performance and generalizability of machine learning models.

The development of simulated scenarios for testing and evaluating healthcare products is possible by generative AI. Healthcare entrepreneurs can evaluate the efficacy and robustness of their algorithms and software solutions in diverse clinical contexts by producing synthetic patient data. This may even save time and resources during the development and testing phases.

The potential of generative AI is immense. Combined with the right tech stack, 

, and a thorough understanding of its capabilities, advantages, and suitable applications, you can make well-informed decisions that foster innovation and significantly enhance people's lives.

Adding Generative AI to your Digital Product 

We just opened a time capsule from 2003 and couldn't believe what we found. An AOL CD-ROM with 2,000 free hours; a Dell blade server chassis; and a Blue Collar Comedy Tour DVD. We couldn't install AOL, the blade server power supply was dead, but our Blue Collar DVD works just fine. We were reminded again about Jeff Foxworthy’s “

” skits as the star of the show. So we channeled our inner blue collar comedy and applied it to container orchestration—because, hey, why not?

If you rewrote 20 Amazon CloudFormation scripts last week...you might be building a PaaS

It’s not 2003 anymore, but it was way easier back then to know what you had running. You couldn’t spin up new clusters and configurations. Your server literally existed in meatspace or it literally didn’t. Today in 2023, you’re spending hours rewriting CloudFormation scripts provisioning servers like a jackhammer. Every. Damn. Day. Just before you punch a hole through your monitor, you import (or write) a tool to handle updating the scripts. Before you know it, you’re on your way to building a kickass PaaS that’ll work until next week.

If you need to learn a new kubectl command every day…you might be building a PaaS

Remember destroying all those little aliens hopping on the screen in the game Space Invaders? Managing k8s clusters can feel a bit like that. Applications and services keep showing up in new clusters, eating resources. And it’s your job to shut them down before everything’s out of control. 

So you download kubectl to control your Kubernetes clusters. And then helm. Then kubectx and kubens and stern and k9s and bootkube and ktunnel. Oh, and Kops. Then maybe monokle to see how that goes.

Soon you’ve got more tools in place to manage your Kubernetes infra than domain names you bought last new years eve. Maybe it’s time to buy a ream of paper for all the pages you’ll need when you print out your “kubectl cheat sheet” - or the docs for what might as well be your new PaaS.

If you find yourself checking to see if there are any updates to Heroku you want to copy…you might be building a PaaS

Hate Heroku, the great grandaddy of PaaS? When you started out with Kubernetes, you took a look at Heroku and rejected it because it wasn’t cool anymore or because some DevOps engineer laughed when you suggested the idea. More realistically, you probably felt like it was too much of a black box. If Heroku didn’t support it, then you were stuck. No flexibility. 

But now that you’re building out your k8s you’re finding that you’re boxing yourself in. Maybe Heroku has a feature you really would like to use. Maybe you can copy it? Maybe Stack Overflow has the answer. Maybe you’re building a PaaS after all?

If you find yourself chasing the perfect service catalog (that devs will refuse to use!)…you might be building a PaaS

Building a service catalog is a necessary evil. You need rigor, you need documentation, you need maturity and management. Most of all, you need your dev teams to use it.

But let’s be honest: they won’t. Deep down in your heart of hearts, you just know they’re going to tear it down in some backchannel Slack discussion or subtweet (sub-X?) you.  

The more you perfect it, the more you hear that it’s not ready. It’s finally dawning on you that it’s *never* going to be good enough for them. But you are so far into your sunken cost that you haven’t realized that you’re halfway to building a PaaS.

If you’re building a Cloud Platform team instead of your product…you might be building a PaaS

NOW HIRING: Engineers who can build a platform for engineers who can deliver value to the market.

Getting your cloud platform right is the secret to getting your product right—running smoothly, scaling up (and down) just as you need. It’s so easy. Just get a few cloud-native DevOps engineers on staff and you’re golden.

The actual problem? There’s a talent shortage. Everyone from 

 to the donut shop around the corner is dealing with it. It sucks. How are you supposed to get your PaaS-like-but-totally-not-a-PaaS provisioning and orchestration platform up and running?

If you’ve used the words “developer experience” when talking about building tools for you infrastructure…you might be building a PaaS

It started so simple. A little JIRA request, a little Cloud Formation templating, a little auto-generated YAML—you were off to the races. You built it at a hackathon. It was going to change the future of the company. Then one of your devs said, “The developer experience around this needs to be better.”

You’re thinking, “Be better? Are you kidding me? I built this in 4 hours with nothing but a bowl of spicy Takis and a 6-pack of Liquid Death. I’m a genius. You should be worshiping at my feet right now.”

Besides, wtf is a “developer experience?” We’re building tools for kubernetes. We’re lucky to even have me here to be orchestrating all this. You don’t like it? Go build your own platfor–uh, service.

If you’re building internal docs and runbooks to keep your services running…you might be building a PaaS

You wrote the main page. That should be good enough, right? Nope.

These cloud services are getting complex. But what’s the limit? How many edge cases do you need to describe? How much do you need to write? You find yourself adding more and more background and pointers to platform docs just to get your team up to speed. The devs keep saying the docs need to be clearer and add more context. You might as well be writing Kubernetes for Dummies. Wait—that’s actually a good idea!

Obviously, this is all in good fun, but if any of this sounds remotely familiar, you gotta ask yourself: What are you doing writing all this stuff? It’s a solved problem. Like, okay, Heroku may not be it anymore. But is there really nothing out there that can solve your Kubernetes problems in the 2023rd year of our lord? It’s not just the time you’re spending. It’s also the cost. Build vs. buy is what they say. If you don’t have a really good reason to build then take that money you are spending on building and just buy a solution. Done.

Of course there is. I mean, there’s a freaking 

 out there. Why wouldn’t there be something that fixes cloud provisioning and orchestration?

Listen, we’re not saying you need to try Aptible… but you should probably try Aptible.

We just opened a time capsule from 2003 and couldn't believe what we found. An AOL CD-ROM with 2,000 free hours; a Dell blade server chassis; and a Blue Collar Comedy Tour DVD. We were reminded again about Jeff Foxworthy’s “You might be a redneck” skits as the star of the show. So we channeled our inner blue collar comedy and applied it to container orchestration—because, hey, why not?

You might be building a PaaS if…

Kubernetes (K8s) often catches users off guard with its complexity. Harnessing its full potential requires paying attention to detail and keenly understanding its core concepts. If you don’t, K8s can fall apart quicker than a wet paper straw.

J.R.R. Tolkien once said, “It does not do to leave a live dragon out of your calculations if you live near one.” Similarly, while Kubernetes offers enormous power and flexibility for container orchestration, you’ll find that even the most experienced professionals have trouble navigating its configuration and services. 

Platform practitioners that manage kubernetes-based platforms often experience significant frustration as they encounter a multitude of challenges, ranging from intricate configuration requirements and unpredictable expenses to limitations in scalability, concerns surrounding secrets management, and a steep learning curve.

Kubernetes is an efficient platform for managing containerized applications. Dev teams use it to quickly deploy and orchestrate applications across a cluster of machines, automating many tasks that would otherwise be time-consuming and error-prone. For example, developers can deploy and manage containerized applications, automating tasks like load balancing, scaling, and self-healing, which streamlines application deployment, ensures high availability, and simplifies infrastructure management. 

A critical aspect of its orchestration capabilities is dynamic scaling, which efficiently allocates resources and seamlessly handles fluctuations in workload demand. Instead of manually monitoring usage and bandwidth, k8s manages it nearly autonomously.

Magical as k8s may seem at first, there are plenty of challenges to overcome to be production-ready. 

Understanding these challenges is essential for anyone who wants to deploy Kubernetes in production. Kubernetes is a complex system, and there are many things that can go wrong if it is not properly configured. Here are some of the most frustrating challenges that Kubernetes users face:

 conducted by Civo, 54% of cloud developers revealed that Kubernetes complexity is slowing down their organization's use of containers. 

One of the main sources of this complexity is writing and managing YAML/JSON manifests. Manifests are configuration files that outline how to use resources within a cluster. Writing them is time-consuming, prone to misinterpretation, and immensely frustrating. There are so many object types in K8s, each with its own specific requirements and specifications, users may find themselves overwhelmed when working with manifests.

 represent different resources, entities, or components within the Kubernetes system, such as pods, services, deployments, and config maps. A piece of code manages each of these object types, and the life cycles of all these objects work in tandem to orchestrate the system as a whole. 

Coordinating all these components requires esoteric knowledge. Depending on the application, you may need to specify additional parameters for certain object types, especially for a 

 (the workload API object used to manage stateful applications). None of this is well-documented—and even if it was, the docs assume a high degree of k8s knowledge to begin with.

Working with a StatefulSet is ideal for managing stateful applications with unique network identities, persistent storage, and ordered deployment and scaling. StatefulSets sometimes require many additional parameters. These parameters, tailored to your application's requirements, typically include:

However, simply configuring an application is not merely a “one and done” task; it typically needs a dedicated DevOps team willing to regularly scan Kubernetes clusters and ensure their proper configuration. This process involves validating pod resource limits and security policies to ensure a smooth operation. Kubernetes administrators also need to evaluate, select, install, and manage myriad 

Depending on the application, K8s administrators may need to install Ingress controllers such as:

With Kubernetes, complexity begets more complexity. As you add more clusters and components, the configurations get more intricate, thanks to the expanded requirements and the challenge of keeping everything consistent in a distributed system. Each new Kubernetes cluster has its own set of resource needs, including storage strategies, networking protocols, and computer nodes.  

As Kubernetes scales, the complexity of maintaining consistent networking policies across multiple pods intensifies. An illustrative challenge in this context is when a network policy selects a specific pod, as it necessitates explicit matching of traffic to and from that pod with a network policy rule to avoid blocking. While there are no inherent restrictions on Pod-to-Pod communication within a Kubernetes namespace, administrators must carefully consider network policy rules to ensure uninterrupted traffic flow and adequate network security.

In addition, the dynamic nature of the Kubernetes ecosystem introduces more complexity. With regular updates, the introduction of new features, and community-driven enhancements, staying current with best practices and emerging trends becomes essential for effectively navigating the evolving Kubernetes landscape.

, "While Kubernetes-based applications may be easy to run, Kubernetes itself is no picnic to operate."

The biggest problem for K8s is uncontrollable costs. In a recent 

 survey, 68% of respondents said Kubernetes costs are increasing, with half experiencing an increase of more than 20% per year. 

Overprovisioning resources occurs when an enterprise fails to carefully monitor spending and loses control over the costs involved. In the CNCF survey, 24% of respondents did not monitor Kubernetes spending at all, while 44% relied on monthly estimates. Only a relative minority employed predictive Kubernetes cost monitoring processes.

When configured correctly, applications are responsive, even under heavy loads and traffic spikes. Improper configuration of K8s can lead to excessive scaling of an application, resulting in over-provisioning. 

Over-allocated resources affect performance by wasting and hoarding shared cluster resources. When deployments request more resources, like CPU and memory, than they actually need, it can cause application performance to degrade. A web application configured to run hundreds of instances may overwhelm available resources, resulting in slow response times, increased latency, and even system crashes well beyond the problematic container.  

Since these environments are dynamic, K8s engineers can unintentionally overprovision their applications if they fail to account for fluctuations in usage. For example, a K8s engineer might configure a web application deployment to use 16 CPU cores and 64GB of memory, even though the actual usage of the application only requires 4 CPU cores and 16GB of memory. Recently, Webb Brown, CEO of Kubecost, a Kubernetes cost monitoring firm, 

 that many teams typically begin with a low 20% cost efficiency.

Teams often waste this extra capacity and don’t factor it into their overall cost calculations. Software company Flexera 

 that teams typically exceed their cloud budgets by an average of 13% and waste approximately 32% of their cloud expenditure due to issues like overprovisioning that leaves cloud resources unused. 

Kubernetes engineers are unlikely to employ usage reports to identify if they can cut their spending without sacrificing a comfortable buffer. Consequently, they must decide between wasting money or introducing risk, leading to overprovisioning as the default option. 

Attempting to gain control over this spending can be incredibly frustrating. As one Chief Technology Officer 

, “I spent the last two years developing with Kubernetes and containers. In one moment, I’d rave about how it handled mundane infrastructure management tasks. In the next, it frustrated me to comical degrees because it was almost impossible to control my own environment.”

This lack of control is primarily due to Kubernetes' proactive creation and disposal of container instances to meet demand, resulting in unpredictable resource usage. This volatility will challenge anyone seeking to track usage levels and allocate overhead expenses. 

Moreover, in multi-cloud environments, users may receive separate bills from different service providers for their clusters, making it hard to even identify specific containers. 

For example, suppose you have a Kubernetes cluster running on AWS and another on Azure. With such an arrangement, you will receive separate bills from AWS and Azure for the resources you use in each cluster. Tracking your overall spending becomes challenging, as well as identifying which containers use the most resources. Additionally, each cloud provider uses its own naming convention for containers, complicating identification.

Autoscaling is a prominent feature of cloud native architectures, and it often involves using on-demand containers ("if it even exists, you pay for it") to meet workload demands. To make the most of your resources, it's essential to embrace rightsizing in Kubernetes, as it allows you to optimize costs effectively.

Rightsizing is all about finding the perfect fit for container sizes, aligning them with the pods' ideal CPU and memory needs. It’s all about making the most of your resources without sacrificing performance. Adopting this approach can significantly cut expenses since you won't be paying for unused resources, making your Kubernetes deployment much more cost-efficient. However, achieving this goal may pose some challenges.

When demand spikes occur, scalability limitations impact the application's performance if resources are scaled up too late (e.g., Out of Memory errors, CPU throttling, or increased latency). Failure to rightsize leads to inefficiencies and bottlenecks. On the other hand, if a user overprovisions resources, autoscaling may increase the cost of running the application, potentially outweighing any performance benefits. Thus, it's not wise to leave resource provisions up to chance. 

Monitoring creates the right balance of resources

Instead, organizations need to closely monitor their applications to achieve the right balance in resource allocation.

Given this need, Kubernetes offers an array of scaling tools to facilitate resource requests:

Kubernetes users should know the drawbacks and difficulties of its three autoscaling techniques. Understanding them can prevent inadvertent overscaling of applications. For instance, when calculating scalability, neither HPA nor its VPA factor in network, storage, or input/output operations per second. As a result, applications risk slowing down or even grinding to a halt. In addition, VPAs do not allow users to update resource limits on active pods, necessitating their removal and the creation of new pods when new resource limits are needed.

Kubernetes Cluster Autoscaler (CA) presents its own challenges. When making scaling decisions, it only considers a pod's resource requests rather than its actual usage. Consequently, the CA will fail to detect any unused resources the user may have requested, creating inefficient and wasteful clusters.

Also, the infrastructure they run on also needs to be scaled, making it necessary to use multiple mechanisms. Merely relying on the HPA or CA alone may not suffice. To effectively manage scalability, all three mechanisms may need to be used in tandem. Neglecting these tools can lead to heightened frustration and significant costs. 

 are not really secret. Although Kubernetes employs the Secret object to store your sensitive data, the data is represented in 

. Consequently, anyone with access to the cluster or the code can decode the K8s Secret.

By default, Kubernetes stores these unencrypted Secrets in the API server's underlying data store (

). Since anyone with API access can retrieve a Secret, anyone with access to etcd can do so as well. In addition, anyone who has permission to create a Pod in a 

 can use that same access to read any Secret stored there, even if they only have indirect access. 

Thus, Kubernetes users should either employ a third-party tool to remedy this situation or thoroughly reconsider the storage of any sensitive data within the platform.

After all, hacking attacks are hardly rare occurrences. Potential attackers constantly search online for exposed Kubernetes components protected by lax access controls, such as access to API servers. In 2022, 

 revealed that over 380,000 K8s API server instances are exposed over the internet every day. In addition, configuring cluster access and authorization roles is highly complex in K8s.

Kubernetes users should be aware that adding components to the Kubernetes environment increases the overall attack surface, including the exposure of secrets. One such component, the Kubernetes Dashboard, presents a web-based interface for managing and visualizing the cluster. Improper configuration or security vulnerabilities in the Dashboard can introduce risks, particularly when it's accessible on the public internet without robust authentication and authorization measures. Unauthorized access to the Dashboard grants attackers the ability to view sensitive information, manipulate resources, and potentially gain access to stored secrets within the cluster.

Introducing new components may also require additional configuration steps and integration with secrets management systems.

Alas, Kubernetes’ security risks appear to be a growing concern for businesses. Earlier this year RedHat 

 600 DevOps and security professionals about the state of K8s security. They found that 67% of respondents have experienced delays or slowdowns in application deployment due to security concerns. In addition, just over a third of respondents reported experiencing revenue loss or customer attrition due to a container and Kubernetes security incident.

Navigating Kubernetes can be daunting. Taking a few hours to learn Kubernetes won’t get anyone over its steep learning curve. It's a complex tool solving a complex problem. In the aforementioned 

, 57% of respondents reported Kubernetes' steep learning curve as their top challenge. 

What makes working with Kubernetes so difficult? Users must familiarize themselves with the software's three essential components: the containers, the kubectl command-line tool, and the Kubernetes manifests.

Among these three components, mastering the art of writing YAML files is the most challenging. While the YAML format is relatively straightforward, the key lies in meticulously structuring and configuring each YAML file to guarantee flawless functionality within the Kubernetes environment. Attentiveness to details such as indentation, syntax, and accurate placement of key-value pairs becomes imperative for seamless deployment. Additionally, YAML files must be thoroughly validated and error-free to prevent deployment issues and unexpected behavior. 

Understanding the underlying logic behind K8s objects is even more demanding. Users must grasp the concepts of pods, services, and deployments, comprehending their functionality and interconnectedness within the Kubernetes ecosystem. They should also comprehend how these objects interact and collaborate with each other.

Kubernetes engineers typically undergo specialized training to work effectively with the platform, since setting up a K8s cluster demands a deep understanding of its architecture and components. This process involves provisioning the infrastructure, including installing and configuring crucial components like the 

. It also involves setting up networking and cluster networking solutions and deploying and configuring storage options. Finally, Kubernetes engineers must be able to add and configure additional tools like monitoring systems and logging solutions.

While Kubernetes offers advanced container orchestration capabilities, the complexity involved in mastering the platform can be daunting and time-consuming. Due to Kubernetes' steep learning curve, enterprises must carefully consider the trade-offs involved in adopting the software. Since the benefits of using K8s may not outweigh the investment required in training and resources, it is essential to consider alternative solutions.

To recap, here’s a list of the issues we’ve discussed in this article. This list is by no means comprehensive as there are many other challenges to working with Kubernetes, but it’s enough to strike fear into even seasoned infrastructure engineers:

Despite Kubernetes' continued popularity, the container orchestration platform still provokes considerable frustration. Kubernetes users face numerous challenges, including complex configuration demands, costly or unpredictable expenses, scalability limitations, secrets management concerns, and a steep learning curve. Nevertheless, many enterprises persevere in adopting K8s due to its unmatched ability to orchestrate containerized applications.

Cloud platforms simplify deployment by abstracting the underlying infrastructure, allowing users to focus on application development and deployment rather than managing Kubernetes clusters. Furthermore, cloud platforms use Kubernetes' scaling capabilities, offering simple scaling options and auto-scaling features to improve performance and resource utilization.

There is a growing demand in the market for an alternative to Kubernetes that supports globally-scalable application delivery. Cloud platform solutions like Aptible are becoming increasingly popular. Aptible allows developers to quickly scale their applications without worrying about infrastructure requirements. As time progresses, we expect more businesses to show interest in cloud platform products as they recognize their many benefits.

Explore the challenges of container orchestration and scaling in Kubernetes. Discover strategies to overcome these obstacles and ensure efficient management of containerized applications in dynamic environments. Gain insights into scaling considerations, resource allocation, and workload distribution to optimize performance and maximize the benefits of Kubernetes orchestration.

Kubernetes Challenges: Container Orchestration and Scaling

Developers are questioning the future of platform as a service (PaaS)—and for good reason. Many of them have built successful products on PaaS, but when their app gets to a certain scale, they find themselves needing more. They can’t easily debug performance issues due to a lack of transparency and opinionated black box features. They can’t implement fully-configurable security and compliance controls like a web application firewall. They’re stuck with whatever limited add-ons the platform offers, if they’re even offered at all. Features like persistent storage, configurable load balancing and auto-scaling are either non-existent or too limited for real-world use on PaaS.

So what do developers do? They port their apps over to IaaS.

It’s not that IaaS is better, but that it has no implementation limits. Devs can build anything they want with IaaS providers. The trade-off is one of time, expertise, and additional staffing. But that’s okay for most developers with successful PaaS apps because they’ve already gotten through the hard part of building something and finding product market fit. They’re not as worried about the learning curve of migrating to IaaS because by that point, they have a user base, revenue, and runway.

It’s likely devs wouldn’t port their apps to IaaS if PaaS was a viable option at scale. But does that mean PaaS should be written off as inferior to IaaS? Hardly. The future of PaaS is bright—much brighter than many developers recognize. In fact, I believe PaaS will dominate the future of software development. The market is hungry for a “great to start, great to scale” platform to emerge and leapfrog today’s developer experience and value. PaaS will absolutely support globally-scalable app delivery, but it’ll have to overcome some challenges first.

The beauty of IaaS is that it offers near-infinite configuration and scalability options. IaaS providers have generalized their products enough to support practically any use case at any scale. It’s incredibly powerful and cost-effective. Plus, IaaS providers are happy to give usage credits and discounted pricing to drive adoption and get teams hooked.

However, it can be incredibly difficult to build scalable configurations without shooting yourself in the foot. Incorrect configuration changes can have catastrophic results that are nearly impossible to troubleshoot and remediate without deep infrastructure expertise. There’s practically no platform support available (outside of costly enterprise engagements) and IaaS documentation is notoriously inconsistent and difficult to read. And don’t even get me started on the actual user experience of IaaS consoles.

Going all in on IaaS means building expensive infrastructure teams to use and manage IaaS. Not only are infrastructure experts some of the most in-demand people in the industry, it’s extremely difficult to find the right people who know how to get through unique growth needs. Adding to these complications are the realities of a tighter fundraising environment, teams being told to “do more with less,” and the rapidly increasing complexity of IaaS solutions. The work of the team is to set up and maintain the infrastructure as it grows—and the job is never “finished.”

The typical journey companies go through with IaaS looks like this:

Between funding issues and lack of available infrastructure talent, it’s hard to recommend deepening investment in infrastructure and platform work. IaaS features and capabilities are growing increasingly complex, further justifying the need for higher-level abstractions, just like the ones provided by PaaS.

IaaS providers know this, which is why they’re making investments in PaaS-like abstractions. Google’s doing a pretty good job at this with their Cloud Platform and 

, their mobile development platform. Amazon has been trying to build abstractions, but their developer experience is… not good. Nevertheless, it’s clear they understand that developers want (and deserve) better experiences, like 

, which allows for simplified deployment on auto-scaling infrastructure.

The major cloud providers are getting better at PaaS-like offerings, but they’re also focused on legacy lift-and-shift strategies or sci-fi “serverless Kubernetes” fantasies. They make most of their cash from enterprises, which biases their feature development to serve more enterprises. Startups and mid-size companies are left crossing their fingers hoping their feedback leads to easier-to-use and easier-to-build PaaS-like environments that don’t require advanced infrastructure expertise to fully utilize.

 that “at least two major hyper-scale cloud providers (Azure and GCP) started their cloud journey with PaaS products, only to quickly realize that real money (and loads and loads of muscle and institutional memory) was still in infrastructure. So, they pivoted to being IaaS-centric (like AWS was at the time).”

In other words, IaaS would be doing PaaS if IaaS wasn’t so lucrative. They’re not serving startups, they’re serving enterprises.

IaaS is not a competitive advantage, it’s a distraction

Building in IaaS does offer plenty of freedom and opportunity, but would you rather spend $150K on an AWS-certified admin or $150K on a developer who can build revenue-generating features?

If you don't have to worry about infrastructure, hiring the developer is a no-brainer. Unfortunately, it’s not uncommon for companies to get the worst of both worlds: they hire the $150K developer and force them to also do all the DevOps work. That’s much less bang for the buck.

There’s really no competitive advantage for product teams that move from PaaS to IaaS, nor for those who start on IaaS and scale up. At some point, it will inevitably require hiring an expensive and experienced cloud infrastructure architect. Of course, I have nothing against skilled cloud infrastructure engineers, but when I’m facing increasing financial pressure to “do more with less,” I’m not very excited about increasing my overhead.

Besides, even if I had the world’s greatest cloud infrastructure team and even if it wasn’t as costly as I hypothetically fear, what is the competitive advantage of having that team? What advantage does that really offer any business? If I’m being honest, I’d much rather have a lean team of value-generating devs than value-protecting admins. Yes, I can spend the time and energy on IaaS to get persistent disks, point-in-time recovery on my databases, configurable load balancing, blue/green deployment logic, and auto-scaling, but I guarantee most developers would rather have all that available with a checkbox in a nicely-designed UI.

The more time I have to focus on keeping the lights on, the less time I’m spending on delivering great features to customers who want to spend money on my products. More infrastructure, less value.

The main reason PaaS exists in the first place is because IaaS is too complicated. Developers want to write and deploy code. They want to build new apps and features, see how the market responds, and iterate. And they want to do this with the fewest hurdles possible.

For example, if I need HIPAA-compliant infrastructure to build a healthcare-focused SaaS, I can either start with 

, which is incomplete and complex, and spend weeks getting it up and running. Or, I can spin up an app on Aptible in minutes and know it’s HIPAA-compliant.

The same goes for PCI compliance. Stack Overflow and Reddit are filled with examples and warnings of the uphill battle for establishing and maintaining PCI-compliant infrastructure in any of the big IaaS clouds. Or, you could find a PCI-compliant PaaS and spend zero time on firewall configuration.

Here’s the thing: no one is choosing to start with complex infrastructure architecture and implementation. They just don’t have many options or they’re not aware of the existing choices. Any developer or product leader would choose PaaS if it was cost-effective and capable of serving long-term scaling needs. PaaS doesn’t have a problem of cost effectiveness, but of painting itself into the corner with its long-term ability to support growing businesses. If that weren’t the case, PaaS would be a no-brainer for starting and scaling apps.

This raises the question: who wouldn’t pay some sustainable amount of extra money to have everything “just work,” excellent documentation, and built-in support?

IaaS has done a great job at offering a generalized solution that works in any use case. PaaS hasn’t gotten there yet. There are too many smaller missing features and obfuscations that cause teams to move off of PaaS. There’s no single PaaS that can support every app a company needs to run, and that’s a non-trivial problem.

Furthermore, application development patterns have also evolved since PaaS began to flourish in 2010. Heroku and its ilk just haven’t kept up. Here are some of the common engineering team “jobs to be done” that are essential to business, but aren’t being covered by PaaS: 

The needs are clearly diverse. The PaaS world has done a great job with specialized platforms that clearly position themselves as best-in-class at a single task, but the world needs a PaaS that can support a collection of business-enabling features.

PaaS needs to support more diverse application architectures

PaaS apps tend to be monolithic so as to take advantage of the various capabilities of the underlying infrastructure platform. However, growing products are likely to eventually need to split into a set of dependent microservices.

Right now, devs have to go through contortions to keep a microservice architecture working correctly on PaaS. They need constructs like service discovery and direct container connectivity to function efficiently. Those are hard to find as native capabilities in the PaaS world.

There’s a problematic dynamic that appears when apps become more complex than the PaaS can understand. It’s a big reason for developers to move to IaaS. If a PaaS can’t support a data warehouse, then there are few reasons to keep anything else deployed on the PaaS.

IaaS users yearn for PaaS-like abstractions

Once a company moves from PaaS to IaaS, paradoxically they’ll yearn for the things that make PaaS great: reduced complexity, a great developer experience, excellent documentation, and predictable costs. Read any “State of DevOps” reports today and you’ll hear about the “platform engineering” movement and its philosophical foundations.

But companies shouldn’t need to reinvent the platform. It’s a major cost to productivity. Creating and evolving a top-notch developer experience is enough work to become a line of business unto itself. Building abstractions over IaaS will likely have little-to-no ROI for companies outside the Fortune 50 (much less for their customers and shareholders). It’s a nonsensical business proposition.

TweetEmbed

PaaS has lots of room to grow—and we’re bullish

PaaS solves a lot of the problems of IaaS:

We see an opportunity for a general-purpose SaaS that doesn’t paint developers into a corner and we are excited. We’re taking what we've already built—the best and only non-Heroku PaaS that scales to meet enterprise needs—and making it easier and more accessible to start building apps.

Here’s how we’re expecting the market to respond (and how we’re approaching it ourselves):

As AWS and other IaaS providers add more and more services to their console, we have to expect more complexity. A growing number of IaaS customers will end up relying on what will soon be legacy cloud architectures and development paradigms. These companies will inevitably seek a PaaS to break free, or even to simply avoid the complexities and frustrations of dealing with utility IaaS.

I’ll close by bringing special attention to developers in smaller, relatively independent teams within larger companies. The "you build it, you run it" movement will likely be a driver for increased PaaS adoption. In fact, it’s already becoming “you build it, you run it, you secure it.” As dev teams become more responsible for their infrastructure, they’ll be responsible for their own DevOps. There’s only so much they’ll be able to manage, so PaaS will be more than appealing. PaaS will become a necessity.

People may be bearish on PaaS today, but the day is coming when these teams demand a true enterprise-grade Heroku competitor. A platform is going to blaze that trail and it will catch on faster than Heroku ever did. 

In 2023, Aptible will show that it is that platform.

Blog: Thoughts & Ideas

DevOps Nightmares — Fate’s Fat Finger

Adding Generative AI to your Digital Product

You might be building a PaaS if…

Kubernetes Challenges: Container Orchestration and Scaling

Return of the PaaS

You need a security-focused platform

Building a developer
platform is a thing of
the past.

Product

Resources

Support

Company

Blog: Thoughts & Ideas

DevOps Nightmares — Fate’s Fat Finger

Adding Generative AI to your Digital Product

You might be building a PaaS if…

Kubernetes Challenges: Container Orchestration and Scaling

Return of the PaaS

You need a security-focused platform

Building a developer platform is a thing of the past.

Product

Resources

Support

Company

Building a developer
platform is a thing of
the past.