Imagine, if you will, a series of tragic tales from dark dimensions where nightmares come to life and best practices go to die. The stories you’re about to witness are all true, pieced together from the shattered psyches of those who lived to tell the tales. Accounts have been anonymized to safeguard the unfortunate souls who were caught in the crosshairs of catastrophe.

As you follow this motley crew of unsuspecting engineers navigating the murky waters of automation, integration, and delivery, each alarming anecdote will become another foreboding reminder that it could all happen to you one day.

Prepare for downtime, disasters, and dilemmas. Your pulse will quicken and your hardware will cringe as you travel through the hair-raising vortex of DevOps Nightmares…

Automation is great because it lets you do good things quickly … But you can do terrible things quickly, too.

“It’s been five years and just thinking about it still makes me tired,” says the former VP of Infrastructure at the biometric security company where this DevOps Nightmare took place. “The whole fiasco was exhausting, physically and emotionally.”

At the time, the company was going through a growth spurt like a middle schooler with NBA dreams. Customers were signing up in record numbers to get various body parts scanned into the system, and the servers running the operation were redlining just to keep up.

No one had bothered to build auto-scaling infrastructure because it had always worked fine. But with the new demand it started wheezing like an old man trying to reach a fifth-floor walk-up with seven bags of groceries.

And then the plastic bags broke and servers started rolling down the stairwell. Systems were going down and customers were unable to access their biometric scan services. Let’s just say: the “stuff” was hitting the fan.

The classic scale-out story: build a new cluster and maybe automate it

Under pressure from higher-ups to get things working NOW, the DevOps team was called in to wave their magic wands. They first tried their go-to spell: slap together a new server cluster. But thanks to the platform's complexity and the manual configuration work required, the whole process took five hours plus another hour for the data to trickle in.

Even after all that effort, the new cluster only worked for a couple days, keeling over like a cow tipping contest gone wrong. Back to square one!

“There was a gremlin in the tech stack and we couldn't figure out what it was,” recalls the former Infrastructure VP. “Then someone said they could write the automation to build new clusters and it would ‘only’ take a week.”

The DevOps engineer turned half a day’s cluster provisioning work into a 20-minute automated job. He was treated like someone who stopped a runaway train with his bare hands. While the automation didn’t fix the performance issues entirely, the DevOps team could rapidly scale capacity on demand.

Every silver lining has a cloud (not that kind)

But then, after an all-hands-on-deck week or two of people working long hours and not sleeping much, Fate’s Fat Finger intervened. The engineer who solved the original problem created an even bigger one with a single mistaken keystroke: instead of adding a cluster, he deleted all the existing ones, making the entire platform inoperable. Behold, the incredible power of automation!

The former VP of Infrastructure recounts what happened next: “He came to me whiter than a ghost, trembling as he said, ‘So...I think I just took us offline. Like, completely offline.’ I had to break the news to the CTO and the CEO that we’d be out of order for about 12 hours. As you can imagine, they were not pleased.”

The exec team was emotionally in pieces. The same DevOps engineer who solved the automation problem also brought down the entire company. Heroics one week, company-crushing news the next. Do they promote the guy or fire him?

Despite the tragic accident, in the end most people came to see that automating cluster creation relieved more heartburn and ulcers than it caused. The managers who knew this stuff is difficult got past their disappointment, and the engineers gained a new respect for peer review and the importance of proceeding with caution, especially when exhausted and addled.

The engineer who came up with the solution that saved the company stayed with the company for a few years, but he was never totally forgiven for the mistake he made. Fortunately, he moved on to bigger and better things.

Have your own tales of automation woes or delivery disasters? We want to hear them! If you've endured a devastating DevOps debacle and are willing to anonymously share the cringe-worthy details, please reach out to us at 

Don’t hold back or hide the scars of your most frightening system scares. Together, we can immortalize the valuable lessons within your darkest DevOps hours. Your therapy is our treasured content, and we’ll gracefully craft your organizational mishap into a cautionary case study for the ages. And, in return for your candor, we'll ship some sweet swag your way as thanks.

Imagine, if you will, a series of tragic tales from dark dimensions where nightmares come to life and best practices go to die. 

The stories you’re about to witness are all true, pieced together from the shattered psyches of those who lived to tell the tales. Accounts have been anonymized to safeguard the unfortunate souls who were caught in the crosshairs of catastrophe.

As you follow this motley crew of unsuspecting engineers navigating the murky waters of automation, integration, and delivery, each alarming anecdote will become another foreboding reminder that it could all happen to you one day. 

DevOps Nightmares — Fate’s Fat Finger

After a decade of building and scaling production infrastructure, we’re just getting started

Today is Aptible's 10th birthday. Ten years is a long time to be in business. 

, somewhere around 90% of software startups fail. So why has Aptible been one of the lucky ones?

I believe it’s because we’ve stuck to our mission since the beginning, building and improving a core competency that our clients continue to love and want more of. We’ve delivered real features that so many companies need to solve their cloud infrastructure problems.

 that didn’t exactly turn out the way we expected, we never stopped doing what we do best. And now we’re focused on bringing our successful self-service features to more mature companies with growing engineering teams. 

It’s a tall order, but the solution is buried in our DNA. It’s only a matter of time before it comes out.

Aptible’s journey began in 2013, when HIPAA forced a whole new class of companies to become regulated businesses. We knew that thousands of digital health companies were about to have an instant compliance problem, and we really wanted to solve it for them. 

I was a software engineer who relished the challenge, and my co-founder, 

, had just ended his service in the U.S. Armed Forces as a regulatory lawyer at the Pentagon. While we came at the problem from different angles, we were both passionate about putting the puzzle pieces together. 

We had a very clear mission to simplify HIPAA for developers in healthcare. Developers are at the heart of innovation, and our goal was to keep compliance out of their workflow, so they never had to worry about it.

And we did it: we brought a platform to market that made HIPAA compliance achievable from day one. We were fortunate to cultivate a few instant clients, who we met while renting office space from Blueprint Health, a startup incubator in New York’s SoHo neighborhood.

In our early days, from the fall of 2013 to the spring of 2014, clients such as Hint Health, Aidin, Healthify, and ChartRequest helped us evolve our platform with a viable product. We quickly established Aptible as the go-to platform as a service (PaaS) for digital health startups, and soon began expanding our scope to support HITRUST, SOC 2, ISO 27001, and other compliance frameworks. 

In May 2014, Y Combinator accepted our application and gave us incredible guidance, along with exposure to investors and other clients. A 

 that summer was the match that ignited everything for us. 

From 2014 to 2018, we scaled entirely from inbound calls and referrals. It was a wild ride, and for the first two years it was all we could do to keep up with the growing demand. 

By 2018, things started to stabilize. Customers kept coming and we were very close to profitability. We had some breathing room to start thinking about what we were going to do next. 

Over all these years, we have continuously focused on supporting our customers’ evolving needs. It allowed our engineering to be 100% focused on delivering improvements to our core platform. And it’s paid off:

Considering the odds of failure in the startup world, I think we are in a fantastic place in 2024 and we’re positioned to do even better.

Building the future: How companies will manage cloud infrastructure

My amazing team at Aptible still believes strongly in the strength of the product we’ve been delivering since the spring of 2014. By improving developer self-service for our clients, we’ve helped them master their infrastructure more affordably and gain incredible advantages when it comes to cost and efficiency.

With that initial success and great feedback, we’re now working on delivering what we do for startups to more mature companies. 

To mature organizations, everything built for startups looks like a Fisher Price solution, and no one we know of has been able to build a viable developer self-service platform internally. As companies grow, infrastructure just naturally gets more complex and harder to maintain. It eventually gets to a point where there's a giant mess of infrastructure, with only the remaining “old-timers” truly understanding how and why things are the way they are. 

This is not a new problem. It’s cyclical and feels perpetual. Here’s what it looks like:

And yet, developers want new capabilities that implement the latest best practices. They want new services and automation to make their work less manual. It’s intense! Everyone wants to take away the pain with solutions like internal developer platforms, but it’s an immense undertaking.

A decade ago, we were passionate about helping healthcare startups instantly comply with HIPAA. Today, we bring that same fervent enthusiasm to solving a new problem: helping mature companies across all industries get control over their infrastructure. The answer isn’t “autonomous infrastructure” or AI-driven. It’s all about delivering developer self-service.

The answer has always been in the core of our product: giving users control over infrastructure to standardized golden paths that allow devs to quickly do what they need without compromising on security, compliance, reliability, performance, or cost.

We want to help DevOps teams know everything about their infrastructure and correct any problems they find. Here’s what that looks like:

We’ve laid the groundwork for a new type of platform and are testing it with our customers. We can’t wait to show you what’s coming.

We’re homing in on what it will take to help the vast swath of companies that are “too big for PaaS.” In recent months, we’ve been discussing the issue with companies who’ve come to a point of pain that’s motivating them to find something that works. 

We know that companies are shopping for solutions. Some organizations are combining point solutions like Vercel and Temporal. Others are implementing portal builders like Port or Cortex, or adopting an internal developer platform builder like Humanitec. We’re even seeing PaaS solutions embrace new models for extensibility (like Porter running on k8s in your own AWS account). A lot of investment and activity is taking place, but nothing is solving the problem successfully.

We understand, better than anybody, the infrastructure struggles plaguing large companies’ development teams. For hundreds of companies, Aptible has become the general-purpose platform the whole development team uses. Leveraging that hard-won knowledge, we are wholly dedicated to crafting an elegant solution that simplifies infrastructure management and empowers developers in any industry to remain singly focused on the kind of innovation that drives sustainable growth.

Looking back on the last decade, it’s clear that our journey to becoming the company we were always meant to be is just beginning. Here’s to another 10 exciting years!

10 Years of Aptible—Highlights from a Decade of Growth

Reflections and Lessons Learned from a Decade of Working with Hundreds of Scaling Companies

 by Edward Ocampo-Gooding (@ Shopify) inspired a list of suggested questions for interview candidates looking for their next web dev employer. Shortly after, this list inspired a popular blog post, 

. Questions I'm asking in interviews open sources an extensive list of questions covering topics such as engineering practices, management's style and company culture, drawn from many others in the tech space sharing questions that are important to them throughout their interview process.

At Aptible, we care about hiring the best team members and thus we strongly encourage and make time for Q&A in each step of our hiring process. Moreover, we’re committed to transparency and open sourcing of our culture and hiring process. Below you’ll find our responses to the questions in Julia Evan's blog post - something we'd like to title: 

Looking for your next engineering role? In 2013, a blog post by Edward Ocampo-Gooding (@ Shopify) inspired a list of suggested questions for interview candidates looking for their next web dev employer. Shortly after, this list inspired a popular blog post, Questions I'm asking in interviews by Julia Evans. Questions I'm asking in interviews open sources an extensive list of questions covering topics such as engineering practices, management's style and company culture, drawn from many others in the tech space sharing questions that are important to them throughout their interview process.

Questions You’re Asking in Interviews

At Aptible, we’re creating the No Infrastructure future. If we have our way, every developer will be able to provision and manage scalable and reliable apps and databases without becoming an infrastructure expert, earning a special certification from AWS, or getting their Ph.D. in Kubernetes.

As outdoor enthusiasts, we're also committed to protecting the environment.

This Earth Day, we're combining our mission with our love for nature. For every developer who signs up and deploys an app on Aptible, we'll plant 25 trees 🌳.

Experience Aptible's platform and contribute to a greener world.

Eden Reforestation Projects

 to plant trees from April 21st to May 22nd, 2023. Sign up below with your work email—no credit card required. Let's make an impact, one tree at a time.

BlogCTA

🌳 Plant 25 Trees for Earth Day 2023

In honor of Earth Day, Aptible is planting 25 trees for every developer who signs up and deploys an app on Aptible. Let’s grow a forest together. 🌲🌳🌲

Deploy the Trees 🌲

🌳 Plant 25 Trees for Earth Day 2023

Latest From Our Blog

DevOps Nightmares — Fate’s Fat Finger

Reflections and Lessons Learned from a Decade of Working with Hundreds of Scaling Companies

Questions You’re Asking in Interviews

Build Your Product.
Not Product Infra.

Product

Resources

Support

Company

Deploy the Trees 🌲

🌳 Plant 25 Trees for Earth Day 2023

Latest From Our Blog

DevOps Nightmares — Fate’s Fat Finger

Reflections and Lessons Learned from a Decade of Working with Hundreds of Scaling Companies

Questions You’re Asking in Interviews

Build Your Product.Not Product Infra.

Product

Resources

Support

Company

Build Your Product.
Not Product Infra.