Blog

We’re proud to announce our latest open-source project: Supercronic. Supercronic is a cron implementation designed with containers in mind.

Why a new cron for containers?

We’ve helped hundreds of Enclave customers roll out scheduled tasks in containerized environments. Along the way, we identified a number of recurring issues using traditional cron implementations such as Vixie cron or dcron in containers:

• They purge the environment before running jobs. As a result, jobs fail, because all their configuration was provided in environment variables.

• They redirect all output to log files, email or /dev/null. As a result, job logs are lost, because the user expected those logs to be routed to stdout / stderr.

• They don’t log anything when jobs fail (or start). As a result, missing jobs and failures go completely unnoticed.

To be fair, there are very good architectural and security reasons traditional cron implementations behave the way they do. The only problem is: they’re not applicable to containerized environments.

Now, all these problems can be worked around, and historically, that is what we’ve suggested:

• You can persist environment variables to a file before starting cron, and read them back when running jobs.

• You can run tail in the background to capture logs from files and route them to stdout.

• You can wrap jobs with some form of logging to capture exit codes.

But wouldn’t it better if workarounds simply weren’t necessary? We certainly think so!

Enter Supercronic

Supercronic is a cron implementation designed for the container age.

Unlike traditional cron implementations, it leaves your environment variables alone, and logs everything to stdout / stderr. It’ll also warn you when your jobs fail or take too long to run.

Perhaps just as importantly, Supercronic is designed with compatibility in mind. If you’re currently using “cron + workarounds” in a container, Supercronic should be a drop-in replacement:

1
2
3
4
5
6
7
8
9
10
11
$ cat ./my-crontab
*/5 * * * * * * echo "hello from Supercronic"

$ ./supercronic ./my-crontab
INFO[2017-07-10T19:40:44+02:00] read crontab: ./my-crontab
INFO[2017-07-10T19:40:50+02:00] starting                                      iteration=0 job.command="echo "hello from Supercronic"" job.position=0 job.schedule="*/5 * * * * * *"
INFO[2017-07-10T19:40:50+02:00] hello from Supercronic                        channel=stdout iteration=0 job.command="echo "hello from Supercronic"" job.position=0 job.schedule="*/5 * * * * * *"
INFO[2017-07-10T19:40:50+02:00] job succeeded                                 iteration=0 job.command="echo "hello from Supercronic"" job.position=0 job.schedule="*/5 * * * * * *"
INFO[2017-07-10T19:40:55+02:00] starting                                      iteration=1 job.command="echo "hello from Supercronic"" job.position=0 job.schedule="*/5 * * * * * *"
INFO[2017-07-10T19:40:55+02:00] hello from Supercronic                        channel=stdout iteration=1 job.command="echo "hello from Supercronic"" job.position=0 job.schedule="*/5 * * * * * *"
INFO[2017-07-10T19:40:55+02:00] job succeeded                                 iteration=1 job.command="echo "hello from Supercronic"" job.position=0 job.schedule="*/5 * * * * * *"

What’s next?

If you’re an Enclave customer, we’ve updated our cron jobs tutorial with instructions to use Supercronic. If you’re not using Enclave, then head on over to Supercronic’s GitHub page for installation and usage instructions.

In a world where application dependency graphs are deeper than ever, secure engineering means more than securing your own software: tracking vulnerabilities in your dependencies is just as important.

We’ve greatly simplified this process for Enclave users with a recent feature release: Docker Image Security Scans. This is a good opportunity to take a step back, review motivations and strategies for vulnerability management, and explain how this new feature fits in.

Why Dependency Vulnerability Management?

Popular dependencies are very juicy targets for malicious actors: a single vulnerability in a project like Rails can potentially affect thousands of apps, so attackers are likely to invest their resources in uncovering and automatically exploiting those.

One infamous (albeit old) example of this is CVE-2013-0156: an unauthenticated remote-code-execution (RCE) vulnerability in Rails that’s trivial to automatically scan for and exploit. Among others, Metasploit provides modules to automatically identify and exploit it.

As an attacker, a vulnerability like CVE-2013-0156 is a gold mine. The exploit can be delivered via a simple HTTP request, so all an an attacker needed to do to compromise vulnerable Rails applications was send that request to as many public web servers as they could find (finding all of them is much easier than it sounds).

In other words: when it comes to vulnerabilities in third-party code, you’re actively being targeted right now, even if no one has ever heard of you or your business.

Strategies for Dependency Vulnerability Management

Now that we’ve established that vulnerability management matters, the question that remains is: what can you do?

Modern apps depend on a number of dependencies that come from diverse sources ranging from OS packages to vendored dependencies. Fundamentally, there’s no one-size-fits-all approach to track of vulnerabilities that affect them.

So let’s divide and conquer: from a vulnerability management perspective, there’s a useful dichotomy between two categories of third-party software.

• On the one hand, there’s, third-party software you installed via a package manager

• And on the other hand, there’s third-party software you didn’t install via a package manager.

The easiest dependencies to look after are those that you installed via a package manager, so let’s start with them.

Using a package manager? Leverage vulnerability databases

Package managers helpfully maintain a list of the packages you installed, which means you can easily compare the software you installed against a vulnerability database, and get a list of packages you need to update and unfixed vulnerabilities you need to mitigate.

Ideally, you want to automate this process in order to be notified about new vulnerabilities when they come out, as opposed to hearing about them when you remember to check. Indeed, remember that when it comes to vulnerable third party dependencies, you’re actively being targeted right now, so speed is of the essence.

How does this work?

There’s a number of open-source projects and commercial products you can use for this type of analysis. A few popular options are Appcanary (which Aptible uses and integrates with), Gemnasium, and Snyk.

They often work like this:

• You extract the list of packages you installed from your package manager

• You feed it to the analyzer

• The analyzer tells you about vulnerabilities (commercial products will also often notify you when new vulnerabilities come up in the future)

That simple!? Almost: you’re probably using multiple package managers in your app, which means you may have to mix and match analyzers to cover everything. Indeed, for most modern apps, you’ll have at least two package managers:

• A system-level package manager: if you’re using Ubuntu or Debian, this is dpkg, which you access via apt-get. If you’re using CentOS / Fedora, this is rpm, which you access via yum or dnf. If you’re using Alpine, it’s apk. Etc.

• An app-level package manager: if you’re writing a Ruby app, this is Bundler. If you’re writing a Node app, it’s NPM or Yarn. Etc.

So, what you need to do here is locate the list of installed packages for each of those, and submit it to a compatible vulnerability analyzer.

New Enclave Feature: Docker Image Security Scans

Now’s the right time to tell you about this new Enclave feature I mentioned earlier in this post.

When you deploy your app on Enclave, we have access to its system image. Last week, we shipped a new feature that lets us extract the list of system packages installed in your app, and submit it to Appcanary for a security scan.

This can work in two different ways:

• You can run a one-shot scan via the Enclave Dashboard. This gives you an idea of what you need to fix right now, but it will not notify you when new vulnerabilities are discovered in packages you use, or if you install a new vulnerable package.

• You can sign up for Appcanary and connect Enclave and Appcanary accounts. Enclave will keep your Appcanary monitors in sync with your app deploys in Enclave, and in turn Appcanary will notify you whenever there’s a vulnerability you need to know about. This puts you in a great position from a security perspective, and will reassure security auditors.

To summarize: Enclave with Appcanary can now handle vulnerability monitoring for your system packages, and it’s really easy for you to set up!

However, for app-level packages, you still have to do a little bit of legwork to find and integrate a vulnerability monitoring tool that works with your app. Note that Appcanary does support scanning Ruby and PHP dependencies, so you might be able to use them for app-level scanning too.

Is that it?

Not quite: we still have to look at third-party code you didn’t install via a package manager. Here are a few examples: software you compiled from source, binaries you downloaded directly from a vendor or project website, and even vendored dependencies.

For these, there is — unfortunately! — no silver bullet. Here’s what we recommend:

• When possible, try and minimize the amount of software you install this way.

• When you absolutely need to install software this way, subscribe to said software’s announcement channels to ensure you’re notified about new vulnerabilities. This may be a mailing list, a blog, or perhaps a GitHub issue tracker. When possible, review how security issues were handled in the past.

This time, that about wraps it up! Or does it? Engineering is turtles all the way down, so even if you covered all your bases in terms of software you installed, there’s still the underlying platform to account for.

That being said, unless you’re hosted on bare metal on your own hardware, this is largely out of your control. At this point, your best strategy is to choose a deployment platform you can trust (if you read this far, hopefully you’ll consider Enclave to be one).

Over the last quarter, we released a number of new features and updates for the Enclave deployment platform. We also began helping customers deployed on AWS to manage their organization’s security and compliance using Gridiron.

Yesterday, on a brief webinar, our team reviewed the updates to the Enclave platform and showed how Gridiron helps software developers build and maintain strong security management programs.

In case you missed it, you can download the slide deck and get the transcript in our resources section, or watch the full event below. We also provide a quick recap in this blog post.

New for Enclave

We intend for Enclave to be the best platform for developers to deploy regulated and sensitive software products. This quarter, we focused on improving Enclave in three ways: security and compliance, database self-service, and general usability improvements.

Security and Compliance

We launched new ways to secure apps and meet compliance goals while improving the security of Enclave itself.

We’ve previously detailed these improvements on our blog. Here’s the list:

• IP Filtering

• Managed HTTPS for Internal Endpoints

• Customizable HTTPS Protocols for ALB Endpoints

• Default AES-256 Database Disk Encryption

• U2F Authentication

Database Self-Service

Self-serve database scaling is coming soon. The Aptible CLI now supports aptible db:reload, disk resizes are a lot faster, and we will launch self-service database scaling soon.

Usability Improvements

We launched a few small improvements that should make developers’ lives easier when deploying with Enclave:

• We now protect against runaway SSH sessions when your session gets disconnected

• Memory management restarts apps in pristine containers when they exceed memory limits

• Enclave Log Drains now integrate with Sumo Logic and Logentries as an alternative to rolling your own ELK stacks

Gridiron

Gridiron is our suite of tools that helps developers build and maintain strong security management programs. Gridiron makes the administrative side of protecting data easy and helps to prepare you for regulatory audits as well as customer security reviews.

In the webinar, we gave a short talk-through of how Gridiron approaches security management. This starts with the Gridiron Data Model: an API that integrates data from your business, our experience working with hundreds of customers in securing sensitive data, and industry-wide security standards provided through NIST Guidance, vulnerability and attack databases and shared intel.

Gridiron ingests data about your business through a series of straightforward and relevant questions that are easy to answer but have important implications for your internal security program.

Gridiron uses that data to create deliverables that help you show security and compliance as well as improve your business operations.

Getting started with Gridiron

If you’d like to improve your organization’s security and compliance and simplify the process for working through customer security reviews and regulatory audits, please get in touch. For a limited time we’re offering early access pricing for customers who have deployed on AWS.

Register Now for July 2017 Aptible Product Update Webinar

Our next product update webinar will be hosted on July 25, 2017 at 11am Pacific / 2pm Eastern.

Please register now.

All registrants will receive a webinar recap and the recording shortly after the conclusion of the webinar.

Are Aptible customers affected by Cloudbleed?

No, not by virtue of using Aptible. Aptible does not use Cloudflare, and as such, our services and customer environments were not affected by the Cloudbleed vulnerability disclosed yesterday.

That said, if you use or used Cloudflare, you may be affected. You can read Cloudflare’s official description of Cloudbleed here.

If I used Cloudflare to cache PHI, what should I do?

Activate your incident response plan and talk to your lawyer immediately, unfortunately. You may be required to conduct mitigation, and breach and/or security incident notifications, by HIPAA or your business associate contracts.

Cloudbleed is one issue. Another issue is that if you were using Cloudflare to cache PHI though their CDN without a BAA, you may have been in breach of the HIPAA rules before this.

Some have suggested that Cloudflare might not be a HIPAA business associate because of an exception to the definition of business associate known as the “conduit” exception. Cloudflare is almost certainly not a conduit. HHS’s recent guidance on cloud computing takes a very narrow view:

The conduit exception applies where the only services provided to a covered entity or business associate customer are for transmission of ePHI that do not involve any storage of the information other than on a temporary basis incident to the transmission service.

OCR hasn’t clarified what “temporary” means or whether a CDN would qualify, but again, almost certainly not, as data storage is a critical, non-incidental component of CDN functionality.

What if I used Cloudflare to cache PII?

Again, activate your incident response plan and talk to your lawyer. HIPAA is just one of many data privacy regulations. Many states require companies to report breaches of personally identifiable information belonging to residents of that state.

What if I used Cloudflare for data aside from PHI or PII?

We encourage you to be safe and rotate all credentials that might have passed through Cloudflare from your app, such as session cookies, API keys, and user passwords.

What else should I do?

We encourage you to rotate your passwords for any service that used Cloudflare between September 22, 2016, and February 18, 2017. Cloudflare has not released a list of services affected. You can find one security researcher’s list of Cloudflare DNS customers (which is likely overinclusive) here.