The Aptible Update Webinar Series is a quarterly presentation that covers recent features and changes to the Enclave deployment platform and Gridiron security management products.
We hosted our second Update Webinar on January 25. In it, we covered:
- A preview of Gridiron, our security management and compliance platform
- Improvements to the Enclave deployment process for reliability, predictability and speed
- Enclave database logging and enhanced Redis and RabbitMQ support
- The Enclave CLI for Windows
- Continual efforts to strengthen Enclave security
Chas Ballew: Okay, great. Let’s get started. Thank you everybody for joining. My name’s Chas. I’m the Aptible CEO, one of the co-founders. Really excited to have you with us. This is our second quarterly update. We did our last one in October. This is gonna cover everything we’ve shipped since October essentially. The update is gonna be broken into two pieces. Let’s see if I can advance my slides here. We’re gonna cover a preview of the [00:00:30] new products that we’re launching separately called Gridiron and Skylar Anderson is here to explain that. He’s our Gridiron lead. And then we’re going to cover all of the Enclave features and updates that we’ve shipped since the last one and Thomas Orozco is gonna cover that and he’s our Enclave lead.
After that we’ll take questions. There’s a Q&A feature and, I believe, a chat feature. The Q&A feature works really well, so if you have questions while we go, just drop them in the Q&A and we will address everything. [00:01:00] Either live during the webinar as we go at the end or we’ll follow up with you after this. The webinar is being recorded. We’ll post the video later. We’ll post a link on our blog and send you an e-mail with the link, and the slides too, as we get going. Our CTO, Frank Macreery is here as well and he’ll be moderating questions and things like that.
Again, thank you everybody for joining. Really happy. Our [00:01:30] main goal at Aptible is to help developers, especially cloud-based developers, keep data secure and have good security practices in the cloud. And Gridiron and Enclave are both the main ways that we’re gonna do that. I’m really excited to introduce Skylar here, who’s gonna tell you a bit about Gridiron.
Skylar Anderson: Thanks, Chas. Let’s see. [00:02:00] Okay. Sorry for that brief delay. Yes, I’m Skylar Anderson. I’m the lead frontend engineer here at Aptible. Primarily my focus has been on this latest version of Gridiron. Some of you may already have access to a version of Gridiron, maybe the V1 that used to live on compliance at Aptible.com. We started to roll out, kind of preview some feature releases to select customers [00:02:30] and we’re now opening up to a more general beta release. Today I’m gonna cover some of the philosophy behind Gridiron, who the target market might be, some of the basic features, and then have a moment to answer any questions.
To get started, who is Gridiron for? Gridiron is really for … It’s the first security program management tool dedicated for cloud-first engineering [00:03:00] teams. Some of you with a history of managing compliance might have used other tools that are more enterprise, like on-premise focused risk management tools or those sorts of things. We’re not that. We’re focused on cloud-first companies that need to scale fast, that are used to building their software in the cloud. They might be deployed on our Enclave product, they might be deployed on AWS directly. The SaaS tools and the way their work force and company operates is around the cloud and [00:03:30] they’re comfortable working there and they need to manage their security program accurately to represent cloud-based teams.
What is Gridiron? Gridiron really is a suite of tools to help you manage your security program. It’s a number of small tools that, kind of, together help you build and define your security program. What this does for you is it actually makes the administrative side of protecting data and [00:04:00] having a security program very easy. It’s not Excel spreadsheets. It’s not long questionnaires. It’s very basic questions focused around what we know about deploying an app in the cloud and companies built on the cloud. It helps prep for regulatory audits, so we maintain a mapping between all of our security controls to actual sections of the security role of HIPAA and can help prove compliance.
This also works great for selling into customers. [00:04:30] We provide professional documentation that you can provide directly to your customers as proof of compliance without going through the paperwork every time. Instead you have a central repository to store all this data. One way to think of it is like Gridiron is to data security as QuickBooks is to accounting. You may not know all the tax law and everything behind accounting but you can answer the questions and QuickBooks can guide you to getting to a result. Gridiron is very much the same in that regard.
[00:05:00] What is Gridiron not? It’s definitely not this. Those of you who that have had to respond to an audit from a customer have probably seen a security questionnaire like this. Hundreds of vague questions that you don’t know how to answer. You have to pass the Excel spreadsheet from person to person at your organization, different technical or administrative leads in your company and it’s scattered about many different people, many different documents and you might have to do this every time for every system you’re selling into. [00:05:30] This is not what we want to do. This isn’t at all what Gridiron is about.
Instead, Gridiron is something more like this. This is our philosophy on Gridiron. It’s, rather than like a point in time snapshot of your compliance at any given time, it’s more of a flow. Starting first with your risk analysis. This is the step of setting up your security program, where Gridiron will guide you through analyzing the threats relevant to you and [00:06:00] your company. It takes into account a number of predisposing conditions, namely how you’re deployed, what sort of SaaS tools you’re using, and then analyzes what threats are relevant specific to you and recommends actual security controls that you can help mitigate those.
Coming out of that, we help write your policies and procedures and these are customized to your specific profile based off how you answered some of the questions in your risk assessment phase. We also ensure that everything you’re doing inside your policies and procedures [00:06:30] and the security controls you’re implementing are properly mapped to actual regulations and that you can easily prove compliance in real time at any given time and there’s no more security questionnaires. That we can build those automatically for you.
Out of your policies and procedures, we help drive security and privacy training. This is, again, highly customized to your specific risk profile. This includes things like how you’re deployed, how you’re securing your deployments, if you, for example, require MFA for all accounts, [00:07:00] how that’s configured, even how you handle pull requests in your secure software development lifecycle. We get very specific in the amount of controls we collect and then we’re able to distribute that to your work force in real time and even as it’s changing, if you’re making changes to how you implement certain controls in your policies, we help ensure that you’ve properly distributed that to your workforce.
Coming out of the privacy training is operations. That’s essentially the phase where you’re running your business and we’re informing [00:07:30] how you can do that better. So what security controls you could implement, enforcing periodic access control, our audit log reviews every month. We can help write a policy and actually send notification reminders each month telling you which systems and which aspects of a given system you might need to review. For example, on AWS you might have to review IAM users every month along with bucket access logs and those sorts of things. And [00:08:00] then, generally, operations leads back into the risk analysis, so you’re own assessment in your operation phase can you help you determine new risks or new controls you may need to start planning for.
Off to the side, occasionally out of operations you may have a breach or some incident that you need to respond to and we help you manage those. The work flow of identifying the incident, documenting it, setting up some remediation steps, which could lead into in the risk assessment. [00:08:30] You might need to remediate and fix, actually implement a code change to fix a bug. It may mean retraining a team or some particular members of your team. And then we help you perform a retrospective on that incident and how you can improve over time.
At the core of all this is what we call the Gridiron risk model. This is everything we know about your company. It’s everything Aptible knows about the [00:09:00] industry and it kind of combines them all into a data model that can inform all steps of your security program. At any given time as your company is changing, or as the industry is changing, you’re actually consuming those updates in real time to all of the pieces of your security program. And for all of these we have tooling built out that makes every one of these easier and then they’re all connected through a single data structure that knows everything about your security program and [00:09:30] is able to build dynamic documents, like policies and procedures or specific training programs. And it’s always dynamic and always real time to the settings you have configured in your risk model.
If I expand on that data model just a little bit, we can kind of see the different layers coming out of it. At the base is industry standards. A lot of the things we’re using as a part of your model are not things we made up. They’re coming from different vulnerability or attack databases like CVE or CWE, from meter or the mappings [00:10:00] of different controls to actual regulatory frameworks comes from the CCM, the cloud computing matrix. That’s kind of the root of your data model.
On top of that, we have the Gridiron shared intelligence data. What that is is that’s essentially our expertise and our years of experience working with digital health start-ups. And we’ve sanitized and reduced the vulnerabilities and attacks that are out there to the most relevant to cloud-based engineering teams so you’re [00:10:30] not overloaded with tens of thousands of vulnerabilities that don’t really affect you. Instead, we’ve distilled those down to simple questions like, “Do you deploy on Enclave?” or, “Do you deploy on EC2?”, “Do you store PHI in S3?” With simple questions like that, we can then narrow the scope of your risk assessment to only the most relevant controls.
And that’s leading into the next phase, which is the customer profile. That’s everything we know about you in the systems or SaaS tools that you use to store regulated data. That’s, this is how most [00:11:00] of the customization to your risk profile is surfaced. It’s simple questions like how you’re deployed, how you do pull requests, those sorts of things. And that ultimately builds your own unique risk model, which as I said in the previous slide identifies, is injected into all phases of the compliance model and is used as the seed data for all those different tools so you’re not starting with a blank slate. Instead you’re starting with these layers already presented to you.
[00:11:30] What this looks like? It’s much easier to use than Excel spreadsheet. Here’s an example of our risk assessment dashboard. We’re analyzing risk by tier and this changes over time. Just the most minor changes to your security program can actually make significant changes to your overall risk profile. For example, if you implement MFA for all accounts, that’s a really big security control that actually will affect many vulnerabilities and can greatly reduce your [00:12:00] overall risk profile. So, if you were to change that security control you were see this change quickly. Same with something like background checks. Anything that hardens actual end-user equipment, so like disk level encryption, mobile device management, those sorts of things. Knock to a lot of vulnerabilities for a lot a different systems. And we can help point you in that direction of what security controls are gonna have the biggest win as far as reducing overall risk.
This is another view of the current risk assessment. I’m essentially looking [00:12:30] at threats that are relevant to me. I can see based off likelihood, meaning the chance that it’s gonna happen, the impact, and then the risk level, which is like an overall quantification of the other attributes. I can see other components of my risk model, like vulnerabilities, predisposing admissions, and then ultimately security controls. If I wanted to see, for example, a prioritized list of controls I should be implementing to reduce risk, I can see which vulnerabilities it will affect and know that they’re specific to me based off [00:13:00] my security program and how I’m deployed in my work force and my unique situation.
Here’s our training dashboard. To start we’re rolling out these three basic HIPAA courses: basic, advanced, and security officer. We’re gonna expand on this in the coming months. The key thing is this is actually highly customized training, it’s no longer static. It’s using the data that you’ve provided in your security program and the [00:13:30] policies that we’ve generated to actually customize this training unique to each customer. We’ll expand on the library as time goes on.
Here’s an example of how easy it is to customize. This is very different than … Even the language in the copy here is much easier to digest than what you might have saw in the CAIQ questionnaire, that Excel questionnaire I showed. This is … It’s very simple questions and by answering these, what we’ve done [00:14:00] is actually map- … This particular security control is mapped to a number of vulnerabilities. And, depending on how you answer a control or the questionnaire attached to it, we’re actually mapping different efficacies of that control to different vulnerabilities. You’re actually changing your risk profile just by specifying additional metadata.
For example, in this code reviews … If you don’t do code reviews or if you only do code reviews for major code changes, that has a different risk profile than if you do do them, or you do them for all [00:14:30] changes, for example. If you’re very stringent that might be more effective at certain vulnerabilities than not. This is just an example. There’s hundreds of security controls and we’re importing them all the time and as we’re adding new systems that we support, we’ll be adding the security controls and the related vulnerabilities for you.
Next step is pricing. Gridiron pricing starts at $1,999 a month, paid annually. We’re are offering, [00:15:00] like I said earlier, a beta access for those interested. If you are interested in moving forward or seeing a demo at least, email Shah, he’s our sales guy, he can get you set up with myself or Chas to do the demo and walk you through Gridiron and how it can help improve your security program. That’s it for the preview. I’m open to any questions.
Frank Macreery: Thanks [00:15:30] Skylar. I got a couple questions that were sent by direct message, so, just as a side note, folks can continue doing that or they, there’s a Q&A dedicated box, if you click the Q&A tab. You can also ask your question there and any that come up that you still have. Feel free to post them in there. You can post them at any time and we’ll collect them at the Q&A breaks. The [00:16:00] couple that I got so far, Skylar … So, first, “Will Gridiron help with SOC 2 or PCI compliance?”
Skylar Anderson: It’ll definitely help. The focus for Gridiron right now is on HIPAA specifically. What that really means, though, is that the mappings that we’re storing for different controls are only there for HIPAA. As far as actually setting up a hardened security program and getting everything in one place and having all your training [00:16:30] documentation, risk assessment policy, having those all in one place rather than all over the place, it’ll definitely help in prep for that. And even though automated reporting and professionalism and presenting these documents to an auditor, it’ll absolutely help there. We’re moving more towards opening it up and storing more mappings of controls to different frameworks. Once that happens then we’ll be able to generate audit reports specific to SOC 2 or ISO [00:17:00] and not just HIPAA. That’s in the works.
Frank Macreery: Cool. Thanks. Another question was, “With Gridiron, if we’re using Gridiron, what else do we need?” I guess, what other things do we need to be doing as part of a compliance program?
Chas Ballew: I can help answer-
Skylar Anderson: Chas-
Chas Ballew: Yeah, I can help answer that one. Generally speaking, you’re gonna want an independent third party to [00:17:30] help you with security assessments. That might be something as simple as web application vulnerability scanning up to pen testing or a more formal or extended security assessment. You can get started very easily, but because up until now we’ve hosted all of our customers on Enclave we haven’t been into providing security assessments because we’re not an independent third party. When someone assesses the security of an Enclave customer, [00:18:00] they’re assessing Aptible and [native US? 00:18:01] as well. So for now, you’d need security assessment services.
Frank Macreery: Awesome. Thanks Chas. Well, thank you. Thanks very much, Skylar, for presenting Gridiron. Anyone who has additional questions about Gridiron, we’re gonna have another final Q&A at the end so feel free to continue either DMing me these questions or dropping them in the Q&A box and we’ll collect them [00:18:30] at the end of the webinar. But for now I would like to hand it over to Thomas, who is going to tell you about some of the Enclave features that we’ve developed in the last quarter.
Thomas Orozco: Thanks, Frank. [00:18:46]. All right, so let’s talk about what changed in Enclave this quarter. First off, I wanted to, before jumping into features, [00:19:00] just mention a few guiding principles that have essentially driven the decisions we’ve made, the features we’ve added. The bottom line is we really want Enclave to be the best place for you to launch regulated or sensitive projects. [00:19:15] mostly like HIPAA, compliance projects. Essentially three things you cannot live without, as most of you as customers probably know. You need a robust hosting platform, you need deployment to be reliable. You also need good [00:19:30] options to store your data, so you want some flexibility there. And finally you need uncompromising security all the time. And these are essentially what we focused on probably with greater emphasis last quarter.
Speaking of what’s actually, in terms of features, what’s new. The deployment platform itself changed. We have a new engine that’s running it and I’ll go into, I am going to go into more detail. Obviously that’s why we’re here. But this essentially … We have a new [00:20:00] engine that will make your deployments essentially faster. That also makes your deployments more reliable as well. We’re also about to provide more options and control for your databases. By options, we have things like you can now use Redis for PHI by Redis, by SSL. We also have more control in the sense that you’re now able to use control access databases. We are also providing broader operating systems, [00:20:27] you can access your Aptible resources in the first place. Specifically this is coming by our Windows supports.
Let’s jump in and look at the changes to the deployment platform now. As I mentioned, three main changes here. The first one is the actual deployment engine itself was overhauled. This brought two main core features. One is the ability to systematically roll back if things go wrong at any point. Another one is faster deployments in general. [00:21:00] We also rolled out a new more secure version of our SSH portal. You may not be familiar with what this portal is, but it’s essentially what the command line uses to connect to Aptible when you run things like a database deploy. And finally we are introducing automated orphan container deletion. This is more of a subtle and I could even say obscure feature. For those that have experienced this in the past, it will surely be a welcome one. I’ll explain what, once we get there I’ll [00:21:30] explain exactly what this means.
Let’s start our focus with the engine overhaul. The biggest change here is that we used to have a more monolithic approach to deploying your apps, whereas right now essentially we’re taking your deployment and breaking it into plenty of little steps which we then coordinate to form essentially a chain of dependencies. And if you deployed an app on recently or restarted an app, these are actually steps that you see in your log [00:22:00] output. This is an example of log output of restarting an app and you can see you have starting [00:22:05] up. Well, that’s one step. You can see it’s pretty smooth, when we’re just creating your record in API.
Essentially, your entire deployment is going to be made up of these small steps that we then coordinate. I’m gonna be essentially, for a few minutes now, talking about why that matters and why that changes for you. But, first, if you wanted a more visual approach to this. This is an example coming from an internal [00:22:30] tool we use. This is provisioning a database. As you can see, we have, we created a new volume for a new database and we choose the instance we need. We attach the volume, then we mount it, and essentially have all these little steps as well as the dependencies. We can see from the arrows right there essentially this is what makes up the deployment or the database provisioning itself. Of course, it’s a little more complex than just this. It looks a little more like this. This is what [00:23:00] provisioning a new database looks like.
At this point, the real question is “why is this useful?” How is this gonna serve you as customers? The answer is twofold. First, it’s going to make your deployments safer. Availability to systematically roll back everything. Second, it’s gonna provide faster deployments by running different steps recurrently. One thing I should mention is that when I said deployments, I mean that pretty liberally. [00:23:30] It’s gonna be deploying your app but also restarting it, so provisioning a new database and so on. Really any operation you execute on Aptible.
Looking at these roll backs. The idea really is that now we have an engine where everything, roll backs are built into everything we do. If you look at your deployment and your dependencies [00:23:49], you know, you do A then B then C then D, which is good. Well, the good thing is if you need to roll back at any point, the path to rolling things back is pretty obvious. If you finished, all you have to do is you undo D, then you [00:24:00] undo C, you undo B, you undo A. But if something unexpected happens, like D happens to fail, then in that case as well, the paths to rolling back and getting things back into the right state still is pretty obvious. You just need to undo C, undo B, then undo A.
Now, of course, our real deployments are a little more complex than this. There’s more than just four steps and also most steps have multiple dependencies, but the bottom line is that with this, we’re essentially able to handle [00:24:30] any error that shows up, even if you’ve never seen that error before. Which is pretty valuable when you’re deploying on say AWS or sometimes API’s fail or we may get errors that we’ve never seen before because they haven’t actually existed in the past. But we get new error from an API, well, now we’re able to handle that properly and as best as we can to make sure that your apps stay available no matter what happens.
Now, errors are one case where you want to roll back, but another case, which is [00:25:00] less unfortunate essentially, is you may decide to want to cancel your deployment at some point. You realize that maybe the health check is failing and it’s never gonna succeed. Maybe you’re deploying and then you realize you don’t actually want that to go out. Well, now, as of, again, Q4 of last year, we’re able to actually cancel a deployment that you say, “I don’t actually want this.” Aptible will acknowledge that and will pull it back. Just as if it had had any random failure, essentially.
[00:25:30] Besides that, it’s also helpful for us. It gives us really the ability to troubleshoot failed deployments more easily. This screenshot is coming from the actual tooling we use. Hopefully you don’t actually need to work here to know what’s wrong with this deployment, so let me try reading it right now. When really the viewable view is you have a bunch of stuff that’s green presumably that’s good, and you have a big red dot in the middle. If we zoom in a little bit, obviously this is gonna be the problem. This red dot is what [00:26:00] caused the deployment to fail because a health check failed. Now, the usefulness of this is that for our support team and for us, when you have a problem, we’re able to respond much faster. We don’t need to look at logs, we don’t need to scroll back into the logs to figure out what happens. We just have to open this up and we’ll know exactly what failed and where it failed. And, also in most cases, why it failed as well. That makes our support more efficient.
That’s not all. Besides that efficiency, besides that ability [00:26:30] to really roll things back at any point, making your deployment safer, this new engine’s also faster. Since we have all these little steps, in most cases there’s no reason to do them sequentially. To give you an example, if you have multiple endpoints for an app there really is no reason to put them in one after the other when you’re updating your app. We can just do them parallel. The example I’m seeing here is you have A1 A2, and A3. We need to do all these things. Well, there’s no reason [00:27:00] that they have to happen sequentially since the only actual synchronization point on the actual dependency is B that you’re going to do later anyway. In this case, what we’ll do is we’ll just run A, A1, A2, and A3 in parallel and then B will just happen afterwards, thus making your deployment faster overall.
The impact of this will depend on your deployment, will depend on your app. If your app has a long build process, you’ll see a shorter impact but on restarts [00:27:30] you’ll probably see a large impact. In fact, we have some example apps. Like, actual apps on Aptible that we’re using as examples is what I mean by that. We some cases where restarts used to take 10 minutes because they had a lot of end points and now it takes two minutes. If you’re using that to try and recover from a failure or if you have a hot fix that needs to go to production faster, obviously you just gained eight minutes of availability right there.
The bottom line for you really [00:28:00] is as of, again, that shipped a couple of months ago. But essentially your deploys are faster and safer than they ever were. All support is better equipped than it ever was and, just as important, you don’t actually have to do anything to get this. This all shipped transparently and these features are just available to you now. Now there is one little caveat to this. There always is. These features are only available on the V2 platform. So if you’re on the legacy D1 infrastructure, you will need to upgrade to V2 to get them.
[00:28:30] Now, the good news is, if you’re not sure, you’re probably already on V2. V2 has been the default for over a year now. It’s started being the default in November 2015, so unless you notice your on V1 you’re probably on V2. But if you do want to check, just reach out to support. Otherwise, in most cases, we probably will have reached out to you already if you’re on V1 to schedule your upgrade.
Do we have … Frank, do we have any questions on that?
Frank Macreery: We did. [00:29:00] Thanks, Thomas. There’s just one question. The question is, “My deploys are still taking 15 minutes or more. Do you have recommendations for how I can make these faster?” I know you talked about some of the ways that the engine improvements will make deployments faster across the board, but for customers who are still seeing slow deployments, do you have any suggestions?
Thomas Orozco: Yeah, that’s a good question. Essentially, where you put an app, [00:29:30] most of your time ends up … There’s really two phases. The first one is building an image for your app and the second one is essentially taking that image and deploying it across our infrastructure. The second part is the one we optimized. This one, at this point, should usually take a maximum of about 10 minutes. Because everything happens in parallel, there really isn’t anything that’s gonna have, that’s gonna grow over time depending on how big your app is essentially. Even if you have 20 endpoints and 100 containers, even in extreme cases like this, it shouldn’t take much [00:30:00] longer than the most complicated apps to deploy.
However, the part that is gonna change quite a bit is when you build the image. The initial docker build phase, the one where you actually do get to see the docker build output. This particular phase, this one really depends on your app. It depends on what you’re doing in the build process. It depends on the state of caches and so on. There’s two things we can do there. The first one is you can, if you end review your build process and see if there are ways you can make it faster [00:30:30] just like review docker best practices and see if there are improvements you can make.
The second one is you can build somewhere else. Just build off of Aptible, possibly getting a bigger instance and then just use a private redistribute deployers, like deploying for a private registry. We touched upon that, we presented a feature on the last webinar so you can review there. This will let you build somewhere else and just push that to Aptible essentially, which will make your deployment much faster because you don’t actually have to do the build when you do that. [00:31:00] That being said, this is something that’s on our map to try improve, like in general. We’d like to make deployments even faster. We’ve actually had you having anything to do, so there might be changes coming there as well. Was there anything else, Frank?
Frank Macreery: That’s the only question for right now, but if anyone has additional questions, again, just drop them in the Q&A.
Thomas Orozco: Great. Thanks. The next main change that has happened on the deployment platform is a new SSH [00:31:30] portal. The SSH portal is what you use when you run things like Aptible SSH, Aptible db:tunnel, even Aptible logs. Essentially it’s like, it kind of punches a hole into your stack for you. It just runs on your dedicated stack. That being said, it’s obvious it’s a pretty sensitive piece of our infrastructure because it’s receiving requests from the entire internet and it’s accessing your stack. What we did is we essentially beefed up security. We used to require just an Aptible access token [00:32:00] to access. Now it also requires a temporary SSH key. That key itself is valid for 15 minutes and it’s booked with single operation as well as with single user. This is going to ask like a really good picture of exactly what’s happening when someone connects to their portal.
And logs, you’ll see below, these are logs from that portal and there’s a piece of the middle line that says “SshPortalConnection dash” and then there’s a new UUID after that. This uniquely identifies the credentials that were used to log in. And using [00:32:30] this we’re able to say exactly essentially, “This is Thomas and he’s connecting to a db:tunnel.” And not only can we say what it’s planning to do, it’s also the only thing we can do when we connect. Right there, we … Essentially we do two things. One is we have some essentially defensive data. We used to have one layer of protection, now we have two. So it’s at least, I don’t know, two times better? We also have strong auditing capabilities which you can get access to as well if you have a need to understand what happened, if [00:33:00] you have a requirement to auditing perspective for example, we can help you with those logs that we have.
That being said, I should mention, you will need an up-to-date CLI to use this, so you’ll need 0.8 or greater. Since all old CLI is welcome to you you connect to the older SSH portal. Now obviously the goal of this changes to reduce our net surface, so these old SSH portals will eventually be going away. If you need … Again, if you’re using old CLI it will eventually [00:33:30] stop working, so make sure where you can do that at Aptible.com/support/toolbelt and you’ll get the newest version which right now is 0.8.4.
All right. That takes us to the last feature of the deployment platform. I checked. It’s a feature called orphan container deletion. Some of you may have had that in the past. Orphan container is essentially … Here’s how they’re created. Sometimes you try to deploy your app and you have old containers for your app. [00:34:00] But sometimes these are running on the instance that’s not responding. So maybe deploy your app, you can’t shut them off because the instance is now responding, maybe because the network is down or something like this.
In that case we have to chose, do we stop the deployment because of that or do we just move on and leave those behind? In practice, what we do is we move on and leave those behind because you don’t want to hold up your deployment due to an unresponsive instance because that would destroy your ability to even fix the problem in the first place. We leave them behind as orphan containers. Now the problem [00:34:30] is, these do two things. First one, they waste some system resources. They’ll use some RAM. They’ll use some CPU when they shouldn’t even be here. Eventually, eventually they’re using resources and wasting them.
More problematically, sometimes they can break background processes. If you’re using Sidekiq or Celery and you push a job to your background workers and an orphaned background worker picks it up, it’s going to be an old version of your code. In some cases, if you’re lucky it doesn’t work, but that code [00:35:00] hasn’t changed. But if you’re unlucky, that code has changed, maybe the job itself is new. In that case, it’s just gonna fail. Otherwise there might have been a perfectly good new worker right next to it ready to pick up that job but the orphan got to it first. The good news is here, we now can do that for you so you don’t have to do anything. It just works and those containers will not bother you again in the future.
Frank Macreery: There’s no [00:35:30] questions on that section, actually. Someone just asked, “I was thinking is there a way to get SSH logs and other access to the” … Basically asking if there’s any way to get Aptible SSH logs, I believe.
Thomas Orozco: Thank you. Oh yes, yeah. Aptible SSH logs, not currently. No. Logs like back Aptible SSH logs, I mean logs from an SSH session. It’s currently not possible. It’s a future we’ve [00:36:00] considered. We don’t have any firm plans to put them in there right now, but it’s something we’ve heard several times in the past. We’ll definitely be considering it. A few reasons not to do it about like apps with PHI a lot more likely than just regular app logs because usually when you get in there’s a problem to fix and usually you’re trying to fix it now and you honestly don’t have as much time to think through, you know, what’s going to print out. Another reason it may be in some cases [00:36:30] that, you know, humongous amounts of log, our logs if you like logging a lot. For now, logs from Aptible SSH are not available. They may eventually become but they’re not on the ledgers yet.
If you’re asking about the logs from the SSH daemon itself, I mean the logs I showed before. These aren’t available either at this moment. Mainly because especially in shared stacks, it’s not just yours. These are more useful from an auditing perspective and we … If you’re under an audit and you need access to these, we’ll be happy to help.
Frank Macreery: [00:37:00] Awesome. Thanks very much Thomas.
Thomas Orozco: Thanks.
Frank Macreery: Oh, another, I guess, important questions. When are older version of the CLI going to be deprecated?
Thomas Orozco: That’s a good question. That’s probably going to be staged deprecation for the most part. It’s gonna be because we have multiple stacks on Aptible, so they’re not all gonna be deprecated at the same time. But on shared stacks, deprecation’s probably gonna start in about a month, so I guess of [00:37:30] March. It will stop working in shared department stacks and gradually stop working everywhere else. You will start receiving warning in your CLI pretty soon if you connect using an older CLI to a new SSH portal.
Frank Macreery: Great. Thanks, Thomas.
Thomas Orozco: Thanks. All right. Moving on to changes to databases. Yeah, our goal is really give you more options for databases, and [00:38:00] more control. The goal here is to let you audit you have database logs that you audit what your database is doing. We have SSL supporting Redis that you can use it for PHI. And, finally, we have a RabbitMQ management interface to give you control over tasks that exist in that RabbitMQ.
Database logs, which we released several weeks ago at this point I think. Really the work is really just like for apps. You’ll just set up a log drain and it will start receiving [00:38:30] logs for databases. You can also use aptible logs to review them. But, here again, you’ll need to make sure you have a recent CLI. That’s another reason to upgrade, getting access to your logs. That being said, one thing to keep in mind for these is that these logs may not always be what you expect them to be. We do have customers that want access to all their queries by default and log everything on the database. Other customers don’t want that, just want logs flow queries and some others. So a default configuration is logs flow silent, but [00:39:00] if you’d like to make changes you can do that.
You can usually do that directly by your databases interface. Using [Postgres for example, you can just do psql session. If you have, for this, if you have any questions on how to vet things up, if you’d like any guidance, we are, of course, happy to help. You can always reach out to us. One case where you will need to reach out to us is for MySQL. [00:39:30] If you database was launched prior to last week, it will not be able to log. Yeah, it will not be able to log your queries or the general query log, essentially or the slow queries. In this case, let us know if you like this feature available and it will just reload the database for you. And this will take about 30 seconds. The database will be unavailable during these 30 seconds so we’ll be happy to scale that off hours for you so that you don’t have to, so [00:40:00] that you don’t have any down time that happens.
Frank, do we have any questions on the-
Frank Macreery: Yeah, do we, does Aptible have any suggestions for ensuring that PHI is not included in database logs?
Thomas Orozco: Yeah. Did you want to answer that one?
Frank Macreery: Sure. Yeah. Yeah. I can grab that.
Thomas Orozco: Thanks.
Frank Macreery: There’s a couple ways that we would answer this. [00:40:30] The first is that we would generally recommend that database logs are probably better suited performance debugging than audit logging. It probably makes sense to, for example, enable slow query logging and use that in your database logs. But it makes not as much sense to use database logging as the sole way for logging [00:41:00] access to PHI and I’ll go more on that in a sec. I guess before that, I will note that log drains can be defined on a per drain basis to apply to either apps or database containers or both.
What you could do is, let’s say that you really like Papertrail or something or accessing your logs but Papertrail doesn’t sign a BAA, so you can’t store PHI there, [00:41:30] you can always set up Papertrail as a destination for your app logs, so all of your app logs go there and you can control and prevent PHI from being logged at your apps. And then you can use a separate internal drain, say an elastic search drain, that’s specific for your database logs. You can mix and match these and you can have a PHI only log drain for your database logs.
Back to the point of [00:42:00] why you might want to implement audit logging at the app level. One big consideration is that at the app level you always know the individual identified user who is accessing PHI and that’s an important component of any PHI audit log and when you’re logging at the database level, often your application will be connecting to the database with a single user, single database user for the whole application, and so you lose that identification of which user was accessing or modifying [00:42:30] PHI. That’s kinda why app audit logs are really necessary in addition to whatever you do at the database level. And why we generally think that doing stuff like slow query logging is a lot more applicable for databases.
Thomas Orozco: Thanks, Frank. I did suspect you were getting this question pretty often in support, so I figured you might have a better answer than I do. Thanks. Do we have anything else?
Frank Macreery: [00:43:00] Not for, not for right now.
Thomas Orozco: All right, so, we go on to … Yeah?
Frank Macreery: We had someone follow up to that one. And some of you asked, or suggested that it may be possible to inject comments into SQL too, include additional info there. Just bringing that up.
Thomas Orozco: That’s a good tip. All right. Getting on with our database [00:43:30] changes. The next big one is support for SSL on Redis databases. Until now on Aptible, Redis databases only supported plain text, which is critical but now if you launch a new Redis database Aptible, you will also get an SSL endpoint. Using SSL means that all your data is encrypted, like all future database and that means you can use it to store PHI. You’ll find those credentials in your dashboard, so if you provision a Redis database, you can just go to your credentials tab and you’ll just find [00:44:00] them right there. One thing to mention is SSL will only be available out of the box for recent Redis instances, so IDs that were launched before, after last week essentially. If you don’t actually see an SSL credential in there, it’s probably because your databases are a little too old and doesn’t actually support that. Just like MySQL, we can enable that and we’ve already done it for several customers so we can do that on demand. Just reach out to us and SSL will be enabled.
[00:44:30] Now, the question you’re probably wondering is, “Well, SSL isn’t actually a feature of Redis so how’s my client going to work with this?” and the good news is that even though SSL is not a feature of Redis itself, most Redis clients do actually support it out of the box. We’ll provide you with a URL that’s stopped with Redis S, so there’ll be two Ss right there and most clients will actually use that out of the box and use that to mean, “Oh, I have to connect to SSL,” [00:45:00] and it will just work for you. If ever you … Of course if you, if for some reason your client isn’t able to connect or doesn’t recognize that URL, make sure you check your documentation or some- … The vast majority of mainstream clients this will actually work out of the box for you.
By the way, you can also use this with the CLI itself. Some of you are probably using the CLI to tunnel to the Redis database, you can also do that here. You can use db:tunnel [00:45:30] and then pass a type argument to expand the type of connection you want. In this case, you can say you want an SSL connection. I should mention, that’s an important thing to keep in mind. The traffic from your work station to Aptible is always encrypted, no matter what. Even if you don’t use the SSI port. Yes, the traffic from the SSH portal to your Redis instance, that becomes encrypted if you use SSL endpoint, which, again, is a good idea to, is a good idea [00:46:00] if you are running a Redis database with PHI in it. That being said, all this requires, again, a recent CLI. This one is 0.8.4. If you get the latest CLI from Aptible.com/support/toolbelt, we’ve shown that thing before, that’s what you’ll get and this will support tunneling.
It’s not just used for Redis, as you’ll find out. It’s also going to be used for RabbitMQ. RabbitMQ, we did the same thing with RabbitMQ. We have a new [00:46:30] port there to match our interface. Using RabbitMQ, you can use dash to manage your queues. If you’re not using RabbitMQ, this obviously will not be as interesting. Essentially the same caveats as Redis apply. If you launch this database, if you did not launch it recently then you will just have to reach out to us to enable the management interface. But once you’ve done so, you’ll able to use the Aptible CLI to access the layers for RabbitMQ management interface and get access to your queues. [00:47:00] Did we have any … I’m afraid we’re running a little behind, but I think we might have …
Frank Macreery: Yeah, one viewer asked what the date was when database logs were implemented and if it was the same as when we shipped Redis with SSL and I can answer this. I’m on the blog, in case that helps, Thomas. I believe-
Thomas Orozco: One thing I’d like to mention first is that database logs do not require database [00:47:30] restart. Regardless of how old your database is, you can just launch. Except if it’s MySQL, which doesn’t launch anything. For other databases, you can just start using this right now. As far as the actual release dates, Frank, I’ll let you get that.
Frank Macreery: Yeah, database logs have been available since November 30 of 2016 and for MySQL, I believe you had the date in your slides.
Thomas Orozco: Yeah, January 19. Everything about, [00:48:00] yeah, Redis SSL, MySQL, RabbitMQ, all of that shipped last week on the 19th.
Frank Macreery: Right, so if you, if you do have a MySQL database and you want to enable general logging or slow query logging and you haven’t provisioned that database in the last week, just contact us.
Thomas Orozco: All right. To wrap this up, we have the Windows CLI. This one is pretty straightforward. The CLI used to only be available, [00:48:30] at least officially, we have some customers that were able to work around this, but the CLI was only available on Unix or on Linux and MacOS essentially, but that’s not the case anymore. Now you can, if you’re using Windows, you can download the CLI and it will just work. You will, however, need a reasonably recent version of Windows, so the requirements are right there on the slide. But you’ll need Windows 8.1 for the desktop side or 2012 on the server side. In both cases, you’ll need a 64 bits OS.
[00:49:00] With that we essentially have support for all mainstream OS’s, so that’s OSX, Windows, Ubuntu, Debian, Red Hat, and also CentOS. And that’s also part of what we call the Aptible Toolbelt. At this point you’re probably wondering when I’m telling you to download the Toolbelt to embed your CLI. One thing I should mention is the Toolbelt is essentially a package that contains the Aptible CLI and a bunch of other stuff you need to run said CLI. An example of other stuff you [00:49:30] need is the actual, you’ll need the actual Ruby interpreter because the CLI is written in Ruby. You’ll need a recent version of SSH, so all of this comes out of CLI so that you have environment that’s essentially going to work. It’s also a little faster than installing it as a gem, which was the original way to install it. That being said, if you absolutely want to, it’s of course possible to not use the Toolbelt. But a Toolbelt makes it essentially easier to get started in running [00:50:00] the CLI.
That’s it for, well Enclave Q&A because, but that’s because I wrote it. But if we had any questions on the Gridiron as well, I think we can probably answer it as well.
Frank Macreery: Awesome. Thanks very much, Thomas. Yeah, if you do have any additional questions, we have a few minutes here at the end to answer them. Just dump them in the Q&A or chat or DM me. [00:50:30] The first question we got, this one’s gonna be for Skylar and Chas. “How does Gridiron training work?” Can you tell us a little bit more about, you know, the training product and how, what’s involved there.
Skylar Anderson: Yeah, I can grab that Frank. Basically any new user that joins your Aptible organization is going to be assigned basic training. You can turn that off for a given user, so if you have a robot user that does your deployments you can remove their [00:51:00] training assignments, you get all the green check marks. That’s for basic training. The other advanced training programs, so for developer training or security officer training, right now you have to go to that training dashboard, the screenshot that I showed, and find the user and click assign next to that training program. We’re still in the process of rolling out being able to access that content on the dashboard itself. Basic training is on the dashboard. The advanced programs are still handled by a webinar session [00:51:30] with Chas directly, but once that content is published those will all be completed from dashboard.
Frank Macreery: Awesome. Thanks, Skylar.
Skylar Anderson: Mm-hmm (affirmative).
Frank Macreery: One additional, this one’s gonna be for Thomas on Enclave. What is required to view the RabbitMQ, the HTTP management dashboard, so viewing the queues. [00:52:00] Can you just recap those steps?
Thomas Orozco: Yep. Sure. First, you’ll need a recently launched RabbitMQ database. You can see if you launched recently enough by just go to your RabbitMQ database in the Aptible dashboard and just see if this is providing essentially see if it’s providing a URL code management. If you have it, you’re all set. If not, then you’ll need just reach out [00:52:30] to us and just ask us to enable the management interface for you. Once you’ve done that, then you just need a recent CLI and you just have to run this one command, which is db:tunnel and the name of your database and then type. That flag is important. That is just type management. It tells the CLI to connect to the new management interface.
By the way, if you don’t actually have a recent enough RabbitMQ database, it wasn’t related, you’ll be, at this point, it will error out and tell you about it. But if it doesn’t error out, at this point, you will, [00:53:00] essentially you’ll be all set. Your database will come up and then you’ll just be able to access it, the URL that’s provided by the database, by the Aptible db:tunnel command as it comes up. But, again, for this to work you will also need a recent CLI. Two things you gotta update, your RabbitMQ database and your CLI.
Frank Macreery: Awesome. Thanks, Thomas. Then we have a question for Skylar [00:53:30] and Chas about Gridiron. What infrastructure, as the service providers, can we use Gridiron with?
Skylar Anderson: Right now we’re shipping with Enclave and an AWS. There’s not so much a technical restriction to shipping to new providers, it’s more us taking the time to establish a baseline for a given provider. Gridiron isn’t an open-ended [00:54:00] tool where you’re necessarily adding everything yourself. We inform a lot of the basic components for a given platform. Traditionally that was just Enclave because we’re obviously the experts there. We’re moving into AWS with this beta launch of Gridiron and then as we move forward we’ll start to add other providers as, once we start to see a need for that and we’ve taken the time to build the baseline for those providers.
Frank Macreery: Awesome. Thank you again, [00:54:30] Skylar. That’s it for the questions that have been asked in Q&A here. If anyone does have additional questions, feel free to reach out to us. You can just go to Aptible.com/support and you can find a link there to contact us. You can also just e-mail support at Aptible.com and we’re happy to answer any questions that you think of after the fact offline. I’ll just hand it back to [00:55:00] Chas for some closing words here.
Chas Ballew: Thanks everybody. I appreciate you. Appreciate the questions. We will have the slides and video up. We’ll put a link to them in the blog. You’ll receive a follow-up e-mail with those materials and then if you want to register for the next one, it’ll be on April 25. Same time. And we’ll see you then. Thanks for joining. We appreciate it. Bye.
Frank Macreery: Thanks guys.
Defense in Brief
Sign up to get the best in security and compliance delivered monthly.
From the Blog
Meltdown and Spectre are Critical Vulnerabilities for Cloud Infrastructure. Here’s How the Aptible Security Team Responded
Meltdown is a critically important vulnerability for any PaaS provider. This post details why that's the case, how Aptible responded to Meltdown and, more broadly, how Enclave is architected to protect against these sort of vulnerabilities by default.Read more
Recap: Aptible October 2017 Quarterly Product Update Webinar
Quick recap of Aptible's October quarterly product update and all that's new with Enclave and Gridiron.Read more