Operations

On Enclave, all changes to your resources are performed through Operations.

Tip

You can find a list of all the Operations in a given Environment in Activity Reports.

What happens when things go wrong?

Reliability is a top priority at Aptible in general and for Enclave in particular. That said, occasional failures during Operations are inevitable, and may be caused by:

  • Failing third-party services: we strive to minimize dependencies on the critical path to deploying an App or restarting a Database, but Enclave nonetheless depends on an number of third party services. Notably, Enclave depends on AWS EC2, AWS S3, AWS ELB, the Docker Hub (with a failover or Quay.io and vice-versa). These can occasionally fail, and when they do, they may cause Enclave operations to fail.
  • Crashing instances: ultimately, Enclave is built on a fleet of Linux instances running Docker. Like any other software, Linux and Docker have bugs, and may occasionally crash. Here again, when they do, Enclave operations may fail.

Rollbacks

With this in mind, Enclave was designed to work around failures and mitigate their impact. With exception of a handful of trivial operations with no side-effects (such as launching Ephemeral SSH Sessions), all Enclave Operations are designed to support rollbacks in the event of a failure.

This means that if an Operation fails, Enclave will proceed to undo anything that can be undone (ideally and usually rolling back all changes that were performed).

If a rollback happens, you’ll be notified in the logs streamed to your terminal while the Operation is running. Ultimately, Enclave will indicate whether the rollback succeeded (i.e. everything was restored back to the way it was before the Operation) or failed (some changes could not be undone).

Warning

Some side-effects of deployments cannot be rolled back by Enclave. In particular, database migrations performed in before_release commands cannot be rolled back (unless you design your migrations to roll back on failure, of course!).

We strongly recommend designing your database migrations so that they are backwards compatible across at least one release. This is a very good idea in general (not just on Enclave), and a best practice for zero-downtime deployments (see Concurrent Releases for more information).

Tip

You can initiate a rollback manually using the aptible operation:cancel command.

Minimal downtime

To further mitigate the impact of failures Enclave Operations are designed to be interruptible at any stage whenever possible.

In particular, when deploying a web application, Enclave performs Zero-Downtime Deployment. This ensures that if the Operation is interrupted at any time and for any reason, it still won’t take your application down.

When downtime is inevitable (such as when resizing a Database volume or redeploying a Database to a bigger instance), Enclave optimizes for minimal downtime.

For example, when redeploying a Database to another instance, Enclave must perform the following steps:

  • Shut down the old Database Container.
  • Unmount then detach the Database volume from the instance the Database was originally scheduled on.
  • Attach then remount the Database volume on the instance the Database is being re-scheduled on.
  • Start the new Database Container.

When performing this Operation, Enclave will minimize downtime by ensuring that all preconditions are in place to start the new Database Container on the new instance before shutting down the old Database Container. In particular, Enclave will ensure the new instance is available and has pre-pulled the Docker image for your Database.