Memory Limits

Enclave enforces memory limits on all Containers deployed in Shared Stacks and all Containers deployed in Dedicated Stacks if you signed up as a customer after June 2017.

Memory limits are enforced through a feature called Memory Management.

Memory Management

When memory management is enabled on your infrastructure and a Container exceeds its memory allocation, the following happens:

  1. Enclave sends a log message to your Log Drains (this includes aptible logs) indicating that your Container exceeded its memory allocation, and dumps a list of the processes running in your Container for troubleshooting purposes.
  2. If there is free memory on the instance, Enclave increases your Container’s memory allowance by 10%. This gives your Container a better shot at exiting cleanly.
  3. Enclave delivers a SIGTERM signal to all the processes in your Container, and gives your Container 10 seconds to exit. If your Container does not exit within 10 seconds, Enclave delivers a SIGKILL signal, effectively terminating all the processes in your Container immediately.
  4. Enclave triggers Container Recovery to restart your Container.

Tip

The metrics you see in the Dashboard are captured every minute. If your Container exceeds its RAM allocation very quickly then being restarted, the metrics you see in the Dashboard may not reflect the usage spike.

As such, it’s a good idea to refer to your logs as the authoritative source of information to know when you’re exceeding your memory allocation.

Indeed, whenever your Containers are restarted, Enclave will log this message to all your Log Drains:

container exceeded its memory allocation

This message will also be followed by a snapshot of the memory usage of all the processes running in your Container, so you can identify memory hogs more easily.

Here is an example. The column that shows RAM usage is RSS, and that usage is expressed in kilobytes.

  PID    PPID      VSZ      RSS  STAT    COMMAND
 2688    2625      820       36  S       /usr/lib/erlang/erts-7.3.1/bin/epmd -daemon
 2625     914     1540      936  S       /bin/sh -e /srv/rabbitmq_server-3.5.8/sbin/rabbitmq-server
13255     914     6248      792  S       /bin/bash
 2792    2708      764       12  S       inet_gethost 4
 2793    2792      764       44  S       inet_gethost 4
 2708    2625  1343560  1044596  S       /usr/lib/erlang/erts-7.3.1/bin/beam.smp [...]

What should my app do when it receives a SIGTERM from Enclave?

Your app should try and exit gracefully within 10 seconds.

If your app is processing background work, you should ideally try and push it back to whatever queue it came from.

How do I know the memory limit for a Container?

You can view the current memory limit for any Container by looking at the Container’s Metrics in your Dashboard: the memory limit is listed right next to your current usage.

Additionally, Enclave sets the APTIBLE_CONTAINER_SIZE environment variable when starting your Containers. This indicates your Container’s memory limit, in MB.

How do I increase the memory limit for a Container?

See Scaling.