Linux has become a dominant OS for application back ends and micro-services in the cloud. Usage limits (aka ulimits) are a critical Linux application performance tuning tool. Docker is now the leading mechanism for application deployment and distribution and AWS ECS is one of the top Docker container services. It's more important than ever for developers to understand ulimits and how to use them in Linux, Docker and a service like AWS ECS.
The purpose of ulimits is to limit a program's resource utilization to prevent a run-away bug or security breach from bringing the whole system down. It is easy for modern applications to exceed default open file limits very quickly.
NoSQL databases such as Cassandra open thousands of memory-mapped files and can handle thousands of network socket connections concurrently. Datastax recommends a set of production ulimits. Tuning ulimits is also essential for adequate performance of popular Web servers such as Nginx and Apache.
In Linux, the ulimits may be set in two ways. The system administrator can set the global limits by editing /etc/security/limits.conf file. Individual users can also set their own limits using the ulimit command. User-applied limits may not exceed the hard limits set by the administrator.
Configuring limits in Docker containers
Controlling the limits becomes a bit trickier when Docker is involved. The Docker daemon runs as root user. By default, the same limits apply to the application running within a container as they would to the Docker daemon. Adjusting limits within a container, however, requires privileges not available to the application inside the Docker container and must be done as parameters to the docker run command.
For example:
docker run -it --ulimit nofile=2048:2048 ubuntu bash
root@01da5d7fa50b:/# ulimit -n
2048
Configuring Docker container limits in AWS ECS
AWS ECS is organized around task definitions, clusters, tasks and services. A task definition describes the docker images and various parameters for the container. A task is a running instance of a task definition. A service is a managed execution of one or more tasks. AWS ECS can ensure that the correct number of service instances is running and that the instances are connected to the Elastic Load Balancer if needed. Services and tasks run on EC2 instances that are part of an ECS cluster.
Since we do not have direct access to Docker command line in AWS ECS, we have to configure ulimits using ECS Task Definitions. The Ubuntu image example from above would be configured like this in the ECS Task Definition:
"containerDefinitions": [
{
...
"ulimits": [
{
"softLimit": 2048,
"hardLimit": 2048,
"name": "nofile"
}
]
...
}
]
Conclusion
Limits are a critical application tuning parameter. Cloud Docker services have their own mechanism for configuring ulimits. In this article I talked about AWS ECS. However, similar configuration tools exist for Google Container Engine and Azure Container Service. Regardless of the service, it is important to learn how ulimits work and what they impact.