The TasksMax Systemd/Linux feature can cause various operational issues related to creating new processes including failures starting containers and failures setting up iptables rules for running containers. Customers affected by this issue will observe that the Docker daemon is unable to create more processes than the TasksMax configured limit.
Error messages in
journalctu -u docker log output may include one or more of the following:
fork/exec /proc/self/exe: resource temporarily unavailable level=error msg="Error running container:  System error: fork/exec /proc/self/exe: resource temporarily unavailable" Resource temporarily unavailable: apr_thread_create: unable to create worker thread fork failed: Resource temporarily unavailable runtime/cgo: pthread_create failed: Resource temporarily unavailable
The output of
systemctl status docker | grep Tasks will list a number of Tasks and a Limit. Customers actively affected by this issue may have a number of running Docker tasks close to the stated limit as shown below:
Tasks: 505 (Limit: 512)
To be affected by this issue, ALL of the following must be true:
- systemd must be greater than or equal to 227 (version 219 for RHEL)
- Linux kernel must be great than or equal to 4.3 (version 3.10 for RHEL)
- The output of
systemctl status docker | grep Tasksincludes a
This issue was resolved by Docker EE 17.06.2-ee-7.
A security feature was added to Systemd 227 which added support for the
pids cgroup controller. This allows for accounting for the number of tasks in a cgroup with task limit enforcement. The purpose of this feature is to limit the scope of fork bomb denial of service attacks by limiting the number of processes a service can create. The default configuration for some Linux distributions may include a TasksMax limit that is too low for some Docker customers, leading to the issues described in this article.
Confirm you are affected by the issue:
$ systemd --version systemd 229 $ uname -r 4.4.0-116-generic $ systemctl status docker | grep Tasks Tasks: 469 (Limit: 512)
Reconfigure the limit for the Docker server to stop limiting the number of tasks that can be created (as root):
# sudo systemctl set-property docker.service TasksMax=infinity
# systemctl daemon-reload
# systemctl restart docker
Verify the Tasks limit has been removed:
$ systemctl status docker | grep Tasks Tasks: 623