0 0 Share PDF

How to resolve Resource Temporarily Unavailable errors due to TaskMax setting

Article ID: KB000661

Issue

The TasksMax Systemd/Linux feature can cause various operational issues related to creating new processes including failures starting containers and failures setting up iptables rules for running containers. Customers affected by this issue will observe that the Docker daemon is unable to create more processes than the TasksMax configured limit.

Error messages in journalctu -u docker log output may include one or more of the following:

fork/exec /proc/self/exe: resource temporarily unavailable
level=error msg="Error running container: [8] System error: fork/exec /proc/self/exe: resource temporarily unavailable"
Resource temporarily unavailable: apr_thread_create: unable to create worker thread
fork failed: Resource temporarily unavailable
runtime/cgo: pthread_create failed: Resource temporarily unavailable

The output of systemctl status docker | grep Tasks will list a number of Tasks and a Limit. Customers actively affected by this issue may have a number of running Docker tasks close to the stated limit as shown below:

Tasks: 505 (Limit: 512)

Prerequisites

To be affected by this issue, ALL of the following must be true:

  • systemd must be greater than or equal to 227 (version 219 for RHEL)
  • Linux kernel must be great than or equal to 4.3 (version 3.10 for RHEL)
  • The output of systemctl status docker | grep Tasks includes a Limit

This issue was resolved by Docker EE 17.06.2-ee-7.

Root Cause

A security feature was added to Systemd 227 which added support for the pids cgroup controller. This allows for accounting for the number of tasks in a cgroup with task limit enforcement. The purpose of this feature is to limit the scope of fork bomb denial of service attacks by limiting the number of processes a service can create. The default configuration for some Linux distributions may include a TasksMax limit that is too low for some Docker customers, leading to the issues described in this article.

Resolution

  1. Confirm you are affected by the issue:

    $ systemd --version
    systemd 229
    $ uname -r
    4.4.0-116-generic
    $ systemctl status docker | grep Tasks
        Tasks: 469 (Limit: 512)
    
  2. Reconfigure the limit for the Docker server to stop limiting the number of tasks that can be created (as root):

    # sudo systemctl set-property docker.service TasksMax=infinity
    
  3. Restart systemd:

    # systemctl daemon-reload
    
  4. Restart docker:

    # systemctl restart docker
    
  5. Verify the Tasks limit has been removed:

    $ systemctl status docker | grep Tasks
        Tasks: 623