0 0 Share PDF

How to workaround high CPU and slow logins on UCP and DTR nodes

Article ID: KB000993

Issue

Customers using UCP and DTR may experience an issue where logging in to UCP or DTR becomes slow, and eventually the UCP and DTR UI may become unavailable due to high CPU.

Prerequisites

This issue may affect clusters where the following is true:

-UCP 3.1.x -DTR 2.6.x -DTR Security Scanning is enabled

Root Cause

An ongoing investigation has identified a performance bottleneck in a new API call which is used to display image vulnerability scan results from DTR Security Scanning in UCP.

Resolution

Note: Before taking steps to correct this issue, please make sure you have a recent UCP and DTR metadata backup.

While a permanent fix is in progress, the issue can be worked around by options (A OR B) AND C below:

A. Disable DTR Security Scanning.

OR

B. Disable the BatchScanningDataEnabled API call UCP uses to collect scanning data from DTR.

AND

C. After completing A. or B. above, remove all running DTR scanning jobs to resolve the high CPU.

Disable DTR Security Scanning and remove all running DTR scanning jobs

While this option may be easier to implement, it disables DTR Security Scanning. After disabling scanning, you may need to clear any running image scan jobs prior to seeing CPU utilization on DTR and UCP nodes decrease.

Disable DTR Security Scanning

  1. In the DTR web UI, navigate to System > Security and click the blue switch labeled "Enable Scanning" to turn scanning off.

Disable the BatchScanningDataEnabled API call UCP uses to collect scanning data from DTR

Opting to disable BatchScanningDataEnabled is the recommended workaround because the DTR Security Scanning feature will continue operating. After disabling BatchScanningDataEnabled with the steps below, you may need to clear any running image scan jobs prior to seeing CPU utilization on DTR and UCP nodes decrease.

To disable the API call directly, use the dlouca/togglescan:latest image available on Docker Hub by following these steps:

  1. Set the UCPIP variable to the IP address of a UCP node:
UCPIP=192.168.1.1
  1. Run the following command to disable the BatchScanningDataEnabled API call:
docker run -it dlouca/togglescan:latest -a $UCPIP -u admin

Warning: This step will need to be repeated if a DTR replica is joined or removed from the DTR cluster.

Usage information for the dlouca/togglescan image is available here: https://github.com/dani-docker/toggleScan

Delete existing DTR scanning jobs

Existing DTR scanning jobs can be deleted by restarting the Docker daemon dockerd process on each DTR replica, or by manually deleting the jobs using the DTR API. To delete the jobs using the API:

  1. Navigate to System > Job Logs and in the Filter box select scan_check_all. Note the ID of any running jobs.
  2. Navigate to the DTR Live API by clicking "API" in the bottom left corner of DTR.
  3. Find the API call for DELETE /api/v0/jobs/{jobID}, click "Try it out", paste the JobID of the a running job from Step 1 in the "jobID" field and click "Execute".
  4. Repeat Step 3 for each running job.