vCenter Appliance root Partition 100% full

vCenter Appliance root Partition 100% full due to Audit.log files

Symptoms
  • 100% capacity used for /dev/sda3.
  • Size of audit.log file is very large and /var/log/audit folder consumes majority of the space.
  • Saved logs from log rotate policy reference a date that is not in line with the policy.
  • Unable to connect to the vCenter Server as services are not started.
  • Running /etc/cron.daily/logrotate manually rotates logs as expected.
  • Accessing vSphere Web Client might fail with error: 503 service unavailable
Resolution
To resolve this issue, truncate the audit.log file and verify the cron job is working correctly.

Truncate audit.log

  1. Log in to the vCenter Server Appliance through SSH.
  2. Run this command to enable access the Bash shell:

    shell.set –enabled true

  3. Type shell and press Enter.
  4. Navigate to the /var/log/audit folder with this command:

    cd /var/log/audit

  5. Run this command to verify the issue is with the audit.log file being too large (a few GBs):

    ls -lh

    For example:

    ls -lh

    total 3.5G
    -rw——- 1 root root 3.5G Feb 3 16:55 audit.log
    -rw——- 1 root root 445K Apr 8 2016 audit.log-20160408.bz2
    -rw——- 1 root root 447K Apr 9 2016 audit.log-20160409.bz2

  6. Truncate (clean the content without deleting the file) the audit.log file with this command:

truncate -s 0 audit.log