Last week I had an issue on a freshly installed dev system with 12c and Oracle Restart.
In about 3 months time there were 7,5 million audit files generated in the GI home.
This caused the zfs filesystem to fill up completely ( zfs doesn’t like this 😉 we didn’t finish monitoring yet on this system so we noticed it when damage was done ….
cleaning up the zfs filesystem was one thing, first we had to make some space otherwise we got …
rm: cannot remove : No space left on device
First since is was a dev system and holidays anyways I stop ASM and the databases …After cleaning up some space we could to following …
cd $GRID_HOME/rdbms/audit
find . > /tmp/rm_audit
remove the first line of that file …
cat rm_audit | xargs -l 100 rm
and then slowly the space was given back…
now time to change the check interval to something more reasonable since it was checking each second …. and thus generating an audit file each second or 86400 each day
to view all the parameters
crsctl stat res -p
for example on the ASM instance following parameters are set.
NAME=ora.asm
TYPE=ora.asm.type
ACL=owner:grid:rwx,pgrp:asmdba:r-x,other::r–
ACTIONS=
ACTION_FAILURE_TEMPLATE=
ACTION_SCRIPT=
ACTION_TIMEOUT=60
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
ALERT_TEMPLATE=
ALIAS_NAME=
ASM_DISKSTRING=/dev/oracle/*
AUTO_START=restore
CHECK_INTERVAL=1
CHECK_TIMEOUT=30
CLEAN_TIMEOUT=60
DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=asm) ELEMENT(INSTANCE_NAME= %GEN_USR_ORA_INST_NAME%)
DEGREE=1
DELETE_TIMEOUT=60
DESCRIPTION=Oracle ASM resource
ENABLED=1
GEN_USR_ORA_INST_NAME=+ASM
INSTANCE_FAILOVER=1
but databases are impacted as well.
so we changed it to every minute a check
-bash-4.1$ crsctl modify resource ora.dba12c.db -attr “CHECK_INTERVAL=60”
-bash-4.1$ crsctl modify resource ora.asm -attr “CHECK_INTERVAL=60”
ofcourse a cron job to delete the audit files on a regular basis installed …
