Open File Limit on Linux

Oracle 10.2.0.2
3-node RAC
RHEL AS release 4

Our application experienced performance downgrade today and user can not get response from application for around an hour. Not long after that, when we tried to log into the system, a message 'Too many open files' showed up and we were unable to login.

Searching a little bit, I found that Linux limits the number of open file handles. I recently installed Oracle Grid Control Agent on our RAC and created new database. The open file handles was increased up to the limit, and the system was unstable.

The file /proc/sys/fs/file-nr lists the number of current allocated file handles, available file handles in the allocated file handles and maximum file handles that can be opend for the whole system.
Oracle recommends that the file handles for the entire system be set to at lease 65536.
You can alter the default setting for the maximum number of file handles without rebooting the machine by making the changes directly to the /proc file system (/proc/sys/fs/file-max) using the following:

# sysctl -w fs.file-max=65536

You should then make this change permanent by inserting the kernel parameter in the /etc/sysctl.conf startup file:

# echo "fs.file-max=65536" >> /etc/sysctl.conf

The soft limit and hard limit of file handles for oracle can be configued inside /etc/security/limits.conf file.

cat >> /etc/security/limits.conf <<EOF
oracle soft mnproc 2047
oracle hard mnproc 16384
oracle soft nofile 1024
oracle hard nofile 65536

nofile is the number of open file handles.
mnproc is hte number of processes available to a single user