Too Many Open Files

Shadus · n00b Joined: 27 Mar 2005 Posts: 42

I'm running Gentoo on a 2.4-Hardened kernel, I'm having an on going problem that I can't seem to resolve, and wondered if perhaps anyone else had some ideas.

The on going problem occurs with the Sybase v7 ASA DB Engine and the error I'm getting is "To many files open" when it attempts to open a database or transaction log. The machine has 4g of ram, duel processor xeon w/ht, using a xfs file system.

I've tried the following:
echo 1048576 > /proc/sys/fs/file-max

Setting sysctl.conf:
fs.file-max = 1048576
kernel.shmmax = 102400000

Raising the ulimit to 4096 (from 1024):
ulimit -n 4096 (in the script that calls db and in roots login shell (since some of the admins manually restart it))

The sybase database engine seems to take up about 750 file handles on it's own.. it also runs 25 database processes total. The server crashes somewhere in the open file handle range of 25000-27500 (my personal guess would be between 25600-26350 [eg: 25*1024 or 25*1024+750]). It varies by a bit. Also as it nears this limit the database itself slows to a crawl.

I'm out of ideas on what I need to do to correct this problem other than move databases off to another server to lower the number of connections, but that really isn't an acceptable solution at the level of load the server/memory usage is at.

Any ideas or suggestions on where to go from here? I've done some searching on google, but not found a good answer to my problem, I'm beginning to think the limitation may be in the kernel itself, but I've had no real luck finding it there either.

Thoughts, comments, and suggestions are greatly appreciated.

Shadus

Shadus · n00b Joined: 27 Mar 2005 Posts: 42

Doesn't look like anyone has any specific ideas right of the bat, anyone have a suggestion on another more relevent location to post this to?

Shadus · n00b Joined: 27 Mar 2005 Posts: 42

Poking this back to front page for one last try at getting some kinda answer.

Shadus · n00b Joined: 27 Mar 2005 Posts: 42

energyman76b · Advocate Joined: 26 Mar 2003 Posts: 2048 Location: Germany

Hi,

emm, well file-max is the way to go.. so you did the right thing .. maybe your ulimit value is too small?

Shadus · n00b Joined: 27 Mar 2005 Posts: 42

Janne Pikkarainen · Posted: Tue Oct 11, 2005 7:48 pm Post subject:

Never used Sybase myself, but maybe it has its own configuration parameter, where you can tell it the maximum number of file handles. That way it could actually lower the limit you've set up via bash. ulimit -n should be right way to do it, although ulimit -SHn could set both soft and hard limit. Please also see if pam_limits is in use and if /etc/security/limits.conf has some settings for sybase.

Also, traditionally ulimit has been per-process limit. With grsecurity (are you using it?) you may setup the limit to be completely per-user, so for example user john couldn't open more than 300 filehandles even if he started new ssh-logins etc.

So, what I suspect is that you're using grsecurity and now your Sybase collides to some kind of global limit you've setup for sybase account. Ok, 750 * 25 is not anywhere near 65000, but still...

If any of this doesn't help, this starts to sound like a bug in Sybase. I administer busy web etc. servers having at least tens of thousands of open file descriptors during busy hours, and they're not having a bad day because of that.
_________________
Yes, I'm the man. Now it's your turn to decide if I meant "Yes, I'm the male." or "Yes, I am the Unix Manual Page.".

Shadus · n00b Joined: 27 Mar 2005 Posts: 42

Janne Pikkarainen · Posted: Thu Oct 13, 2005 3:45 pm Post subject:

Nice to hear you got your problem "solved".
_________________
Yes, I'm the man. Now it's your turn to decide if I meant "Yes, I'm the male." or "Yes, I am the Unix Manual Page.".