| View previous topic :: View next topic |
| Author |
Message |
kashani
Advocate
Joined: 02 Sep 2002
Posts: 2032
Location: San Francisco
|
 Posted: Wed Sep 15, 2004 11:31 pm Post subject: How much does metalog improve disk I/O? |
 |
|
I've finally put out enough fires at work that I can start dusting off some research projects. One that came up today is that disk I/O tends to be one of our performance bottlenecks. We generally get around this by using a RAID card with a hefty cache to buffered the disk reads and writes. And also by putting the OS and logging on a single disk and putting content on a second drive. Both seem to work well especially in tandem and generally solve the problem for us, but we have to spend an extra $300-500 per server depending on the config.
It is my understanding that metalog, a common syslogd replacement, has a caching system. Instead of constantly writing to disk it hold it in RAM and then dumps out by blocks which is more efficient. The $64 question is:
Does this really increase performance significantly?
Ideally I'd like to hear from someone who generates around 1 GB of weblogs per day per server and has seen metalog increase system performance around at least 10-20% by decreasing disk I/O due to logging.
kashani
_________________
Will personally fix your server in exchange for motorcycle related shop tools in good shape. |
|
| Back to top |
|
 |
fleed
l33t
Joined: 28 Aug 2002
Posts: 756
Location: London
|
| Posted: Thu Sep 16, 2004 9:58 am Post subject: |
|
|
| Sorry dude, no such experience. But you might also want to have a look at syslog-ng. While it's setup is more involved, I get the feeling it lets you control your syslog much better. |
|
| Back to top |
|
|
kashani
Advocate
Joined: 02 Sep 2002
Posts: 2032
Location: San Francisco
|
| Posted: Thu Sep 16, 2004 4:37 pm Post subject: |
|
|
syslog-ng is what I normally use, but in this case all I care about is getting better disk I/O. Hell I'd turn off logging completely if I could get away with it.
kashani
_________________
Will personally fix your server in exchange for motorcycle related shop tools in good shape. |
|
| Back to top |
|
|
fleed
l33t
Joined: 28 Aug 2002
Posts: 756
Location: London
|
| Posted: Fri Sep 17, 2004 11:10 am Post subject: |
|
|
| You probably have but... have you thought about dedicating a machine solely for logging? So you just send your logs over the network to that machine from all the other ones? |
|
| Back to top |
|
|
mirko_3
l33t
Joined: 02 Nov 2003
Posts: 605
Location: Birreria
|
| Posted: Fri Sep 17, 2004 12:07 pm Post subject: |
|
|
Note that metalog now turns buffering off by default - I read that somewhere, maybe upgrading. Just something to look at, if you don't notice a performance increase..
_________________
Non fa male! Non fa male! |
|
| Back to top |
|
|
kashani
Advocate
Joined: 02 Sep 2002
Posts: 2032
Location: San Francisco
|
| Posted: Fri Sep 17, 2004 6:12 pm Post subject: |
|
|
| fleed wrote: | | You probably have but... have you thought about dedicating a machine solely for logging? So you just send your logs over the network to that machine from all the other ones? |
That's also something we're thinking about, but has very little bearing on the original question. Besides relative performance of the logger is of greater importance if I'm tossing 30-40 GB of logs to a central server.
kashani
_________________
Will personally fix your server in exchange for motorcycle related shop tools in good shape. |
|
| Back to top |
|
|
|
Networking & Security
