| View previous topic :: View next topic |
| Author |
Message |
dnadesign
Apprentice
Joined: 23 Dec 2006
Posts: 172
Location: Poland
|
 Posted: Sun Feb 25, 2007 9:00 am Post subject: [FALSE ALARM] updatedb memory leaks!! |
 |
|
Hi all,
I noticed a very concerning fact about updatedb (presumably a part of the slocate package?). It's configurated as a background job that goes around once per day. What a did notice is that the process consumes most of my RAM memory (I have 1,5 GB in my notebook) and after finishing the execution the memory is not freed!! Here is the output of dstat -afv:
| Code: | -------cpu0-usage--------------cpu1-usage------ -disk/total --net/eth0- ---paging-- ---system-- ---procs--- ------memory-usage----- ---paging-- -disk/total ---system-->
usr sys idl wai hiq siq:usr sys idl wai hiq siq|_read write|_recv _send|__in_ _out_|_int_ _csw_|run blk new|_used _buff _cach _free|__in_ _out_|_read write|_int_ _csw_>
9 0 91 0 0 0: 2 3 0 95 0 0| 348k 316k| 0 0 | 0 0 | 418 733 | 2 1 0| 926M 310M 133M 11M| 0 0 | 348k 316k| 418 733 >
1 1 83 15 0 0: 9 4 2 85 0 0| 296k 1636k| 0 0 | 0 0 | 431 623 | 0 2 0| 926M 310M 132M 12M| 0 0 | 296k 1636k| 432 623 >
2 1 96 0 0 0: 2 7 0 89 1 0| 348k 4096B| 0 0 | 0 0 | 380 597 | 2 1 0| 928M 310M 130M 12M| 0 0 | 348k 4096B| 380 597 >
0 0 94 6 0 0: 5 5 2 88 0 0| 316k 0 | 0 0 | 0 0 | 391 553 | 2 1 0| 929M 310M 129M 12M| 0 0 | 316k 0 | 391 553 >
3 1 94 2 0 0: 4 5 2 87 1 1| 312k 0 | 0 0 | 0 0 | 411 659 | 3 1 0| 930M 311M 128M 12M| 0 0 | 312k 0 | 411 659 >
3 2 88 7 0 0: 4 4 3 88 0 0| 260k 0 | 0 0 | 0 0 | 379 530 | 1 1 0| 931M 311M 126M 12M| 0 0 | 260k 0 | 378 530 >
8 1 88 3 0 0: 1 4 3 92 0 0| 276k 1300k| 0 0 | 0 0 | 409 622 | 1 2 0| 931M 311M 126M 11M| 0 0 | 276k 1300k| 410 624 >
8 0 92 0 0 0: 2 4 0 92 0 1| 244k 4096B| 0 0 | 0 0 | 367 547 | 2 1 0| 932M 312M 124M 12M| 0 0 | 244k 4096B| 367 545 >
2 2 92 4 0 0: 6 4 2 88 0 0| 248k 0 | 0 0 | 0 0 | 393 600 | 1 1 0| 933M 312M 123M 12M| 0 0 | 252k 0 | 395 600 >
10 1 89 0 0 0: 4 4 2 90 0 0| 392k 0 | 0 0 | 0 0 | 469 1121 | 4 0 0| 933M 312M 123M 11M| 0 0 | 388k 0 | 468 1122 >
20 0 80 0 0 0: 17 4 50 27 0 2| 108k 2068k| 407B 435B| 0 0 | 595 1090 | 1 0 0| 934M 312M 122M 12M| 0 0 | 108k 2068k| 593 1089 >
<-- this is the point where updatedb stopped running and the process is not visible in top, htop or ps anymore -->
<-- as you see the memory is still being used by something!! when updatedb started only 150MB of ram was used, after it has finished it's 934MB -->
10 0 90 0 0 0: 19 0 80 0 0 1| 0 60k| 0 0 | 0 0 | 327 970 | 1 1 0| 934M 312M 122M 12M| 0 0 | 0 60k| 328 970 >
0 0 100 0 0 0: 10 0 84 4 1 0| 0 828k| 66B 60B| 0 0 | 397 1218 | 1 0 0| 934M 312M 122M 12M| 0 0 | 0 828k| 397 1218 >
1 1 98 0 0 0: 2 0 98 0 0 0| 0 0 | 0 0 | 0 0 | 313 410 | 1 0 0| 934M 312M 122M 12M| 0 0 | 0 0 | 313 413 >
7 1 92 0 0 0: 2 1 97 0 0 0| 0 0 | 0 0 | 0 0 | 332 480 | 2 0 0| 934M 312M 122M 12M| 0 0 | 0 0 | 333 477 >
8 1 91 0 0 0: 1 0 24 72 1 2| 0 2944k| 60B 60B| 0 0 | 624 875 | 4 1 0| 934M 312M 122M 12M| 0 0 | 0 2944k| 623 877 >
7 1 92 0 0 0: 1 1 0 97 0 1| 0 1468k| 60B 0 | 0 0 | 618 973 | 2 2 0| 934M 312M 122M 12M| 0 0 | 0 1512k| 619 971 >
9 1 66 24 0 0: 2 1 25 69 1 1|8192B 1632k| 0 0 | 0 0 | 589 1239 | 2 0 0| 934M 312M 122M 12M| 0 0 |8192B 1588k| 588 1243 >
4 0 96 0 0 0: 3 1 96 0 0 0| 0 0 | 0 0 | 0 0 | 331 538 | 1 0 0| 934M 312M 122M 12M| 0 0 | 0 0 | 332 535 >
6 0 94 0 0 0: 3 0 97 0 0 0| 0 0 | 0 0 | 0 0 | 312 483 | 3 0 0| 934M 312M 122M 12M| 0 0 | 0 0 | 311 482 >
4 0 96 0 0 0: 1 0 99 0 0 0| 0 0 | 0 0 | 0 0 | 332 523 | 1 0 0| 934M 312M 122M 12M| 0 0 | 0 0 | 331 523 > |
As you see, the memory is still being reserved by "something". Running "ps aux | grep updatedb" shows:
| Code: | bitinfo-dnalap ~ # ps aux | grep updatedb
root 9193 0.0 0.0 3024 732 pts/0 S+ 09:55 0:00 grep updatedb |
Here's the full ps aux output:
| Code: | bitinfo-dnalap ~ # ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 2672 608 ? Ss 09:06 0:01 init [3]
root 2 0.0 0.0 0 0 ? S 09:06 0:00 [migration/0]
root 3 0.0 0.0 0 0 ? SN 09:06 0:00 [ksoftirqd/0]
root 4 0.0 0.0 0 0 ? S 09:06 0:00 [watchdog/0]
root 5 0.0 0.0 0 0 ? S 09:06 0:00 [migration/1]
root 6 0.0 0.0 0 0 ? SN 09:06 0:00 [ksoftirqd/1]
root 7 0.0 0.0 0 0 ? S 09:06 0:00 [watchdog/1]
root 8 0.0 0.0 0 0 ? S< 09:06 0:00 [events/0]
root 9 0.0 0.0 0 0 ? S< 09:06 0:00 [events/1]
root 10 0.0 0.0 0 0 ? S< 09:06 0:00 [khelper]
root 11 0.0 0.0 0 0 ? S< 09:06 0:00 [kthread]
root 60 0.0 0.0 0 0 ? S< 09:06 0:00 [kblockd/0]
root 61 0.0 0.0 0 0 ? S< 09:06 0:00 [kblockd/1]
root 62 0.0 0.0 0 0 ? S< 09:06 0:00 [kacpid]
root 170 0.0 0.0 0 0 ? S< 09:06 0:00 [kseriod]
root 171 0.0 0.0 0 0 ? S< 09:06 0:00 [ata/0]
root 172 0.0 0.0 0 0 ? S< 09:06 0:00 [ata/1]
root 173 0.0 0.0 0 0 ? S< 09:06 0:00 [ata_aux]
root 174 0.0 0.0 0 0 ? S< 09:06 0:00 [ksuspend_usbd]
root 177 0.0 0.0 0 0 ? S< 09:06 0:00 [khubd]
root 212 0.0 0.0 0 0 ? S 09:06 0:00 [pdflush]
root 213 0.0 0.0 0 0 ? S 09:06 0:00 [pdflush]
root 214 0.0 0.0 0 0 ? S< 09:06 0:00 [kswapd0]
root 215 0.0 0.0 0 0 ? S< 09:06 0:00 [aio/0]
root 216 0.0 0.0 0 0 ? S< 09:06 0:00 [aio/1]
root 348 0.0 0.0 0 0 ? S< 09:06 0:00 [kpsmoused]
root 368 0.0 0.0 0 0 ? S< 09:06 0:00 [scsi_eh_0]
root 369 0.0 0.0 0 0 ? S< 09:06 0:00 [scsi_eh_1]
root 383 0.0 0.0 0 0 ? S< 09:06 0:00 [exec-osm/0]
root 384 0.0 0.0 0 0 ? S< 09:06 0:00 [exec-osm/1]
root 391 0.0 0.0 0 0 ? S< 09:06 0:00 [kmmcd]
root 1347 0.0 0.0 0 0 ? S< 09:06 0:00 [kmirrord]
root 2618 0.0 0.0 0 0 ? S< 09:06 0:00 [kjournald]
root 2728 0.0 0.0 7128 744 ? S<s 09:06 0:00 /sbin/udevd --daemon
root 3404 0.0 0.0 0 0 ? S< 09:06 0:00 [hda_codec]
root 6423 0.0 0.0 4472 812 ? Ss 09:06 0:00 /sbin/dhclient -e PEER_ROUTERS=yes -e PEER_DNS=yes -e PEER_NTP=yes -q -1 -pf /var/run/dhclient-eth0.pid eth0
root 7264 0.0 0.0 7184 696 ? Ss 09:06 0:00 /usr/sbin/syslog-ng
root 7395 0.0 0.1 54220 2072 ? Ss 09:06 0:00 /usr/bin/gdm
root 7397 0.0 0.2 62952 3056 ? S 09:06 0:00 /usr/bin/gdm
root 7400 8.6 9.5 171384 134276 ? RL 09:06 4:16 /usr/bin/Xgl :1 :1 -ac -accel glx:pbuffer -accel xv:pbuffer -auth /var/gdm/:1.Xauth -nolisten tcp vt7
root 7401 0.1 0.8 61156 11932 tty7 SLs+ 09:06 0:05 /usr/bin/Xorg vt7 -auth /tmp/.Xgl-auth-7XYLB5 -nolisten tcp :94 -terminate
root 7463 0.0 0.0 2660 568 ? S<s 09:06 0:01 /usr/sbin/cpudynd -i 1 -p 0.5 0.9 -l 7 -h /dev/hda,/dev/sda
root 7521 0.0 0.0 10740 948 ? Ss 09:06 0:00 /usr/sbin/xinetd -pidfile /var/run/xinetd.pid -stayalive -reuse
root 7617 0.0 0.0 1324 156 ? S 09:06 0:00 /opt/vmware/server/bin/vmnet-bridge -d /var/run/vmnet-bridge-0.pid /dev/vmnet0 eth0
root 7677 0.0 0.1 30452 2564 ? Ss 09:06 0:00 /usr/sbin/cupsd
101 7735 0.0 0.0 8408 500 ? Ss 09:07 0:00 /usr/bin/dbus-daemon --system
stunnel 7792 0.0 0.0 17016 1188 ? Ss 09:07 0:00 /usr/sbin/stunnel /etc/stunnel/stunnel.conf
root 7848 0.0 0.0 9336 732 ? Ss 09:07 0:00 /usr/sbin/cron
root 7912 0.0 0.0 3728 692 tty1 Ss+ 09:07 0:00 /sbin/agetty 38400 tty1 linux
root 7913 0.0 0.0 3732 700 tty2 Ss+ 09:07 0:00 /sbin/agetty 38400 tty2 linux
root 7914 0.0 0.0 3732 696 tty3 Ss+ 09:07 0:00 /sbin/agetty 38400 tty3 linux
root 7915 0.0 0.0 3732 696 tty4 Ss+ 09:07 0:00 /sbin/agetty 38400 tty4 linux
root 7916 0.0 0.0 3728 692 tty5 Ss+ 09:07 0:00 /sbin/agetty 38400 tty5 linux
root 7917 0.0 0.0 3728 696 tty6 Ss+ 09:07 0:00 /sbin/agetty 38400 tty6 linux
moros 7935 0.0 0.1 6480 1432 ? Ss 09:07 0:00 sh /etc/xdg/xfce4/xinitrc
moros 7962 0.0 0.0 10068 732 ? S 09:07 0:00 /usr/bin/dbus-launch --sh-syntax --exit-with-session
moros 7963 0.0 0.0 8408 920 ? Ss 09:07 0:00 /usr/bin/dbus-daemon --fork --print-pid 4 --print-address 6 --session
moros 7966 0.0 0.0 15484 840 ? Ss 09:07 0:00 /usr/bin/ssh-agent -- startxfce4
moros 7968 0.0 0.6 64132 9216 ? S 09:07 0:02 emerald
moros 7969 0.0 0.6 159576 9828 ? Ssl 09:07 0:00 beryl-manager
moros 7977 0.0 0.0 6480 588 ? S 09:07 0:00 sh /etc/xdg/xfce4/xinitrc
moros 7980 0.0 0.1 26932 2280 ? S 09:07 0:00 xscreensaver -no-splash
moros 7981 0.0 0.9 74228 12808 ? S 09:07 0:00 /usr/bin/xfce4-session
moros 8040 0.4 1.0 98608 14972 ? S 09:07 0:12 beryl-xgl
moros 8048 0.0 0.4 108696 6524 ? Ss 09:07 0:00 xfce-mcs-manager
moros 8053 0.4 1.1 109432 16140 ? S 09:07 0:12 xfce4-panel
moros 8055 0.0 0.6 63268 8772 ? S 09:07 0:01 xftaskbar4
moros 8057 0.0 1.1 93688 16360 ? S 09:07 0:01 xfdesktop
moros 8089 0.0 0.1 6616 1688 ? S 09:13 0:00 /bin/bash /usr/libexec/mozilla-launcher
moros 8098 5.8 7.1 428240 100968 ? Rl 09:14 2:28 /usr/lib64/mozilla-firefox/firefox-bin
moros 8107 0.0 0.2 21656 3164 ? S 09:14 0:00 /usr/libexec/gconfd-2 12
moros 8156 0.0 0.4 37720 6484 ? S 09:16 0:01 xterm -title Terminal
moros 8158 0.0 0.1 10916 2032 pts/0 Ss 09:16 0:00 bash
root 8162 0.0 0.1 11664 2112 pts/0 S 09:16 0:00 -bash
moros 8929 0.0 0.3 36128 4820 ? S 09:37 0:00 xterm -title Terminal
moros 8931 0.0 0.1 10912 2028 pts/1 Ss 09:37 0:00 bash
root 9167 0.0 0.1 11584 1976 pts/1 S+ 09:45 0:00 -bash
root 9194 0.0 0.0 8648 996 pts/0 R+ 09:56 0:00 ps aux |
Having that, should I submit a bug report? If so, where should I submit: gentoo or someone from non-gentoo community responsible for slocate?
Thanks in advance,
DNA DesigN
Last edited by dnadesign on Wed Feb 28, 2007 9:06 am; edited 1 time in total |
|
| Back to top |
|
 |
moocha
Watchman
Joined: 21 Oct 2003
Posts: 5722
|
| Posted: Sun Feb 25, 2007 10:32 am Post subject: |
|
|
I fail to see any leak there. There are two possibilities:- If indeed there is memory allocated and never freed after updatedb has finished, then that's a bug in the kernel. However, again, I don't see any leak in what you posted - the dfstat output you posted shows no significant process-allocated memory increase at the point you marked. This possibility is highly unlikely.
- You got confused by the (addmittedly easy to misunderstand) terms used in kernel space - I'm referring to "free", "buffers" and "cache" memory. Please see here: http://gentoo-wiki.com/FAQ_Linux_Memory_Management. I'm willing to bet this is the cause of your mysterious leak - the simple fact that Linux actually makes use of memory for disk caching and doesn't leave it lying around unused, and of course updatedb fills up the cache because it walks the entire file system hierarchy.
_________________
Military Commissions Act of 2006: http://tinyurl.com/jrcto
"Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety."
-- attributed to Benjamin Franklin |
|
| Back to top |
|
|
dnadesign
Apprentice
Joined: 23 Dec 2006
Posts: 172
Location: Poland
|
| Posted: Sun Feb 25, 2007 12:35 pm Post subject: |
|
|
| It's true it can't be seen in the dstat output, because it's just the end of the output that I posted (the original is long as hell, that's why I posted only the ending). At the beginning, before executing updatedb the _used column showed only 150MB of usage. After the updatedb process started the memory usage began to rise and at the point I marked the process finished it's execution, but the memory allocated by it was not freed. |
|
| Back to top |
|
|
devsk
Advocate
Joined: 24 Oct 2003
Posts: 3003
Location: Bay Area, CA
|
| Posted: Tue Feb 27, 2007 5:07 pm Post subject: |
|
|
short answer: kernel has buffered whatever updatedb plowed thru on disk. it will release it lazily if and when needed.
long answer: read the link moocha pointed to in 2. |
|
| Back to top |
|
|
i92guboj
Bodhisattva
Joined: 30 Nov 2004
Posts: 10315
Location: Córdoba (Spain)
|
| Posted: Tue Feb 27, 2007 5:32 pm Post subject: |
|
|
| ++ Read up on the differences between used, free, buffered and cached memory. There is no leak. |
|
| Back to top |
|
|
dnadesign
Apprentice
Joined: 23 Dec 2006
Posts: 172
Location: Poland
|
| Posted: Tue Feb 27, 2007 8:05 pm Post subject: |
|
|
Well ok. If you say so. I'll just wait for the next updatedb (too lazy to run it manually  ) to execute and after seeing my RAM go to hell I'll just run Eclipse and BEA WebLogic with some of my currently developed projects (they involve clustering  ). If you're right then everything should return to normal levels (because these two apps in addition to taking my full RAM capacity - Java... it's normal - take an additional 512MB of swap). Ahh... the simple pleasures of running a public sector clustered app on a laptop (development purposes). 
BTW: I already read the HowTO, so no need telling me this 3 times guys.
BTW2: So the kernel cache memory isn't displayed in dstat? Only in "free"? |
|
| Back to top |
|
|
i92guboj
Bodhisattva
Joined: 30 Nov 2004
Posts: 10315
Location: Córdoba (Spain)
|
| Posted: Tue Feb 27, 2007 9:01 pm Post subject: Re: updatedb memory leaks!! |
|
|
| dnadesign wrote: | Hi all,
I noticed a very concerning fact about updatedb (presumably a part of the slocate package?). It's configurated as a background job that goes around once per day. What a did notice is that the process consumes most of my RAM memory (I have 1,5 GB in my notebook) and after finishing the execution the memory is not freed!! Here is the output of dstat -afv:
|
If there was a memory leak in this circunstance, and the memory is not freed as you say, you could not even start a java vm, not to talk about BEA Weblogic... The kernel would kill one or the other randomly.
As you say in your latest post, test it. The ram should be freed in the measure that it is needed. Of course, unless it is needed for other execution, caching of buffering purposes, most of them will continue to be marked as cached/buffered unless it is really needed. Note that starting a java vm will not magically redime the kernel of the rest of its memory and i/o administration tasks, so unless the memory is needed for any other thing it will continue that way.
If that is not what happen, then you have a base to claim a memory leak, but it is not very likely to find such a thing into updatedb... not impossible, though. |
|
| Back to top |
|
|
moocha
Watchman
Joined: 21 Oct 2003
Posts: 5722
|
| Posted: Tue Feb 27, 2007 10:50 pm Post subject: Re: updatedb memory leaks!! |
|
|
| 6thpink wrote: | | If that is not what happen, then you have a base to claim a memory leak, but it is not very likely to find such a thing into updatedb... not impossible, though. |
Again, the probability that the behavior described is a "memory leak in updatedb" is exactly zero, nada and zilch.
The kernel rescinds any and all memory allocated by a process via the malloc family or via directly adjusting brk/sbrk the moment that process dies. If the kernel should fail to do that, it's a bug in the kernel. Unlikely, since a bug with effects of this magnitude would in all likelihood been encountered before - at the scale of Linux deployments, a one in a million chance means "next Friday".
A process could conceivably trigger a memory leak in the kernel by asking it to allocate kernel-side memory via some syscall or the other, but such resources are also freed when the process dies. If they're not, it's a bug in the kernel. Again unlikely for the abovementioned reason.
The third way by which a process could eat up memory post-mortem is to indirectly cause a third process to leak memory if that third process has registered itself as a callback from the kernel somehow triggered by updatedb (an example would be a fuse module like, say, ntfs-3g allocating some memory because the kernel-side fuse module asked it to because updatedb walked the file system controlled by ntfs-3g), or if our trigger process (updatedb) directly communicates with the leaker (e.g., famd or gamin). This would be a bug in the third process and, again, not a bug in updatedb. This intuitively sounds more likely but there is not enough data to make an asessment.
I wish people would stop throwing around terms like "memory leak" without first being reasonably sure those terms apply to the problem (although, in all fairness, I'll never forget Mr. Chen's article about how A cache with a bad policy is another name for a memory leak.)
_________________
Military Commissions Act of 2006: http://tinyurl.com/jrcto
"Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety."
-- attributed to Benjamin Franklin |
|
| Back to top |
|
|
dnadesign
Apprentice
Joined: 23 Dec 2006
Posts: 172
Location: Poland
|
| Posted: Wed Feb 28, 2007 9:06 am Post subject: [FALSE ALARM] updatedb memory leaks!! |
|
|
No. Nothing like that. Just a simple execution of updatedb on local ext2/ext3 file systems. No VFAT (no "Windows Partitions Of Shame" on my laptop ;P) or NTFS (keep them unmounted, for use only with WinXP when I --really-- need it). Well then, I'll have to brush up on my Linux knowledge. I never went that far into it, I know what I needed to to operate this OS (config, application compilation and instalation, kernel recompilation and some tuning and few other things that I don't remember at the moment).
Thanks for the help.  |
|
| Back to top |
|
|
dnadesign
Apprentice
Joined: 23 Dec 2006
Posts: 172
Location: Poland
|
| Posted: Fri Mar 02, 2007 10:16 am Post subject: |
|
|
Ok, it's officially verified that... you're right. As expected, the memory usage normalized it self once I launched Eclipse and BEA WL after updatedb was running. For the first time I saw the RAM memory usage actually falling when starting a cluster node (kernel was freeing it's memory faster than Java was allocating it). 
Anyway everything works fine. Thanks again. |
|
| Back to top |
|
|
moocha
Watchman
Joined: 21 Oct 2003
Posts: 5722
|
| Posted: Fri Mar 02, 2007 2:44 pm Post subject: |
|
|
Anytime, and have fun
_________________
Military Commissions Act of 2006: http://tinyurl.com/jrcto
"Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety."
-- attributed to Benjamin Franklin |
|
| Back to top |
|
|
|
Other Things Gentoo
