Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[FALSE ALARM] updatedb memory leaks!!
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo
View previous topic :: View next topic  
Author Message
dnadesign
Apprentice
Apprentice


Joined: 23 Dec 2006
Posts: 172
Location: Poland

PostPosted: Sun Feb 25, 2007 9:00 am    Post subject: [FALSE ALARM] updatedb memory leaks!! Reply with quote

Hi all,
I noticed a very concerning fact about updatedb (presumably a part of the slocate package?). It's configurated as a background job that goes around once per day. What a did notice is that the process consumes most of my RAM memory (I have 1,5 GB in my notebook) and after finishing the execution the memory is not freed!! Here is the output of dstat -afv:

Code:
-------cpu0-usage--------------cpu1-usage------ -disk/total --net/eth0- ---paging-- ---system-- ---procs--- ------memory-usage----- ---paging-- -disk/total ---system-->
usr sys idl wai hiq siq:usr sys idl wai hiq siq|_read write|_recv _send|__in_ _out_|_int_ _csw_|run blk new|_used _buff _cach _free|__in_ _out_|_read write|_int_ _csw_>
  9   0  91   0   0   0:  2   3   0  95   0   0| 348k  316k|   0     0 |   0     0 | 418   733 |  2   1   0| 926M  310M  133M   11M|   0     0 | 348k  316k| 418   733 >
  1   1  83  15   0   0:  9   4   2  85   0   0| 296k 1636k|   0     0 |   0     0 | 431   623 |  0   2   0| 926M  310M  132M   12M|   0     0 | 296k 1636k| 432   623 >
  2   1  96   0   0   0:  2   7   0  89   1   0| 348k 4096B|   0     0 |   0     0 | 380   597 |  2   1   0| 928M  310M  130M   12M|   0     0 | 348k 4096B| 380   597 >
  0   0  94   6   0   0:  5   5   2  88   0   0| 316k    0 |   0     0 |   0     0 | 391   553 |  2   1   0| 929M  310M  129M   12M|   0     0 | 316k    0 | 391   553 >
  3   1  94   2   0   0:  4   5   2  87   1   1| 312k    0 |   0     0 |   0     0 | 411   659 |  3   1   0| 930M  311M  128M   12M|   0     0 | 312k    0 | 411   659 >
  3   2  88   7   0   0:  4   4   3  88   0   0| 260k    0 |   0     0 |   0     0 | 379   530 |  1   1   0| 931M  311M  126M   12M|   0     0 | 260k    0 | 378   530 >
  8   1  88   3   0   0:  1   4   3  92   0   0| 276k 1300k|   0     0 |   0     0 | 409   622 |  1   2   0| 931M  311M  126M   11M|   0     0 | 276k 1300k| 410   624 >
  8   0  92   0   0   0:  2   4   0  92   0   1| 244k 4096B|   0     0 |   0     0 | 367   547 |  2   1   0| 932M  312M  124M   12M|   0     0 | 244k 4096B| 367   545 >
  2   2  92   4   0   0:  6   4   2  88   0   0| 248k    0 |   0     0 |   0     0 | 393   600 |  1   1   0| 933M  312M  123M   12M|   0     0 | 252k    0 | 395   600 >
 10   1  89   0   0   0:  4   4   2  90   0   0| 392k    0 |   0     0 |   0     0 | 469  1121 |  4   0   0| 933M  312M  123M   11M|   0     0 | 388k    0 | 468  1122 >
 20   0  80   0   0   0: 17   4  50  27   0   2| 108k 2068k| 407B  435B|   0     0 | 595  1090 |  1   0   0| 934M  312M  122M   12M|   0     0 | 108k 2068k| 593  1089 >

<-- this is the point where updatedb stopped running and the process is not visible in top, htop or ps anymore -->
<-- as you see the memory is still being used by something!! when updatedb started only 150MB of ram was used, after it has finished it's 934MB -->

 10   0  90   0   0   0: 19   0  80   0   0   1|   0    60k|   0     0 |   0     0 | 327   970 |  1   1   0| 934M  312M  122M   12M|   0     0 |   0    60k| 328   970 >
  0   0 100   0   0   0: 10   0  84   4   1   0|   0   828k|  66B   60B|   0     0 | 397  1218 |  1   0   0| 934M  312M  122M   12M|   0     0 |   0   828k| 397  1218 >
  1   1  98   0   0   0:  2   0  98   0   0   0|   0     0 |   0     0 |   0     0 | 313   410 |  1   0   0| 934M  312M  122M   12M|   0     0 |   0     0 | 313   413 >
  7   1  92   0   0   0:  2   1  97   0   0   0|   0     0 |   0     0 |   0     0 | 332   480 |  2   0   0| 934M  312M  122M   12M|   0     0 |   0     0 | 333   477 >
  8   1  91   0   0   0:  1   0  24  72   1   2|   0  2944k|  60B   60B|   0     0 | 624   875 |  4   1   0| 934M  312M  122M   12M|   0     0 |   0  2944k| 623   877 >
  7   1  92   0   0   0:  1   1   0  97   0   1|   0  1468k|  60B    0 |   0     0 | 618   973 |  2   2   0| 934M  312M  122M   12M|   0     0 |   0  1512k| 619   971 >
  9   1  66  24   0   0:  2   1  25  69   1   1|8192B 1632k|   0     0 |   0     0 | 589  1239 |  2   0   0| 934M  312M  122M   12M|   0     0 |8192B 1588k| 588  1243 >
  4   0  96   0   0   0:  3   1  96   0   0   0|   0     0 |   0     0 |   0     0 | 331   538 |  1   0   0| 934M  312M  122M   12M|   0     0 |   0     0 | 332   535 >
  6   0  94   0   0   0:  3   0  97   0   0   0|   0     0 |   0     0 |   0     0 | 312   483 |  3   0   0| 934M  312M  122M   12M|   0     0 |   0     0 | 311   482 >
  4   0  96   0   0   0:  1   0  99   0   0   0|   0     0 |   0     0 |   0     0 | 332   523 |  1   0   0| 934M  312M  122M   12M|   0     0 |   0     0 | 331   523 >


As you see, the memory is still being reserved by "something". Running "ps aux | grep updatedb" shows:
Code:
bitinfo-dnalap ~ # ps aux | grep updatedb
root      9193  0.0  0.0   3024   732 pts/0    S+   09:55   0:00 grep updatedb

Here's the full ps aux output:
Code:
bitinfo-dnalap ~ # ps aux               
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0   2672   608 ?        Ss   09:06   0:01 init [3] 
root         2  0.0  0.0      0     0 ?        S    09:06   0:00 [migration/0]
root         3  0.0  0.0      0     0 ?        SN   09:06   0:00 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S    09:06   0:00 [watchdog/0]
root         5  0.0  0.0      0     0 ?        S    09:06   0:00 [migration/1]
root         6  0.0  0.0      0     0 ?        SN   09:06   0:00 [ksoftirqd/1]
root         7  0.0  0.0      0     0 ?        S    09:06   0:00 [watchdog/1]
root         8  0.0  0.0      0     0 ?        S<   09:06   0:00 [events/0]
root         9  0.0  0.0      0     0 ?        S<   09:06   0:00 [events/1]
root        10  0.0  0.0      0     0 ?        S<   09:06   0:00 [khelper]
root        11  0.0  0.0      0     0 ?        S<   09:06   0:00 [kthread]
root        60  0.0  0.0      0     0 ?        S<   09:06   0:00 [kblockd/0]
root        61  0.0  0.0      0     0 ?        S<   09:06   0:00 [kblockd/1]
root        62  0.0  0.0      0     0 ?        S<   09:06   0:00 [kacpid]
root       170  0.0  0.0      0     0 ?        S<   09:06   0:00 [kseriod]
root       171  0.0  0.0      0     0 ?        S<   09:06   0:00 [ata/0]
root       172  0.0  0.0      0     0 ?        S<   09:06   0:00 [ata/1]
root       173  0.0  0.0      0     0 ?        S<   09:06   0:00 [ata_aux]
root       174  0.0  0.0      0     0 ?        S<   09:06   0:00 [ksuspend_usbd]
root       177  0.0  0.0      0     0 ?        S<   09:06   0:00 [khubd]
root       212  0.0  0.0      0     0 ?        S    09:06   0:00 [pdflush]
root       213  0.0  0.0      0     0 ?        S    09:06   0:00 [pdflush]
root       214  0.0  0.0      0     0 ?        S<   09:06   0:00 [kswapd0]
root       215  0.0  0.0      0     0 ?        S<   09:06   0:00 [aio/0]
root       216  0.0  0.0      0     0 ?        S<   09:06   0:00 [aio/1]
root       348  0.0  0.0      0     0 ?        S<   09:06   0:00 [kpsmoused]
root       368  0.0  0.0      0     0 ?        S<   09:06   0:00 [scsi_eh_0]
root       369  0.0  0.0      0     0 ?        S<   09:06   0:00 [scsi_eh_1]
root       383  0.0  0.0      0     0 ?        S<   09:06   0:00 [exec-osm/0]
root       384  0.0  0.0      0     0 ?        S<   09:06   0:00 [exec-osm/1]
root       391  0.0  0.0      0     0 ?        S<   09:06   0:00 [kmmcd]
root      1347  0.0  0.0      0     0 ?        S<   09:06   0:00 [kmirrord]
root      2618  0.0  0.0      0     0 ?        S<   09:06   0:00 [kjournald]
root      2728  0.0  0.0   7128   744 ?        S<s  09:06   0:00 /sbin/udevd --daemon
root      3404  0.0  0.0      0     0 ?        S<   09:06   0:00 [hda_codec]
root      6423  0.0  0.0   4472   812 ?        Ss   09:06   0:00 /sbin/dhclient -e PEER_ROUTERS=yes -e PEER_DNS=yes -e PEER_NTP=yes -q -1 -pf /var/run/dhclient-eth0.pid eth0
root      7264  0.0  0.0   7184   696 ?        Ss   09:06   0:00 /usr/sbin/syslog-ng
root      7395  0.0  0.1  54220  2072 ?        Ss   09:06   0:00 /usr/bin/gdm
root      7397  0.0  0.2  62952  3056 ?        S    09:06   0:00 /usr/bin/gdm
root      7400  8.6  9.5 171384 134276 ?       RL   09:06   4:16 /usr/bin/Xgl :1 :1 -ac -accel glx:pbuffer -accel xv:pbuffer -auth /var/gdm/:1.Xauth -nolisten tcp vt7
root      7401  0.1  0.8  61156 11932 tty7     SLs+ 09:06   0:05 /usr/bin/Xorg vt7 -auth /tmp/.Xgl-auth-7XYLB5 -nolisten tcp :94 -terminate
root      7463  0.0  0.0   2660   568 ?        S<s  09:06   0:01 /usr/sbin/cpudynd -i 1 -p 0.5 0.9 -l 7 -h /dev/hda,/dev/sda
root      7521  0.0  0.0  10740   948 ?        Ss   09:06   0:00 /usr/sbin/xinetd -pidfile /var/run/xinetd.pid -stayalive -reuse
root      7617  0.0  0.0   1324   156 ?        S    09:06   0:00 /opt/vmware/server/bin/vmnet-bridge -d /var/run/vmnet-bridge-0.pid /dev/vmnet0 eth0
root      7677  0.0  0.1  30452  2564 ?        Ss   09:06   0:00 /usr/sbin/cupsd
101       7735  0.0  0.0   8408   500 ?        Ss   09:07   0:00 /usr/bin/dbus-daemon --system
stunnel   7792  0.0  0.0  17016  1188 ?        Ss   09:07   0:00 /usr/sbin/stunnel /etc/stunnel/stunnel.conf
root      7848  0.0  0.0   9336   732 ?        Ss   09:07   0:00 /usr/sbin/cron
root      7912  0.0  0.0   3728   692 tty1     Ss+  09:07   0:00 /sbin/agetty 38400 tty1 linux
root      7913  0.0  0.0   3732   700 tty2     Ss+  09:07   0:00 /sbin/agetty 38400 tty2 linux
root      7914  0.0  0.0   3732   696 tty3     Ss+  09:07   0:00 /sbin/agetty 38400 tty3 linux
root      7915  0.0  0.0   3732   696 tty4     Ss+  09:07   0:00 /sbin/agetty 38400 tty4 linux
root      7916  0.0  0.0   3728   692 tty5     Ss+  09:07   0:00 /sbin/agetty 38400 tty5 linux
root      7917  0.0  0.0   3728   696 tty6     Ss+  09:07   0:00 /sbin/agetty 38400 tty6 linux
moros     7935  0.0  0.1   6480  1432 ?        Ss   09:07   0:00 sh /etc/xdg/xfce4/xinitrc
moros     7962  0.0  0.0  10068   732 ?        S    09:07   0:00 /usr/bin/dbus-launch --sh-syntax --exit-with-session
moros     7963  0.0  0.0   8408   920 ?        Ss   09:07   0:00 /usr/bin/dbus-daemon --fork --print-pid 4 --print-address 6 --session
moros     7966  0.0  0.0  15484   840 ?        Ss   09:07   0:00 /usr/bin/ssh-agent -- startxfce4
moros     7968  0.0  0.6  64132  9216 ?        S    09:07   0:02 emerald
moros     7969  0.0  0.6 159576  9828 ?        Ssl  09:07   0:00 beryl-manager
moros     7977  0.0  0.0   6480   588 ?        S    09:07   0:00 sh /etc/xdg/xfce4/xinitrc
moros     7980  0.0  0.1  26932  2280 ?        S    09:07   0:00 xscreensaver -no-splash
moros     7981  0.0  0.9  74228 12808 ?        S    09:07   0:00 /usr/bin/xfce4-session
moros     8040  0.4  1.0  98608 14972 ?        S    09:07   0:12 beryl-xgl
moros     8048  0.0  0.4 108696  6524 ?        Ss   09:07   0:00 xfce-mcs-manager
moros     8053  0.4  1.1 109432 16140 ?        S    09:07   0:12 xfce4-panel
moros     8055  0.0  0.6  63268  8772 ?        S    09:07   0:01 xftaskbar4
moros     8057  0.0  1.1  93688 16360 ?        S    09:07   0:01 xfdesktop
moros     8089  0.0  0.1   6616  1688 ?        S    09:13   0:00 /bin/bash /usr/libexec/mozilla-launcher
moros     8098  5.8  7.1 428240 100968 ?       Rl   09:14   2:28 /usr/lib64/mozilla-firefox/firefox-bin
moros     8107  0.0  0.2  21656  3164 ?        S    09:14   0:00 /usr/libexec/gconfd-2 12
moros     8156  0.0  0.4  37720  6484 ?        S    09:16   0:01 xterm -title Terminal
moros     8158  0.0  0.1  10916  2032 pts/0    Ss   09:16   0:00 bash
root      8162  0.0  0.1  11664  2112 pts/0    S    09:16   0:00 -bash
moros     8929  0.0  0.3  36128  4820 ?        S    09:37   0:00 xterm -title Terminal
moros     8931  0.0  0.1  10912  2028 pts/1    Ss   09:37   0:00 bash
root      9167  0.0  0.1  11584  1976 pts/1    S+   09:45   0:00 -bash
root      9194  0.0  0.0   8648   996 pts/0    R+   09:56   0:00 ps aux


Having that, should I submit a bug report? If so, where should I submit: gentoo or someone from non-gentoo community responsible for slocate?

Thanks in advance,
DNA DesigN


Last edited by dnadesign on Wed Feb 28, 2007 9:06 am; edited 1 time in total
Back to top
View user's profile Send private message
moocha
Watchman
Watchman


Joined: 21 Oct 2003
Posts: 5722

PostPosted: Sun Feb 25, 2007 10:32 am    Post subject: Reply with quote

I fail to see any leak there. There are two possibilities:
  1. If indeed there is memory allocated and never freed after updatedb has finished, then that's a bug in the kernel. However, again, I don't see any leak in what you posted - the dfstat output you posted shows no significant process-allocated memory increase at the point you marked. This possibility is highly unlikely.
  2. You got confused by the (addmittedly easy to misunderstand) terms used in kernel space - I'm referring to "free", "buffers" and "cache" memory. Please see here: http://gentoo-wiki.com/FAQ_Linux_Memory_Management. I'm willing to bet this is the cause of your mysterious leak - the simple fact that Linux actually makes use of memory for disk caching and doesn't leave it lying around unused, and of course updatedb fills up the cache because it walks the entire file system hierarchy.

_________________
Military Commissions Act of 2006: http://tinyurl.com/jrcto

"Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety."
-- attributed to Benjamin Franklin
Back to top
View user's profile Send private message
dnadesign
Apprentice
Apprentice


Joined: 23 Dec 2006
Posts: 172
Location: Poland

PostPosted: Sun Feb 25, 2007 12:35 pm    Post subject: Reply with quote

It's true it can't be seen in the dstat output, because it's just the end of the output that I posted (the original is long as hell, that's why I posted only the ending). At the beginning, before executing updatedb the _used column showed only 150MB of usage. After the updatedb process started the memory usage began to rise and at the point I marked the process finished it's execution, but the memory allocated by it was not freed.
Back to top
View user's profile Send private message
devsk
Advocate
Advocate


Joined: 24 Oct 2003
Posts: 3003
Location: Bay Area, CA

PostPosted: Tue Feb 27, 2007 5:07 pm    Post subject: Reply with quote

short answer: kernel has buffered whatever updatedb plowed thru on disk. it will release it lazily if and when needed.
long answer: read the link moocha pointed to in 2.
Back to top
View user's profile Send private message
i92guboj
Bodhisattva
Bodhisattva


Joined: 30 Nov 2004
Posts: 10315
Location: Córdoba (Spain)

PostPosted: Tue Feb 27, 2007 5:32 pm    Post subject: Reply with quote

++ Read up on the differences between used, free, buffered and cached memory. There is no leak.
Back to top
View user's profile Send private message
dnadesign
Apprentice
Apprentice


Joined: 23 Dec 2006
Posts: 172
Location: Poland

PostPosted: Tue Feb 27, 2007 8:05 pm    Post subject: Reply with quote

Well ok. If you say so. I'll just wait for the next updatedb (too lazy to run it manually :P ) to execute and after seeing my RAM go to hell I'll just run Eclipse and BEA WebLogic with some of my currently developed projects (they involve clustering :twisted: ). If you're right then everything should return to normal levels (because these two apps in addition to taking my full RAM capacity - Java... it's normal - take an additional 512MB of swap). Ahh... the simple pleasures of running a public sector clustered app on a laptop (development purposes). :D

BTW: I already read the HowTO, so no need telling me this 3 times guys. :P

BTW2: So the kernel cache memory isn't displayed in dstat? Only in "free"?
Back to top
View user's profile Send private message
i92guboj
Bodhisattva
Bodhisattva


Joined: 30 Nov 2004
Posts: 10315
Location: Córdoba (Spain)

PostPosted: Tue Feb 27, 2007 9:01 pm    Post subject: Re: updatedb memory leaks!! Reply with quote

dnadesign wrote:
Hi all,
I noticed a very concerning fact about updatedb (presumably a part of the slocate package?). It's configurated as a background job that goes around once per day. What a did notice is that the process consumes most of my RAM memory (I have 1,5 GB in my notebook) and after finishing the execution the memory is not freed!! Here is the output of dstat -afv:


If there was a memory leak in this circunstance, and the memory is not freed as you say, you could not even start a java vm, not to talk about BEA Weblogic... The kernel would kill one or the other randomly.

As you say in your latest post, test it. The ram should be freed in the measure that it is needed. Of course, unless it is needed for other execution, caching of buffering purposes, most of them will continue to be marked as cached/buffered unless it is really needed. Note that starting a java vm will not magically redime the kernel of the rest of its memory and i/o administration tasks, so unless the memory is needed for any other thing it will continue that way.

If that is not what happen, then you have a base to claim a memory leak, but it is not very likely to find such a thing into updatedb... not impossible, though.
Back to top
View user's profile Send private message
moocha
Watchman
Watchman


Joined: 21 Oct 2003
Posts: 5722

PostPosted: Tue Feb 27, 2007 10:50 pm    Post subject: Re: updatedb memory leaks!! Reply with quote

6thpink wrote:
If that is not what happen, then you have a base to claim a memory leak, but it is not very likely to find such a thing into updatedb... not impossible, though.
Again, the probability that the behavior described is a "memory leak in updatedb" is exactly zero, nada and zilch.
The kernel rescinds any and all memory allocated by a process via the malloc family or via directly adjusting brk/sbrk the moment that process dies. If the kernel should fail to do that, it's a bug in the kernel. Unlikely, since a bug with effects of this magnitude would in all likelihood been encountered before - at the scale of Linux deployments, a one in a million chance means "next Friday".
A process could conceivably trigger a memory leak in the kernel by asking it to allocate kernel-side memory via some syscall or the other, but such resources are also freed when the process dies. If they're not, it's a bug in the kernel. Again unlikely for the abovementioned reason.
The third way by which a process could eat up memory post-mortem is to indirectly cause a third process to leak memory if that third process has registered itself as a callback from the kernel somehow triggered by updatedb (an example would be a fuse module like, say, ntfs-3g allocating some memory because the kernel-side fuse module asked it to because updatedb walked the file system controlled by ntfs-3g), or if our trigger process (updatedb) directly communicates with the leaker (e.g., famd or gamin). This would be a bug in the third process and, again, not a bug in updatedb. This intuitively sounds more likely but there is not enough data to make an asessment.
I wish people would stop throwing around terms like "memory leak" without first being reasonably sure those terms apply to the problem (although, in all fairness, I'll never forget Mr. Chen's article about how A cache with a bad policy is another name for a memory leak.)
_________________
Military Commissions Act of 2006: http://tinyurl.com/jrcto

"Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety."
-- attributed to Benjamin Franklin
Back to top
View user's profile Send private message
dnadesign
Apprentice
Apprentice


Joined: 23 Dec 2006
Posts: 172
Location: Poland

PostPosted: Wed Feb 28, 2007 9:06 am    Post subject: [FALSE ALARM] updatedb memory leaks!! Reply with quote

No. Nothing like that. Just a simple execution of updatedb on local ext2/ext3 file systems. No VFAT (no "Windows Partitions Of Shame" on my laptop ;P) or NTFS (keep them unmounted, for use only with WinXP when I --really-- need it). :P Well then, I'll have to brush up on my Linux knowledge. I never went that far into it, I know what I needed to to operate this OS (config, application compilation and instalation, kernel recompilation and some tuning and few other things that I don't remember at the moment).

Thanks for the help. :)
Back to top
View user's profile Send private message
dnadesign
Apprentice
Apprentice


Joined: 23 Dec 2006
Posts: 172
Location: Poland

PostPosted: Fri Mar 02, 2007 10:16 am    Post subject: Reply with quote

Ok, it's officially verified that... you're right. :) As expected, the memory usage normalized it self once I launched Eclipse and BEA WL after updatedb was running. For the first time I saw the RAM memory usage actually falling when starting a cluster node (kernel was freeing it's memory faster than Java was allocating it). :lol:
Anyway everything works fine. Thanks again. :)
Back to top
View user's profile Send private message
moocha
Watchman
Watchman


Joined: 21 Oct 2003
Posts: 5722

PostPosted: Fri Mar 02, 2007 2:44 pm    Post subject: Reply with quote

Anytime, and have fun :)
_________________
Military Commissions Act of 2006: http://tinyurl.com/jrcto

"Those who would give up essential liberty to purchase a little temporary safety deserve neither liberty nor safety."
-- attributed to Benjamin Franklin
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Other Things Gentoo All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum