| View previous topic :: View next topic |
| Author |
Message |
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3687
Location: Rasi, Finland
|
 Posted: Thu Aug 22, 2024 8:03 am Post subject: Something is _slowly_ eating memory, then releasing it |
 |
|
| btm "screen capture": | ┌ Memory ──────────────────────────────────────────────── Esc to go back ┐
│100% │ ┌─────────────────────────┐│
│ │ │RAM: 60% 4.0GiB/6.7GiB ││
│ │ │SWP: 9% 2.5GiB/27.0GiB││
│ │ └─────────────────────────┘│
│ │ │
│ │ ⡀ │
│ │ ⢀⣠⢴⣰⠴⠚⠓⡇ │
│ │ ⣠⠏⠁ ⠃ ⡇ ⣄ │
│ │ ⡼⠃ ⡇ ⡜⢸ ⣠ │
│ │ ⣠⢤⡼⠁ ⡇ ⡼ ⢸ ⡿⡆ │
│ │ ⣠⠃⠈ ⡇ ⡀⡼⠁ ⢸ ⡇⡇ │
│ │ ⢸ ⡰⠃ ⡇ ⡸⠿⠁ ⢸ ⡀⡼ ⡇ │
│ │ ⡀ ⢀⣿ ⣰⠃ ⡇ ⣰⠁ ⢸ ⣰⠻⠁ ⡇ │
│ │⣠⣄⠿⣀⡞⢹⡶⠁ ⡇ ⢠⠇ ⢸ ⢠⠃ ⡇ │
│ │⠃⠃ ⠻ ⡇ ⢠⠇ ⢸ ⢠⠇ ⡇ │
│ │ ⡇ ⢀⣦⠏ ⢸ ⢠⣦⠏ ⡇ │
│ │ ⡇ ⢠⠏⠉ ⢸ ⢠⠏⠋ ⡇ │
│ │ ⡇ ⢠⠏ ⢸ ⢠⠏ ⡇ │
│ │ ⡇ ⢀⡏ ⢸ ⢀⠏ ⡇ │
│ │ ⡇ ⢀⣆⡏ ⢸ ⢀⣀⡎ ⡇ ⢰ │
│ │ ⡇ ⢀⡎⠋ ⢸ ⡞⠏ ⡇ ⢸⢿ │
│ │ ⡇ ⡜ ⢸ ⢀⡼ ⡇ ⢀⠞⢸⠦│
│ │ ⡇ ⡜ ⢸⢀⡞ ⣇⣄⣠⠚ ⠘ │
│ │ ⡇⡼ ⢸⡼ ⠉⠈⠁ │
│ │ ⠿⠁ ⠘ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ ⢀⣀⣠⠤⠖⠦⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⠤⣀⣀│
│ │⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠒⠋⠉⠉ │
│ │ │
│ │ │
│ │ │
│ 0% │ │
│ └─────────────────────────────────────────────────────────────────│
│ 43200s 0s│
└────────────────────────────────────────────────────────────────────────┘ |
(sorry for bad paste, I couldn't get btm to switch to pure ascii)
That's a 12-hour long graph.
The lower one is swap usage. Nothing strange in it.
The upper graph is RAM usage. I had firefox and signal (electron based program) running over night.
So I wonder if this is normal... I suspect Firefox is making this as I have loads of tabs open in at least four windows. Does the kernel reclaim some RAM once in a while, or if this something else?
Not that this affects anything, just found this interesting.
_________________
..: Zucca :..
| My gentoo installs: | init=/sbin/openrc-init
-systemd -logind -elogind seatd |
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
 |
flexibeast
Guru
Joined: 04 Apr 2022
Posts: 440
Location: Naarm/Melbourne, Australia
|
| Posted: Fri Aug 23, 2024 1:50 am Post subject: |
|
|
FF seems like a definite candidate as the cause of this behaviour, given how FF (and Chrom[ium|e]) are such memory hogs. i use the Auto Tab Discard Web extension to try to manage this, and i've found it makes a significant difference to FF's memory usage.
My understanding is that, on Linux, once a process has been allocated memory, that memory isn't returned to the system until the process exits - any memory that the process frees is merely returned to the pool of free memory available to that particular process. i'd be happy to be corrected on this point.
There is the kernel Out-of-Memory (OOM) killer, but if that were involved, i think that would cause FF to shut down entirely, which doesn't seem to be the case? |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3687
Location: Rasi, Finland
|
| Posted: Mon Aug 26, 2024 7:41 am Post subject: |
|
|
Hi.
That sounds like a nice extension.
But does "discard" completely close the tabs? Or does it leave them open, but not loaded until activated?
_________________
..: Zucca :..
| My gentoo installs: | init=/sbin/openrc-init
-systemd -logind -elogind seatd |
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
flexibeast
Guru
Joined: 04 Apr 2022
Posts: 440
Location: Naarm/Melbourne, Australia
|
| Posted: Mon Aug 26, 2024 9:31 am Post subject: |
|
|
| The latter: open, but not loaded until activated. There are various options available to control the conditions under which tabs should be unloaded. |
|
| Back to top |
|
|
CaptainBlood
Advocate
Joined: 24 Jan 2010
Posts: 3848
|
| Posted: Mon Aug 26, 2024 10:25 am Post subject: |
|
|
| flexibeast wrote: | | i use the Auto Tab Discard Web extension to try to manage this, and i've found it makes a significant difference to FF's memory usage. |
| Zucca wrote: | Hi.
That sounds like a nice extension. |
+1
Thks 4 ur attention, interest & support.
_________________
USE="-* ..." in /etc/portage/make.conf here, i.e. a countermeasure to portage implicit braces, belt & diaper paradigm
LT: "I've been doing a passable imitation of the Fontana di Trevi, except my medium is mucus. Sooo much mucus. " |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3687
Location: Rasi, Finland
|
| Posted: Wed Aug 28, 2024 6:33 am Post subject: |
|
|
It looks like the memory usage is now stable | Code: | ┌ Memory ──────────────────────────────────────────────────────────┐
│100% │ ┌─────────────────────────┐│
│ │ │RAM: 58% 3.9GiB/6.7GiB ││
│ │ •• │SWP: 7% 1.8GiB/27.0GiB││
│ │ ••• └─────────────────────────┘│
│ │ •• •• │
│ │ •••• ••• │
│ │ ••• •• │
│ │ • │
│ │ ••••••••••••••••••••••••••••••••••••••••••••••│
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ │
│ │ ••••••••••••••••••••••••••••••••••••••••••••••••••│
│ │ •••• │
│ 0% │ │
│ └───────────────────────────────────────────────────────────│
│ 43200s 0s│
└──────────────────────────────────────────────────────────────────┘ |
After initial delay the new extension now put all the tabs in sleep and memory usage stayed at the same level the whole night. Nice.
_________________
..: Zucca :..
| My gentoo installs: | init=/sbin/openrc-init
-systemd -logind -elogind seatd |
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20484
|
| Posted: Wed Aug 28, 2024 4:27 pm Post subject: |
|
|
| Zucca wrote: | | After initial delay the new extension now put all the tabs in sleep and memory usage stayed at the same level the whole night. Nice. |
The Auto Tab Discard extension? That sounds like it kills the tab rather than "sleep".
| Quote: | | When a tab is discarded by the native method, it is completely removed from memory and does not use any resources, but it can be restored to its previous state, including preserving page state such as scroll position. |
Hmm. So is the tab itself there, or does it go away? | Quote: | | The content of a discarded tab is removed from memory and is not visible to the user. However, it is still listed in the tab bar and can be reopened by clicking on it. |
The extension seems poorly named. If I wanted the tab gone, I'd be disappointed in how this worked. I was initially not interested because I didn't want something randomly killing off my tabs. Without your having referred to it as sleep, I wouldn't have bothered to check how it worked.
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3687
Location: Rasi, Finland
|
| Posted: Wed Aug 28, 2024 4:48 pm Post subject: |
|
|
| pjp wrote: | | If I wanted the tab gone, I'd be disappointed in how this worked. I was initially not interested because I didn't want something randomly killing off my tabs. Without your having referred to it as sleep, I wouldn't have bothered to check how it worked. |
There is an option to make it completely destroy (close) tabs. | Quote: | | Permanently delete (remove) old discarded tabs if they are inactive for [XX] hours. |
| pjp wrote: | | So is the tab itself there, or does it go away? |
Tabs are prsesent and, optinally, have a "zzZ" icon on them when put to "sleep".
_________________
..: Zucca :..
| My gentoo installs: | init=/sbin/openrc-init
-systemd -logind -elogind seatd |
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20484
|
| Posted: Wed Aug 28, 2024 5:08 pm Post subject: |
|
|
I don't see it in the FAQ, but is the default to do nothing, or just sleep a tab? I don't want to install it and have it randomly destroying any tabs at some point. :)
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3687
Location: Rasi, Finland
|
| Posted: Wed Aug 28, 2024 6:02 pm Post subject: |
|
|
Default is to put tabs to sleep.
_________________
..: Zucca :..
| My gentoo installs: | init=/sbin/openrc-init
-systemd -logind -elogind seatd |
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
dmpogo
Advocate
Joined: 02 Sep 2004
Posts: 3417
Location: Canada
|
| Posted: Thu Aug 29, 2024 2:50 am Post subject: |
|
|
| flexibeast wrote: |
My understanding is that, on Linux, once a process has been allocated memory, that memory isn't returned to the system until the process exits - any memory that the process frees is merely returned to the pool of free memory available to that particular process. i'd be happy to be corrected on this point.
|
I don't believe it (which is saying that I don't not for sure  I often run my own code which allocates and deallocates a large chucks of memory, significant fraction of all available, and if I monitor its progress with top, I see immediately how free memory changes when the code deallocates something big |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3687
Location: Rasi, Finland
|
| Posted: Thu Aug 29, 2024 10:58 am Post subject: |
|
|
Well, it's not perfect or I'm somehow giving it REs in invalid format or something.
I tried to give it , but all "gentoo tabs" I have open still go to sleep.
EDIT: Ok, I figured it out. You probably need to make the RE to match the whole url, including the protocol:// part. Also you must prepend re: to it. | example: | | re:^https?://[a-z][a-z]*\.gentoo\.org/.*$ |
... I'm testing this now.
_________________
..: Zucca :..
| My gentoo installs: | init=/sbin/openrc-init
-systemd -logind -elogind seatd |
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 3687
Location: Rasi, Finland
|
| Posted: Thu Aug 29, 2024 3:22 pm Post subject: |
|
|
| Zucca wrote: | | ... I'm testing this now. |
And The conclusion is that the format I mentioned in my previous post does, indeed, work.
_________________
..: Zucca :..
| My gentoo installs: | init=/sbin/openrc-init
-systemd -logind -elogind seatd |
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
pjp
Administrator
Joined: 16 Apr 2002
Posts: 20484
|
| Posted: Fri Aug 30, 2024 2:54 am Post subject: |
|
|
@Zucca,
Thanks for the feedback. I've installed it and made some minor changes. Although I'll feel guilty if I use it for long since they're asking for payment. I don't randomly use online payments, and I won't use paypal.
_________________
Quis separabit? Quo animo? |
|
| Back to top |
|
|
flexibeast
Guru
Joined: 04 Apr 2022
Posts: 440
Location: Naarm/Melbourne, Australia
|
| Posted: Sat Aug 31, 2024 1:28 am Post subject: |
|
|
| dmpogo wrote: | | I don't believe it (which is saying that I don't not for sure I often run my own code which allocates and deallocates a large chucks of memory, significant fraction of all available, and if I monitor its progress with top, I see immediately how free memory changes when the code deallocates something big |
*nod* Interesting - thank you.
i've actually been seeking clarification on this topic for a while, because the behaviour i described in my previous post is something that i seem to recall reading about in a few places, and which matched at least some of my own experiences. But i've not yet found any concrete information about the specifics of how Linux currently works in this regard, e.g. in the "Reclaim" section of the "[Memory management] Concepts overview" page in the kernel docs, although it does at least say:
| Quote: | | Depending on the page usage it is treated differently by the Linux memory management. The pages that can be freed at any time, either because they cache the data available elsewhere, for instance, on a hard disk, or because they can be swapped out, again, to the hard disk, are called reclaimable. The most notable categories of the reclaimable pages are page cache and anonymous memory. |
So there might not be a single kernel behaviour involved. Normally i'd simply go read the source, but i suspect that i'd quickly get overwhelmed trying to do so:
| Quote: | | The memory management in Linux is a complex system that evolved over the years and included more and more functionality to support a variety of systems from MMU-less microcontrollers to supercomputers. |
Finally, the man pages for (glibc) free(3) and free(3p) don't seem to clearly mandate whether or not memory must be freed to the system rather than the process. free(3):
| Quote: | | The free() function frees the memory space pointed to by ptr, which must have been returned by a previous call to malloc() or related functions. Otherwise, or if ptr has already been freed, undefined behavior occurs. If ptr is NULL, no operation is performed. |
free(3p):
| Quote: | | The free() function shall cause the space pointed to by ptr to be deallocated; that is, made available for further allocation. If ptr is a null pointer, no action shall occur. Otherwise, if the argument does not match a pointer earlier returned by a function in POSIX.1‐2008 that allocates memory as if by malloc(), or if the space has been deallocated by a call to free() or realloc(), the behavior is undefined. |
|
|
| Back to top |
|
|
sublogic
Apprentice
Joined: 21 Mar 2022
Posts: 269
Location: Pennsylvania, USA
|
| Posted: Sat Aug 31, 2024 9:14 pm Post subject: |
|
|
From glibc's malloc(3) man page: | Code: | Normally, malloc() allocates memory from the heap, and adjusts the size
of the heap as required, using sbrk(2). When allocating blocks of mem‐
ory larger than MMAP_THRESHOLD bytes, the glibc malloc() implementation
allocates the memory as a private anonymous mapping using mmap(2).
MMAP_THRESHOLD is 128 kB by default, but is adjustable using mallopt(3).
Prior to Linux 4.7 allocations performed using mmap(2) were unaffected
by the RLIMIT_DATA resource limit; since Linux 4.7, this limit is also
enforced for allocations performed using mmap(2). |
My understanding is that "mmap" memory is returned to the kernel by free(). Freed "sbrk" memory on the other hand stays with the process (in a pool available for later malloc calls). |
|
| Back to top |
|
|
flexibeast
Guru
Joined: 04 Apr 2022
Posts: 440
Location: Naarm/Melbourne, Australia
|
| Posted: Sun Sep 01, 2024 1:23 am Post subject: |
|
|
| sublogic wrote: | | My understanding is that "mmap" memory is returned to the kernel by free(). Freed "sbrk" memory on the other hand stays with the process (in a pool available for later malloc calls). |
Ah! Nice, thank you! |
|
| Back to top |
|
|
dmpogo
Advocate
Joined: 02 Sep 2004
Posts: 3417
Location: Canada
|
| Posted: Sun Sep 01, 2024 1:56 am Post subject: |
|
|
| sublogic wrote: | From glibc's malloc(3) man page: | Code: | Normally, malloc() allocates memory from the heap, and adjusts the size
of the heap as required, using sbrk(2). When allocating blocks of mem‐
ory larger than MMAP_THRESHOLD bytes, the glibc malloc() implementation
allocates the memory as a private anonymous mapping using mmap(2).
MMAP_THRESHOLD is 128 kB by default, but is adjustable using mallopt(3).
Prior to Linux 4.7 allocations performed using mmap(2) were unaffected
by the RLIMIT_DATA resource limit; since Linux 4.7, this limit is also
enforced for allocations performed using mmap(2). |
My understanding is that "mmap" memory is returned to the kernel by free(). Freed "sbrk" memory on the other hand stays with the process (in a pool available for later malloc calls). |
One the other hand from Wikipedia article on sbrk
| Quote: |
sbrk and brk were considered legacy even by 1997 standards (Single UNIX Specification v2 or POSIX.1-1998).[5] They were removed in POSIX.1-2001.
|
this is also what man sbrk says on your system
and further
| Quote: |
Not every Unix-like system entertains the concept of having the user control the data segment. The Mac OS X implementation of sbrk is an emulation and has a maximum allocation of 4 megabytes. On first call an area exactly this large is allocated to hold the simulated segment. When this limit is reached, −1 is returned and the errno is set to ENOMEM. brk always errors.[
|
at the same time man malloc says that malloc uses sbrk for allocations below MMAP_THRESHOLD which by default on Linux is 128 kB. Above that mmap is used |
|
| Back to top |
|
|
|
Other Things Gentoo
