| View previous topic :: View next topic |
| Author |
Message |
Sargastic
n00b
Joined: 28 Aug 2011
Posts: 64
|
 Posted: Sat Jan 25, 2020 8:29 pm Post subject: rlimit hit despite ignore_rlimit_data |
 |
|
Hello,
I am running a very memory-intensive program (psteal.py, from plaso, if it may be useful).
On a 32Gb-Ram computer, after several days of work, the program dies with
| Code: | | kernel: mmap: psteal.py (26061): VmData 4295028736 exceed data ulimit 4294967296. Update limits or use boot option ignore_rlimit_data. |
So I modified grub and boot options, which are now
| Code: | | linux /vmlinuz-4.19.97-gentoo-LTS root=/dev/mapper/root ro keymap=fr crypt_root=UUID=<UUID> root=/dev/mapper/root ignore_rlimit_data |
After a reboot, ulimit seems to tell that there are no limits anymore :
| Code: |
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 127973
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 127973
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
|
Despite this, the program still dies, with
| Code: | | kernel: mmap: psteal.py (8553): VmData 4295192576 exceed data ulimit 4294967296. Update limits. |
The « use ignore_rlimit_data boot option » has disappeared, but the problem still remains. I may have misunderstood something somewhere.
Any idea as to how to proceed ?
Tia. |
|
| Back to top |
|
 |
Ionen
Developer
Joined: 06 Dec 2018
Posts: 2892
|
| Posted: Sun Jan 26, 2020 2:11 am Post subject: |
|
|
I'm not overly familiar with how rlimit works but "unlimited" means as high as it can go rather than unlimited, so it's simply the max value of unsigned integers. However it should be using 64bit ints (allowing for 16 exa bytes ram usage instead of 4GB) unless it's a 32bit python interpreter or something (is it? alternatively, maybe it's self-limiting for security and changing the limits, can't say I know how that script or python in general does things)
For ignore_rlimit_data, you can check/change the value from "/sys/module/kernel/parameters/ignore_rlimit_data", not that it's something I ever touched nor should be needed. If set it should still show the warning (minus the notice to use ignore_rlimit_data), but allow expansion of vm space anyway (if it's possible). |
|
| Back to top |
|
|
mike155
Advocate
Joined: 17 Sep 2010
Posts: 4438
Location: Frankfurt, Germany
|
| Posted: Sun Jan 26, 2020 2:46 am Post subject: |
|
|
@Sargastic: I can't tell you why your program fails - but I can explain some of your observations.
The only place where I found the 'exceed data ulimit' message is function 'may_expand_vm()' in '/usr/src/linux/mm/mmap.c'.
| Code: | /*
* Return true if the calling process may expand its vm space by the passed
* number of pages
*/
bool may_expand_vm(struct mm_struct *mm, vm_flags_t flags, unsigned long npages)
{
if (mm->total_vm + npages > rlimit(RLIMIT_AS) >> PAGE_SHIFT)
return false;
if (is_data_mapping(flags) &&
mm->data_vm + npages > rlimit(RLIMIT_DATA) >> PAGE_SHIFT) {
/* Workaround for Valgrind */
if (rlimit(RLIMIT_DATA) == 0 &&
mm->data_vm + npages <= rlimit_max(RLIMIT_DATA) >> PAGE_SHIFT)
return true;
pr_warn_once("%s (%d): VmData %lu exceed data ulimit %lu. Update limits%s.\n",
current->comm, current->pid,
(mm->data_vm + npages) << PAGE_SHIFT,
rlimit(RLIMIT_DATA),
ignore_rlimit_data ? "" : " or use boot option ignore_rlimit_data");
if (!ignore_rlimit_data)
return false;
}
return true;
} |
First of all, it's a warning message, not an error message.
The code shows why 'or use boot option ignore_rlimit_data' wasn't printed after you added 'ignore_rlimit_data' to the kernel command line. 'ignore_rlimit_data' was accepted by the kernel, and for that reason, the message wasn't printed.
So it seems that adding 'ignore_rlimit_data' to the kernel command line doesn't solve the problem. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Kernel & Hardware
