| View previous topic :: View next topic |
| Author |
Message |
tnt
Veteran
Joined: 27 Feb 2004
Posts: 1222
|
 Posted: Sun May 29, 2005 1:45 pm Post subject: [SOLVED] dctc hangs with 100% system cpu usage |
 |
|
dctc-0.85.9 worked fine for 24.5 days on my local file server.
then, suddenly, without any package update or dctc recompilation, it started to use 100% of cpu time (cpu "system" time) and hang until 'killall -9 dctc'.
i've tried to run it as other user, to delete ~/.dctc directory before starting, to share only 2 files (to be sure it's not share-size-related), but still no success.
i've even reemerged dctc, and nothing changed... 
only one thing (as far as I can remember) changed - my server's uptime is over 24.5 days:
| Code: | | 15:41:41 up 27 days, 8:28, 2 users, load average: 0.20, 0.29, 0.46 |
right now.
so, the only thing I can think of is that dctc have some conter that checks or depends on uptime and that that counter overflowed.
unfortunately, I can't reboot server right now to see if I'm right about uptime
any similar expiriance?
_________________
gentoo user
Last edited by tnt on Wed Jun 08, 2005 12:23 am; edited 1 time in total |
|
| Back to top |
|
 |
tnt
Veteran
Joined: 27 Feb 2004
Posts: 1222
|
| Posted: Mon May 30, 2005 2:09 pm Post subject: |
|
|
Is there any other program that could replace dctc for my file server (no UI, just command line starting from cron script)?
_________________
gentoo user |
|
| Back to top |
|
|
tnt
Veteran
Joined: 27 Feb 2004
Posts: 1222
|
| Posted: Sat Jun 04, 2005 2:18 am Post subject: |
|
|
Anyone?
Any replacemant for DCTC ???
_________________
gentoo user |
|
| Back to top |
|
|
Corona688
Veteran
Joined: 10 Jan 2004
Posts: 1204
|
| Posted: Sat Jun 04, 2005 5:50 am Post subject: |
|
|
'strace dctc' might be enlightening. Also, have you tried rebooting? I'd almost never advocate this but it could rule out the obvious.
_________________
Petition for Better 64-bit ATI Drivers - Sign Here
http://www.petitiononline.com/atipet/petition.html |
|
| Back to top |
|
|
tnt
Veteran
Joined: 27 Feb 2004
Posts: 1222
|
| Posted: Sat Jun 04, 2005 11:10 am Post subject: |
|
|
Well, I'm going to avoid rebooting as long as I can. It's because (if I'm right about uptime wirdness) I will not be able to reproduce and solve problem for next 25 days if I do reboot.
I'll try strace and report what I've got.
_________________
gentoo user |
|
| Back to top |
|
|
tnt
Veteran
Joined: 27 Feb 2004
Posts: 1222
|
| Posted: Sun Jun 05, 2005 12:00 am Post subject: |
|
|
So, I've run:
| Code: | | strace -o /tmp/strace.log dctc -n x.b44 --info="proba" -a 10.0.2.9 -p 5444 -g 10.0.1.33 -t -s /tmp/Solarijum/ |
and in few seconds I've got 200MB log file /tmp/strace.log !!!
I've killed dctc and here's what's in /tmp/strace.log (just the beginning - not 200MB  ):
| Code: | execve("/usr/bin/dctc", ["dctc", "-n", "x.b44", "--info=proba", "-a", "10.0.2.9", "-p", "5444", "-g", "10.0.1.33", "-t", "-s", "/tmp/Solar
ijum/"], [/* 33 vars */]) = 0
uname({sys="Linux", node="titan", ...}) = 0
brk(0) = 0x53c000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaaaaac0000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/x86_64/libgthread-2.0.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib/tls/x86_64", 0x7fffffffea70) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/libgthread-2.0.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib/tls", 0x7fffffffea70) = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64/libgthread-2.0.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib/x86_64", 0x7fffffffea70) = -1 ENOENT (No such file or directory)
open("/usr/lib/libgthread-2.0.so.0", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \31\0\0"..., 640) = 640
fstat(3, {st_mode=S_IFREG|0755, st_size=17912, ...}) = 0
mmap(NULL, 1064328, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2aaaaabc1000
mprotect(0x2aaaaabc5000, 1047944, PROT_NONE) = 0
mmap(0x2aaaaacc1000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x2aaaaacc1000
close(3) = 0
open("/usr/lib/libpthread.so.0", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=26669, ...}) = 0
mmap(NULL, 26669, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2aaaaacc5000
close(3) = 0
open("/lib/libpthread.so.0", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`V\0\0\0"..., 640) = 640
lseek(3, 624, SEEK_SET) = 624
read(3, "\4\0\0\0\20\0\0\0\1\0\0\0GNU\0\0\0\0\0\2\0\0\0\6\0\0\0"..., 32) = 32
fstat(3, {st_mode=S_IFREG|0755, st_size=135668, ...}) = 0
mmap(NULL, 1131384, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2aaaaaccc000
mprotect(0x2aaaaacdc000, 1065848, PROT_NONE) = 0
mmap(0x2aaaaadcc000, 69632, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x2aaaaadcc000
mmap(0x2aaaaaddd000, 13176, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2aaaaaddd000
close(3) = 0
open("/usr/lib/libglib-2.0.so.0", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\337"..., 640) = 640
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaaaade1000
fstat(3, {st_mode=S_IFREG|0755, st_size=567736, ...}) = 0
mmap(NULL, 1612208, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2aaaaade2000
mprotect(0x2aaaaae6b000, 1051056, PROT_NONE) = 0
mmap(0x2aaaaaee2000, 561152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x2aaaaaee2000
mmap(0x2aaaaaf6b000, 2480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2aaaaaf6b000
close(3) = 0
open("/usr/lib/libdb-4.1.so", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\300"..., 640) = 640
fstat(3, {st_mode=S_IFREG|0755, st_size=829856, ...}) = 0
mmap(NULL, 1864408, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2aaaaaf6c000
mprotect(0x2aaaab031000, 1057496, PROT_NONE) = 0
mmap(0x2aaaab06c000, 815104, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x2aaaab06c000
mmap(0x2aaaab133000, 728, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2aaaab133000
close(3) = 0
open("/usr/lib/libgcrypt.so.11", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`b\0\0\0"..., 640) = 640
fstat(3, {st_mode=S_IFREG|0755, st_size=311056, ...}) = 0
mmap(NULL, 1353264, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2aaaab134000
mprotect(0x2aaaab179000, 1070640, PROT_NONE) = 0
mmap(0x2aaaab234000, 307200, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x2aaaab234000
close(3) = 0
open("/usr/lib/libgpg-error.so.0", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\t\0\0"..., 640) = 640
fstat(3, {st_mode=S_IFREG|0755, st_size=12856, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaaab27f000
mmap(NULL, 1059040, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2aaaab280000
mprotect(0x2aaaab283000, 1046752, PROT_NONE) = 0
mmap(0x2aaaab380000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x2aaaab380000
close(3) = 0
open("/usr/lib/libnsl.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/lib/libnsl.so.1", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360G\0\0"..., 640) = 640
lseek(3, 624, SEEK_SET) = 624
read(3, "\4\0\0\0\20\0\0\0\1\0\0\0GNU\0\0\0\0\0\2\0\0\0\6\0\0\0"..., 32) = 32
fstat(3, {st_mode=S_IFREG|0755, st_size=86152, ...}) = 0
mmap(NULL, 1137416, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2aaaab383000
mprotect(0x2aaaab396000, 1059592, PROT_NONE) = 0
mmap(0x2aaaab483000, 81920, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x2aaaab483000
mmap(0x2aaaab497000, 6920, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2aaaab497000
close(3) = 0
open("/usr/lib/libz.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/lib/libz.so.1", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300\32\0"..., 640) = 640
fstat(3, {st_mode=S_IFREG|0755, st_size=78080, ...}) = 0
mmap(NULL, 1123592, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2aaaab499000
mprotect(0x2aaaab4ab000, 1049864, PROT_NONE) = 0
mmap(0x2aaaab599000, 77824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x2aaaab599000
close(3) = 0
open("/usr/lib/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/lib/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\310\1\0"..., 640) = 640
lseek(3, 64, SEEK_SET) = 64
read(3, "\6\0\0\0\5\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0"..., 616) = 616
lseek(3, 680, SEEK_SET) = 680
read(3, "\4\0\0\0\20\0\0\0\1\0\0\0GNU\0\0\0\0\0\2\0\0\0\6\0\0\0"..., 32) = 32
fstat(3, {st_mode=S_IFREG|0755, st_size=1289568, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaaab5ac000
lseek(3, 64, SEEK_SET) = 64
read(3, "\6\0\0\0\5\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0"..., 616) = 616
mmap(NULL, 2248808, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2aaaab5ad000
mprotect(0x2aaaab6c8000, 1089640, PROT_NONE) = 0
mmap(0x2aaaab7ad000, 135168, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x100000) = 0x2aaaab7ad000
mmap(0x2aaaab7ce000, 16488, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2aaaab7ce000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaaab7d3000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaaab7d4000
mprotect(0x2aaaab7c8000, 12288, PROT_READ) = 0
mprotect(0x2aaaaaddb000, 4096, PROT_READ) = 0
arch_prctl(ARCH_SET_FS, 0x2aaaab7d3ae0) = 0
munmap(0x2aaaaacc5000, 26669) = 0
set_tid_address(0x2aaaab7d3b70) = 29399
rt_sigaction(SIGRTMIN, {0x2aaaaacd1210, [], SA_RESTORER|SA_SIGINFO, 0x2aaaaacd8210}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x2aaaaacd1290, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x2aaaaacd8210}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
_sysctl({{CTL_KERN, KERN_VERSION}, 2, 0x7ffffffff180, 31, (nil), 0}) = 0
open("/dev/urandom", O_RDONLY) = 3
read(3, "$\212]i\1\320\362C", 8) = 8
close(3) = 0
brk(0) = 0x53c000
brk(0x55d000) = 0x55d000
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 6), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaaaacc5000
write(1, "INFO ] \"\"Direct Connect Text Cli"..., 45) = 45
write(1, "hubip: 10.0.1.33 hubport: 411\n", 30) = 30
write(2, "Using 10.0.2.9 as localhost IP.\n", 32) = 32
sched_getparam(29399, { 0 }) = 0
sched_getscheduler(29399) = 0 (SCHED_OTHER)
sched_get_priority_min(SCHED_OTHER) = 0
sched_get_priority_max(SCHED_OTHER) = 0
sched_get_priority_max(SCHED_OTHER) = 0
open("/usr/lib/charset.alias", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/locale.alias", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2528, ...}) = 0
mmap(NULL, 131072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2aaaab7d5000
read(3, "# Locale name alias data base.\n#"..., 131072) = 2528
read(3, "", 131072) = 0
close(3) = 0
munmap(0x2aaaab7d5000, 131072) = 0
rt_sigaction(SIGPIPE, {SIG_IGN}, NULL, 8) = 0
rt_sigaction(SIGHUP, {SIG_IGN}, NULL, 8) = 0
rt_sigaction(SIGCHLD, {SIG_IGN}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [HUP PIPE CHLD], NULL, 8) = 0
rt_sigaction(SIGINT, {0x412140, [HUP PIPE CHLD], SA_RESTORER|SA_RESTART, 0x2aaaaacd8210}, NULL, 8) = 0
rt_sigaction(SIGQUIT, {0x412140, [HUP PIPE CHLD], SA_RESTORER|SA_RESTART, 0x2aaaaacd8210}, NULL, 8) = 0
rt_sigaction(SIGTERM, {0x412140, [HUP PIPE CHLD], SA_RESTORER|SA_RESTART, 0x2aaaaacd8210}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [HUP PIPE CHLD], NULL, 8) = 0
open("/dev/null", O_WRONLY) = 3
dup2(3, 0) = 0
dup2(3, 1) = 1
dup2(3, 2) = 2
close(3) = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(411), sin_addr=inet_addr("10.0.1.33")}, 16) = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 4
setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [8589934593], 4) = 0
bind(4, {sa_family=AF_INET, sin_port=htons(5444), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
listen(4, 64) = 0
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 5
setsockopt(5, SOL_SOCKET, SO_REUSEADDR, [17179869185], 4) = 0
bind(5, {sa_family=AF_INET, sin_port=htons(5444), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
getsockname(5, {sa_family=AF_INET, sin_port=htons(5444), sin_addr=inet_addr("0.0.0.0")}, [4294967312]) = 0
listen(4, 64) = 0
stat("/root/.dctc", {st_mode=S_IFDIR|0755, st_size=248, ...}) = 0
access("/root/.dctc", R_OK|W_OK|X_OK) = 0
stat("/root/.dctc/running", {st_mode=S_IFDIR|0755, st_size=184, ...}) = 0
access("/root/.dctc/running", R_OK|W_OK|X_OK) = 0
open("/root/.dctc/md_db", O_RDWR|O_CREAT, 0666) = 6
flock(6, LOCK_EX|LOCK_NB) = 0
fstat(6, {st_mode=S_IFREG|0644, st_size=480048, ...}) = 0
flock(6, LOCK_UN) = 0
open("/root/.dctc/md_db.var", O_RDWR|O_CREAT, 0666) = 7
flock(7, LOCK_EX|LOCK_NB) = 0
fstat(7, {st_mode=S_IFREG|0644, st_size=640000, ...}) = 0
flock(7, LOCK_UN) = 0
socket(PF_FILE, SOCK_STREAM, 0) = 8
bind(8, {sa_family=AF_FILE, path="/root/.dctc/running/dctc-000072D7-10.0.1.33"}, 110) = 0
listen(8, 3) = 0
socket(PF_FILE, SOCK_DGRAM, 0) = 9
bind(9, {sa_family=AF_FILE, path="/root/.dctc/running/dctc-000072D7-10.0.1.33.udp"}, 110) = 0
open("/root/.dctc/running/dctc-000072D7-10.0.1.33.userinfo", O_RDWR|O_CREAT, 0666) = 10
flock(10, LOCK_EX|LOCK_NB) = 0
fstat(10, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
write(10, "\244", 1) = 1
flock(10, LOCK_UN) = 0
open("/root/.dctc/gts.4184", O_RDWR|O_CREAT, 0666) = 11
flock(11, LOCK_EX|LOCK_NB) = 0
fstat(11, {st_mode=S_IFREG|0644, st_size=4184, ...}) = 0
flock(11, LOCK_UN) = 0
open("/root/.dctc/uaddr_ready.184", O_RDWR|O_CREAT, 0666) = 12
flock(12, LOCK_EX|LOCK_NB) = 0
fstat(12, {st_mode=S_IFREG|0644, st_size=184, ...}) = 0
flock(12, LOCK_UN) = 0
open("/root/.dctc/uaddr_partial.104", O_RDWR|O_CREAT, 0666) = 13
flock(13, LOCK_EX|LOCK_NB) = 0
fstat(13, {st_mode=S_IFREG|0644, st_size=104, ...}) = 0
flock(13, LOCK_UN) = 0
flock(10, LOCK_EX) = 0
fstat(10, {st_mode=S_IFREG|0644, st_size=216, ...}) = 0
mmap(NULL, 216, PROT_READ|PROT_WRITE, MAP_SHARED, 10, 0) = 0x2aaaab7d5000
munmap(0x2aaaab7d5000, 216) = 0
flock(10, LOCK_UN) = 0
uname({sys="Linux", node="titan", ...}) = 0
open("/root/.dctc/titan.10", O_WRONLY|O_CREAT|O_EXCL, 0600) = -1 EEXIST (File exists)
write(1, "file exists.\n", 13) = 13
open("/root/.dctc/titan.10", O_RDWR) = 14
read(14, "", 4) = 0
close(14) = 0
write(1, "creating new sema.\n", 19) = 19
open("/root/.dctc/titan.10", O_WRONLY|O_CREAT, 0600) = 14
write(1, "creating.\n", 10) = 10
semget(1142054546, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054547, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054548, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054549, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054550, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054551, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054552, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054553, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054554, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054555, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054556, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054557, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054558, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054559, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054560, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054561, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054562, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054563, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054564, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054565, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054566, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054567, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054568, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054569, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054570, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054571, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054572, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054573, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054574, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054575, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054576, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054577, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054578, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054579, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054580, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054581, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054582, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054583, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054584, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054585, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054586, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054587, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054588, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054589, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054590, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054591, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054592, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054593, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054594, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054595, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054596, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054597, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054598, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054599, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054600, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054601, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054602, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054603, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054604, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054605, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054606, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054607, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054608, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054609, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054610, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054611, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054612, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054613, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054614, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054615, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054616, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054617, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054618, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054619, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054620, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054621, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054622, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054623, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054624, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054625, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054626, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054627, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054628, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054629, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054630, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054631, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054632, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054633, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054634, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054635, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054636, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054637, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054638, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054639, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054640, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054641, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054642, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054643, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054644, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054645, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054646, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054647, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054648, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054649, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054650, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054651, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054652, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054653, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054654, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054655, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054656, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054657, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054658, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054659, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054660, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054661, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054662, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054663, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054664, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054665, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054666, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054667, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054668, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054669, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054670, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054671, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054672, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054673, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054674, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054675, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054676, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054677, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054678, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054679, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054680, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054681, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054682, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054683, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054684, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054685, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054686, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054687, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054688, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054689, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054690, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054691, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054692, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054693, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054694, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054695, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054696, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054697, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054698, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054699, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054700, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054701, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054702, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054703, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054704, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054705, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054706, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054707, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054708, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054709, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054710, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054711, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054712, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054713, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054714, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054715, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054716, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054717, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054718, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054719, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054720, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054721, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054722, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054723, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054724, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054725, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054726, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054727, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054728, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054729, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054730, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054731, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054732, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054733, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054734, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054735, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054736, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054737, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054738, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054739, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054740, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054741, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054742, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054743, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054744, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054745, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054746, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054747, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054748, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054749, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054750, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054751, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054752, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054753, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054754, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054755, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054756, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054757, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054758, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054759, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054760, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054761, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054762, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054763, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054764, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054765, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device)
semget(1142054766, 10, IPC_CREAT|IPC_EXCL|0600) = -1 ENOSPC (No space left on device) |
After that, the last line repeats to the end of log file.
Any ideas?
_________________
gentoo user |
|
| Back to top |
|
|
mirko_3
l33t
Joined: 02 Nov 2003
Posts: 605
Location: Birreria
|
| Posted: Sun Jun 05, 2005 3:54 pm Post subject: |
|
|
Well... are any partitions full??
df -h
_________________
Non fa male! Non fa male! |
|
| Back to top |
|
|
tnt
Veteran
Joined: 27 Feb 2004
Posts: 1222
|
| Posted: Sun Jun 05, 2005 8:26 pm Post subject: |
|
|
Well, there are almost full partition, but they are not shared - I've shared just one directory - /tmp/Solarijum (see my last post):
| Code: | Filesystem Size Used Avail Use% Mounted on
/dev/sda3 4.6G 3.1G 1.6G 67% /
/dev/sda2 4.2G 479M 3.8G 12% /var/cache/squid/cache00
/dev/sdb2 4.2G 479M 3.8G 12% /var/cache/squid/cache01
/dev/sdc2 4.2G 480M 3.8G 12% /var/cache/squid/cache02
/dev/sdd2 4.2G 479M 3.8G 12% /var/cache/squid/cache03
/dev/sda6 20G 20G 598M 98% /users/dooki
/dev/sda7 60G 60G 412M 100% /users/mica
/dev/sda8 99G 35G 64G 36% /users/mare
/dev/sda9 27G 33M 27G 1% /users/jeja
/dev/sda10 9.4G 6.9G 2.5G 74% /users/ilija
/dev/sda11 9.4G 735M 8.6G 8% /users/ana
/dev/sdb5 37G 19G 18G 53% /users/sija
/dev/sdb6 88G 37G 51G 42% /users/boki
/dev/sdb7 105G 52G 54G 50% /users/zola
/dev/sdc5 60G 58G 1.8G 98% /users/marko
/dev/sdc6 4.7G 38M 4.7G 1% /users/transfer
/dev/sdc7 20G 15G 5.0G 76% /users/vlada
/dev/sdc8 138G 49G 90G 36% /users/dome
/dev/sdc9 6.7G 33M 6.6G 1% /users/dusan
/dev/sdd5 90G 76G 14G 85% /users/tnt
/dev/sdd6 93G 82G 12G 88% /users/peleizoki
/dev/sdd7 20G 8.3G 12G 42% /users/nesha
/dev/sdd8 6.7G 1.4G 5.3G 22% /users/gergana
none 502M 0 502M 0% /dev/shm
none 1.5G 318M 1.2G 21% /tmp
|
_________________
gentoo user |
|
| Back to top |
|
|
mirko_3
l33t
Joined: 02 Nov 2003
Posts: 605
Location: Birreria
|
| Posted: Sun Jun 05, 2005 9:46 pm Post subject: |
|
|
I'm sorry, it was just a shot in the dark, from what I understand of the strace, it says that there is no space in /root/.dctc, but then, I've never read an strace output...
Other than that - you're running dctc as root?!?
_________________
Non fa male! Non fa male! |
|
| Back to top |
|
|
tnt
Veteran
Joined: 27 Feb 2004
Posts: 1222
|
| Posted: Mon Jun 06, 2005 12:08 am Post subject: |
|
|
Normaly I run dctc as non-privileged user but since I've found myself in this crazy situation I've tried everything - even to run dctc as a root.
Always same result...
_________________
gentoo user |
|
| Back to top |
|
|
tnt
Veteran
Joined: 27 Feb 2004
Posts: 1222
|
| Posted: Tue Jun 07, 2005 12:56 am Post subject: |
|
|
My dctc even fails to start dctc_master proces:
| Code: | root 32692 0.0 0.2 29256 2356 ? Ss 02:54 0:00 sshd: root@pts/8
root 32697 0.0 0.1 8744 1512 pts/8 Ss 02:54 0:00 -bash
root 303 91.4 0.1 12900 1196 pts/8 R+ 02:54 0:13 dctc -n x --info=xx -a 10.0.2.9 -p 5444 -g 10.0.1.33 -t -s /tmp/loto/
root 369 0.0 0.0 6272 548 ? S 02:54 0:00 sleep 10
root 370 0.0 0.0 8688 916 pts/7 R+ 02:54 0:00 ps aux |
that normaly would appear few moments after one starts dctc...
_________________
gentoo user |
|
| Back to top |
|
|
tnt
Veteran
Joined: 27 Feb 2004
Posts: 1222
|
| Posted: Tue Jun 07, 2005 3:09 am Post subject: |
|
|
Well, this is just great! 
I've tried to compile versions 0.85.6 and even 0.83.8 and (after some tricks with BerkeleyDB) I've installed 0.85.6 and later 0.83.8. Before that I've unmerged my 0.85.9 ebuild.
But, result was the same - 100% CPU usage without getting dctc_master started. 
So, it' not version dependent bug.
P.S. Can someone please give me some 32-bit staticly compiled binary of dctc that I could run on my server? (I hope it's 64-bit OS related bug)
_________________
gentoo user |
|
| Back to top |
|
|
tnt
Veteran
Joined: 27 Feb 2004
Posts: 1222
|
|
| Back to top |
|
|
widan
Veteran
Joined: 07 Jun 2005
Posts: 1512
Location: Paris, France
|
| Posted: Tue Jun 07, 2005 4:39 pm Post subject: |
|
|
ENOSPC for semget has nothing to do with full disks. From semget's man page :
| Code: | ENOSPC A semaphore set has to be created but the system limit for
the maximum number of semaphore sets (SEMMNI), or the system
wide maximum number of semaphores (SEMMNS), would be
exceeded.
|
Could you run those:
Maybe some program (maybe dctc, maybe some other one) created semaphores but never deleted them, and the kernel table slowly filled up over time. If ipcs returns a very long list under the title "semaphore arrays", there are probably too many semaphore sets. |
|
| Back to top |
|
|
tnt
Veteran
Joined: 27 Feb 2004
Posts: 1222
|
| Posted: Tue Jun 07, 2005 4:45 pm Post subject: |
|
|
Wow! Thank you for advise!
| Code: | titan / # ipcs -ls
------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767
titan / #
|
| Code: | titan / # ipcs
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x10feed01 0 root 644 25656 4
------ Semaphore Arrays --------
key semid owner perms nsems
0x00feed00 32768 root 644 2
0x00000000 3637249 apache 600 1
0x00000000 3670018 apache 600 1
0x6eab5d4e 131075 titan 600 10
0x73692b8e 163844 titan 600 10
0x096d3adf 196613 titan 600 10
0x6c3a7223 229382 titan 600 10
0x0be0cd95 262151 titan 600 10
0x3735deb4 294920 titan 600 10
0x4a5dbae7 327689 titan 600 10
0x0dd1f3bb 360458 titan 600 10
0x22d456f7 393227 titan 600 10
0x38bcb84b 425996 titan 600 10
0x60010ebe 458765 titan 600 10
0x0b86becb 491534 titan 600 10
0x60e931d2 524303 titan 600 10
0x36a3a232 557072 titan 600 10
0x0b91d49e 589841 titan 600 10
0x6117c0a8 622610 titan 600 10
0x0c22aa5c 655379 titan 600 10
0x0fe20bf4 688148 titan 600 10
0x3785fe61 720917 titan 600 10
0x3b025a35 753686 titan 600 10
0x6277846a 786455 titan 600 10
0x0ceacca9 819224 titan 600 10
0x22844cd2 851993 titan 600 10
0x77c84297 884762 titan 600 10
0x3b906f7d 917531 titan 600 10
0x1100f450 950300 titan 600 10
0x4dc9a6c6 983069 titan 600 10
0x23b467bf 1015838 titan 600 10
0x6c333e51 1048607 titan 600 10
0x66c59b8c 1081376 titan 600 10
0x4e553ab6 1114145 titan 600 10
0x23810e17 1146914 titan 600 10
0x0f02f590 1179683 titan 600 10
0x64a2fddd 1212452 titan 600 10
0x39998e77 1245221 titan 600 10
0x4f2c2bfb 1277990 titan 600 10
0x52d51f8d 1310759 titan 600 10
0x4f92c7f7 1343528 titan 600 10
0x256611c1 1376297 titan 600 10
0x3a5b6ab4 1409066 titan 600 10
0x3e78ecf5 1441835 titan 600 10
0x65ac2fa8 1474604 titan 600 10
0x50a678d2 1507373 titan 600 10
0x25a76d59 1540142 titan 600 10
0x7b9fe085 1572911 titan 600 10
0x50c3913c 1605680 titan 600 10
0x51e54d9f 1638449 titan 600 10
0x54d9ea05 1671218 titan 600 10
0x3c7ef9fa 1703987 titan 600 10
0x528cb509 1736756 titan 600 10
0x27464464 1769525 titan 600 10
0x520d0f86 1802294 titan 600 10
0x67c62b84 1835063 titan 600 10
0x7d3d9eb9 1867832 titan 600 10
0x12f85944 1900601 titan 600 10
0x563ba1d0 1933370 titan 600 10
0x5361de35 1966139 titan 600 10
0x175f2692 1998908 titan 600 10
0x3eba763b 2031677 titan 600 10
0x2920dd9e 2064446 titan 600 10
0x5197c1bd 2097215 titan 600 10
0x2a181cf9 2129984 titan 600 10
0x3f41418d 2162753 titan 600 10
0x1548edbf 2195522 titan 600 10
0x2abb8b33 2228291 titan 600 10
0x14f38722 2261060 titan 600 10
0x2a97c547 2293829 titan 600 10
0x4028d5bc 2326598 titan 600 10
0x03ab7869 2359367 titan 600 10
0x6bef1878 2392136 titan 600 10
0x78dbad7a 2424905 titan 600 10
0x2bd4c0fe 2457674 titan 600 10
0x6ed33ac8 2490443 titan 600 10
0x164f267e 2523212 titan 600 10
0x6c167d48 2555981 titan 600 10
0x3d490107 2588750 titan 600 10
0x6c6b35df 2621519 titan 600 10
0x2feb048a 2654288 titan 600 10
0x56c0a560 2687057 titan 600 10
0x494f0af8 2719826 titan 600 10
0x25cacfb3 2752595 titan 600 10
0x22b3c15f 2785364 titan 600 10
0x37da8a3f 2818133 titan 600 10
0x0dfa2df1 2850902 titan 600 10
0x62cdfae5 2883671 titan 600 10
0x78d64081 2916440 titan 600 10
0x23f120c0 2949209 titan 600 10
0x78f822fd 2981978 titan 600 10
0x4e2f906e 3014747 titan 600 10
0x6382d3c6 3047516 titan 600 10
0x38d7c1be 3080285 titan 600 10
0x7dc60b23 3113054 titan 600 10
0x5293e65d 3145823 titan 600 10
0x7a429154 3178592 titan 600 10
0x0f451b1f 3211361 titan 600 10
0x64f9111d 3244130 titan 600 10
0x27b5e76f 3276899 titan 600 10
0x6557b205 3309668 titan 600 10
0x7a92b297 3342437 titan 600 10
0x10d89ce7 3375206 titan 600 10
0x13d50053 3407975 titan 600 10
0x3b4947eb 3539048 titan 600 10
0x65765f1d 3571817 titan 600 10
0x7ba28d61 3702890 titan 600 10
0x647cfccb 3735659 titan 600 10
0x549b7a16 3768428 titan 600 10
0x6a1c22e3 3801197 titan 600 10
0x552ed449 3833966 titan 600 10
0x3c6afe99 3866735 titan 600 10
0x4076b217 3899504 titan 600 10
0x15e70076 3932273 titan 600 10
0x7d6634c0 3965042 titan 600 10
0x682fc620 3997811 titan 600 10
0x6b64e8c7 4030580 titan 600 10
0x1383bc26 4063349 titan 600 10
0x56a5ea49 4096118 titan 600 10
0x7e73e2ff 4128887 titan 600 10
0x16d23b83 4161656 titan 600 10
0x2729074e 4194425 titan 600 10
0x41b68535 4227194 titan 600 10
0x293db85d 4259963 titan 600 10
0x6ce24989 4292732 titan 600 10
0x17bed7a0 4325501 titan 600 10
0x3fb77fab 4358270 titan 600 10
0x6a6294c3 4391039 titan 600 10
------ Message Queues --------
key msqid owner perms used-bytes messages
titan / #
|
Is this right or this is too much?
What to do about it? How to track which program made those semaphores and how to delete unused ones?
[edit]
I can see that the most of the semaphores are owned by user "titan" and that user I use only to start dctc!
So, I guess that when I | Code: | killall -9 dctc
killall -9 dctc_master |
(because I don't know how to stop dctc in other way), kernel leaves some semaphores...
How to delete them too when I kill dctc?
[/edit]
_________________
gentoo user |
|
| Back to top |
|
|
widan
Veteran
Joined: 07 Jun 2005
Posts: 1512
Location: Paris, France
|
| Posted: Tue Jun 07, 2005 5:29 pm Post subject: |
|
|
It's way too much. The default kernel configuration allows you to have 128 semaphore groups, and it looks you have exactly that number, so it is "normal" dctc can't create more.
First, you need to delete the ones that exist, to free up the kernel table entries :
| Code: | | for i in $(ipcs -s | grep titan | cut -d ' ' -f 1); do ipcrm -S $i; done |
As to why it leaks semaphores, it's "normal" if you kill dctc that way: programs killed with SIGKILL don't get a chance to clean up. This is usually not a problem, but it is for IPC objects (semaphores, shared mem space and message queues) : they are not associated to a process, so the kernel can't reclaim them. You could try to kill with SIGTERM:
| Code: | killall -TERM dctc
killall -TERM dctc_master |
It's a bit less aggressive, and dctc might handle it better. If it fails, you could use a script that does both kills and runs the code above to delete the IPCs. |
|
| Back to top |
|
|
tnt
Veteran
Joined: 27 Feb 2004
Posts: 1222
|
| Posted: Wed Jun 08, 2005 12:17 am Post subject: |
|
|
Men, you saved my life!
IT WORKS !!!
I've deleted all semaphores that were owned by 'titan' user and dctc started normaly.
I couldn't stop dctc and dctc_master by sending -TERM signal, so I had to make stop script like this:
| Code: | #!/bin/bash
killall -9 dctc 2>&1
killall -9 dctc_master 2>&1
sleep 2
for i in $(ipcs -s | grep titan | cut -d ' ' -f 1); do ipcrm -S $i; done
rm -f -r /home/titan/.dctc 2>&1
|
It works just fine for now and I have no semaphores owned by 'titan' when I turn off dctc.
User 'titan' is not used to do anything else except starting dctc, so I guess it will not be a problem to delete all his semaphores everytime I kill dctc.
Any suggestions on other way of stoping dctc are welcome.
Once more: big THANK YOU 
_________________
gentoo user |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Gentoo on AMD64
