View previous topic :: View next topic |
Author |
Message |
hitman200ca Tux's lil' helper
Joined: 02 Oct 2002 Posts: 110 Location: Canada
|
Posted: Thu May 20, 2004 3:57 pm Post subject: Strange SSH / SSHD behavior [RESOLVED] |
|
|
I am running SSHD on my Gentoo box and have poked out port 22 in my firewall.
I am able to connect to it and log in as any user however after a short period
it will freeze up when running commands. For example.
Code: | # ssh ace.me.com -l root
Password: <password>
# ls
<listing>
# netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
|
And then it is unresponsive. The command started to run and I ASSUME did run
but the output didn't all arrive. This also occurs if I try to start vi or even type set
to see my env vars.
Has anybody seen this before? _________________ "Against stupidity, the Gods themselves contend in vain."
-- Friedrich von Schiller
Last edited by hitman200ca on Fri May 21, 2004 12:31 am; edited 1 time in total |
|
Back to top |
|
|
hex4def6 n00b
Joined: 13 Mar 2004 Posts: 39 Location: Pleasanton, CA
|
Posted: Thu May 20, 2004 4:12 pm Post subject: |
|
|
hmm..
Is this a reliable network connection (ie - cat5 vs wifi, etc). Perhaps the link is dropping packets and screwing up the connection.
On the idea that its a dodgy connectinon: try pinging the box and seeing the number of dropped packets. |
|
Back to top |
|
|
hitman200ca Tux's lil' helper
Joined: 02 Oct 2002 Posts: 110 Location: Canada
|
Posted: Thu May 20, 2004 4:41 pm Post subject: |
|
|
I have an excellent connection to the box.
It is on DSL and I am at work which provides a huge pipe
(university). I tried the ping test with 50 packets. No dropped
packets and replies are less than 100ms.
The box is also a router so I will try this from my internal network
tonight after work. I don't think its a firewall issue since port 22
is wide open.
Another wierd issue. I can run a regular ls in a directory with few files
and it finishes no problem but if I go into /etc and do an ls it gets out
about 10 lines then freezes. It must have something to do with ssh's
output buffer or something. (Grasps at straws) _________________ "Against stupidity, the Gods themselves contend in vain."
-- Friedrich von Schiller |
|
Back to top |
|
|
MagicMonkey n00b
Joined: 17 Dec 2003 Posts: 37 Location: UK
|
Posted: Thu May 20, 2004 4:55 pm Post subject: |
|
|
Is it a consistent problem, ie does it always stop after the same ten lines, or do you get slightly differing results each time? I ask because I've had random problems before, and it turned out to be a slightly faulty memory module. I assume it didn't freeze the whole machine either because I was lucky, or because the kernel keeps itself in lower memory areas. I'm just guessing though... |
|
Back to top |
|
|
hitman200ca Tux's lil' helper
Joined: 02 Oct 2002 Posts: 110 Location: Canada
|
Posted: Thu May 20, 2004 5:25 pm Post subject: |
|
|
It is not exactly random either.
I can run a thousand ls / 's which only write out one line to the
console. But if I try ls /etc I always get 10 lines and ls -R /etc
is always only 2 lines. netstat -an is always 2 lines etc.
I have tried using different clients on different machines to the
same effect. _________________ "Against stupidity, the Gods themselves contend in vain."
-- Friedrich von Schiller |
|
Back to top |
|
|
hitman200ca Tux's lil' helper
Joined: 02 Oct 2002 Posts: 110 Location: Canada
|
Posted: Thu May 20, 2004 11:24 pm Post subject: |
|
|
UPDATE)
I have a Fedora Core 2 box behind my Gentoo router and I can ssh in no problem
and all command work without error. There are two possibilities at this point (that
are obvious anyway) The first is the since the connection is slower over the internet
something to do with the buffer is messing up when it sends the commands response.
Or
My firewall setting are screwing up the connection midway ??? This sounds quite
improbable. I'm off to try googling this issue. _________________ "Against stupidity, the Gods themselves contend in vain."
-- Friedrich von Schiller |
|
Back to top |
|
|
hitman200ca Tux's lil' helper
Joined: 02 Oct 2002 Posts: 110 Location: Canada
|
Posted: Thu May 20, 2004 11:54 pm Post subject: |
|
|
UPDATE 2)
I found out partially what is going on but as for what is ACTUALLY going on I
have no idea. I cut out a bunch of extraneous log messages generated by my
firewall script that where just getting in the way. Afterward I run a test and found
this
Code: |
May 20 19:44:13 DarkCloud INVALID INPUT IN=eth0 OUT= MAC=00:50:bf:3a:49:6e:00:04:e2:18:55:2e:08:00 SRC=192.168.2.1 DST=192.168.2.7 LEN=56 TOS=0x00 PREC=0x00 TTL=64 ID=4542 DF PROTO=ICMP TYPE=3 CODE=4 [SRC=192.168.2.7 DST=xxxxxxxxxx LEN=1520 TOS=0x10 PREC=0x00 TTL=63 ID=61197 FRAG:64 PROTO=TCP ] MTU=1492
May 20 19:44:23 DarkCloud INVALID INPUT IN=eth0 OUT= MAC=00:50:bf:3a:49:6e:00:04:e2:18:55:2e:08:00 SRC=192.168.2.1 DST=192.168.2.7 LEN=56 TOS=0x00 PREC=0x00 TTL=64 ID=4544 DF PROTO=ICMP TYPE=3 CODE=4 [SRC=192.168.2.7 DST=xxxxxxxxxxxx LEN=1520 TOS=0x10 PREC=0x00 TTL=63 ID=61198 FRAG:64 PROTO=TCP ] MTU=1492
May 20 19:44:41 DarkCloud INVALID INPUT IN=eth0 OUT= MAC=00:50:bf:3a:49:6e:00:04:e2:18:55:2e:08:00 SRC=192.168.2.1 DST=192.168.2.7 LEN=56 TOS=0x00 PREC=0x00 TTL=64 ID=4546 DF PROTO=ICMP TYPE=3 CODE=4 [SRC=192.168.2.7 DST=xxxxxxxxxxxx LEN=1520 TOS=0x10 PREC=0x00 TTL=63 ID=61199 FRAG:64 PROTO=TCP ] MTU=1492
May 20 19:45:19 DarkCloud INVALID INPUT IN=eth0 OUT= MAC=00:50:bf:3a:49:6e:00:04:e2:18:55:2e:08:00 SRC=192.168.2.1 DST=192.168.2.7 LEN=56 TOS=0x00 PREC=0x00 TTL=64 ID=4548 DF PROTO=ICMP TYPE=3 CODE=4 [SRC=192.168.2.7 DST=xxxxxxxxxxx LEN=1520 TOS=0x10 PREC=0x00 TTL=63 ID=61200 FRAG:64 PROTO=TCP ] MTU=1492
May 20 19:46:33 DarkCloud INVALID INPUT IN=eth0 OUT= MAC=00:50:bf:3a:49:6e:00:04:e2:18:55:2e:08:00 SRC=192.168.2.1 DST=192.168.2.7 LEN=56 TOS=0x00 PREC=0x00 TTL=64 ID=4550 DF PROTO=ICMP TYPE=3 CODE=4 [SRC=192.168.2.7 DST=xxxxxxxxxxxx LEN=1520 TOS=0x10 PREC=0x00 TTL=63 ID=61201 FRAG:64 PROTO=TCP ] MTU=1492
|
The 192.168.2.1 is my DSL router and 192.168.2.7 is the box running SSHD.
It is setup in the DMZ so everything from the net goes to it. THe xxxxxxxxxxx
is the machine ssh'ing into the box that eventually locks up because the traffic
is dropped by the firewall.
Here is my FW script.
Code: | #!/bin/sh
echo -e "\n\nLoading Firewall ...\n"
# Location of Key components
IPT=/sbin/iptables
# Setting the EXTERNAL and INTERNAL interfaces for the network
EXTIF="eth0"
INTIF="eth1"
echo " External Interface: $EXTIF"
echo " Internal Interface: $INTIF"
# Load needed modules
echo " Loading Modules"
modprobe ip_tables
modprobe ip_conntrack
modprobe ip_conntrack_ftp
modprobe iptable_nat
modprobe iptable_filter
modprobe ipt_LOG
modprobe ipt_state
modprobe ipt_MASQUERADE
# Enable IP forwarding since it is disabled by default
echo " Enabling forwarding.."
echo "1" > /proc/sys/net/ipv4/ip_forward
# Enable Dynamic IP resolution
echo " Enabling DynamicAddr.."
echo "1" > /proc/sys/net/ipv4/ip_dynaddr
# Enable Non-Local Bind
echo " Enabling Non-Local Bind.."
echo "1" > /proc/sys/net/ipv4/ip_nonlocal_bind
#Clearing any previous configuration
#
# Unless specified, the defaults for INPUT and OUTPUT is ACCEPT
# The default for FORWARD is DROP (REJECT is not a valid policy)
#
echo " Clearing any existing rules"
$IPT -F
$IPT -X
$IPT -Z
$IPT -t nat -F
$IPT -t nat -X
$IPT -t nat -Z
echo " [all] default DROP"
$IPT -P INPUT DROP
$IPT -P OUTPUT DROP
$IPT -P FORWARD DROP
echo " [$EXTIF]+ SNAT (MASQUERADE) functionality"
$IPT -t nat -A POSTROUTING -o $EXTIF -j MASQUERADE
echo " [all]- NEW without SYN"
$IPT -A INPUT -i $EXTIF -p tcp ! --syn -m state --state NEW -j LOG --log-prefix "INPUT didn't SYN first "
$IPT -A INPUT -i $EXTIF -p tcp ! --syn -m state --state NEW -j DROP
$IPT -A FORWARD -i $EXTIF -p tcp ! --syn -m state --state NEW -j LOG --log-prefix "FORWARD didn't SYN first "
$IPT -A FORWARD -i $EXTIF -p tcp ! --syn -m state --state NEW -j DROP
echo " [all]- INVALID packets"
$IPT -A INPUT -m state --state INVALID -j LOG --log-prefix "INVALID INPUT "
$IPT -A INPUT -m state --state INVALID -j DROP
$IPT -A OUTPUT -m state --state INVALID -j LOG --log-prefix "INVALID OUTPUT "$IPT -A OUTPUT -m state --state INVALID -j DROP
$IPT -A FORWARD -m state --state INVALID -j LOG --log-prefix "INVALID FORWARD "
$IPT -A FORWARD -m state --state INVALID -j DROP
echo " [$EXTIF]- traffic from 10.0.0.0/8"
$IPT -A INPUT -i $EXTIF -s 10.0.0.0/8 -j LOG --log-prefix "INPUT Class A "
$IPT -A INPUT -i $EXTIF -s 10.0.0.0/8 -j DROP
echo " [$EXTIF]- traffic from 172.16.0.0/12"
$IPT -A INPUT -i $EXTIF -s 172.16.0.0/12 -j LOG --log-prefix "INPUT Class B "$IPT -A INPUT -i $EXTIF -s 172.16.0.0/12 -j DROP
#echo " [$EXTIF]- Traffic from 192.168.0.0/16"
#$IPT -A INPUT -i $EXTIF -s 192.168.0.0/16 -j LOG --log-prefix "INPUT Class C "
#$IPT -A INPUT -i $EXTIF -s 192.168.0.0/16 -j DROP
echo " [$EXTIF]- traffic from 127.0.0.0/8"
$IPT -A INPUT -i $EXTIF -s 127.0.0.0/8 -j LOG --log-prefix "INPUT Loopback "
$IPT -A INPUT -i $EXTIF -s 127.0.0.0/8 -j DROP
# Allow all traffic to and from the loopback interface
$IPT -A INPUT -i lo -j ACCEPT
$IPT -A OUTPUT -o lo -j ACCEPT
# Allow traffic from the internal network
$IPT -A INPUT -i $INTIF -j ACCEPT
# Allow traffic to internal network
$IPT -A OUTPUT -o $INTIF -j ACCEPT
# Allow traffic to external network
$IPT -A OUTPUT -o $EXTIF -j ACCEPT
# Allow traffic through from internal network
$IPT -A FORWARD -i $INTIF -j ACCEPT
# Allow all established or related connection traffic
$IPT -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPT -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPT -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
echo " [$EXTIF]+ fwd bittorrent to 10.0.0.11"
$IPT -t nat -A PREROUTING -p tcp --dport 6881:6889 -j DNAT --to-destination 10.0.0.11
$IPT -A FORWARD -i $EXTIF -o $INTIF -p tcp -d 10.0.0.11 --dport 6881:6999 -j ACCEPT
echo " [all]+ active mode "
$IPT -A INPUT -p tcp --sport 20 -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPT -A OUTPUT -p tcp --dport 20 -m state --state ESTABLISHED -j ACCEPT
echo " [$INTIF]+ dhcpd"
$IPT -A INPUT -i $INTIF -p tcp --sport 68 --dport 67 -j ACCEPT
$IPT -A INPUT -i $INTIF -p udp --sport 68 --dport 67 -j ACCEPT
echo " [$INTIF]+ named"
$IPT -A INPUT -i $INTIF -p tcp --dport 53 -j ACCEPT
$IPT -A INPUT -i $INTIF -p udp --dport 53 -j ACCEPT
echo " [$EXTIF]+ sshd"
$IPT -A INPUT -i $EXTIF -p tcp --dport 22 -j ACCEPT
echo " [$EXTIF]+ httpd"
$IPT -A INPUT -i $EXTIF -p tcp --dport 80 -j ACCEPT
echo " [all] dropping all other packets"
#$IPT -A INPUT -j LOG --log-prefix "INPUT default DROP "
#$IPT -A OUTPUT -j LOG --log-prefix "OUTPUT default DROP "
#$IPT -A FORWARD -j LOG --log-prefix "FORWARD default DROP "
echo -e "\nDone loading firewall.\n"
r
|
_________________ "Against stupidity, the Gods themselves contend in vain."
-- Friedrich von Schiller |
|
Back to top |
|
|
hitman200ca Tux's lil' helper
Joined: 02 Oct 2002 Posts: 110 Location: Canada
|
Posted: Fri May 21, 2004 12:12 am Post subject: |
|
|
UPDATE 3) Yet another reply to myself
OK so the iptables log messages are ICMP error packets that specify that
fragmentation is needed to send the data. This makes sense because it
only happens when the command generates a lot of info on the console.
How do I enable packet fragmentation? Or is there something else
that needs to be done to fix this? Something with the MTU ? _________________ "Against stupidity, the Gods themselves contend in vain."
-- Friedrich von Schiller |
|
Back to top |
|
|
Chris W l33t
Joined: 25 Jun 2002 Posts: 972 Location: Brisbane, Australia
|
Posted: Fri May 21, 2004 12:32 am Post subject: |
|
|
If you are using Roaring Penguin PPPOE you could look at the CLAMPMSS option. Alternatively, you should allow ICMP Dest Unreachable (Fragmentation Needed) (ICMP Type 3 Code 4) packets in to your machine so that the system can act on them in the intended fashion. _________________ Cheers,
Chris W
"Common sense: The collection of prejudices acquired by age 18." -- Einstein |
|
Back to top |
|
|
hitman200ca Tux's lil' helper
Joined: 02 Oct 2002 Posts: 110 Location: Canada
|
Posted: Fri May 21, 2004 12:37 am Post subject: |
|
|
SOLUTION)
Finally I found out what is wrong. It is surprising that this doesn't happen to
more of us since most of the iptables tutorials block this by default.
The issue is that when the client executes a command that generates a lot
of console output the server sends a larger packet to the client which the
client may not accept (based on MTU size). In my case the default MTU
is 1492 which is set somewhere, possibly in my DSL router.
So the gateway sends back an ICMP error fragmentation needed packet for
every transfered packet which registers in iptables as INVALID and in my setup
is dropped. Thus the server doesn't know that the packet should be fragmented
and the client is kept waiting, something the ssh protocol doesn't handle gracefully.
So to fix this issue you must add some lines to your FW script.
Code: | iptables -A INPUT -p icmp --icmp-type fragmentation-needed -j ACCEPT
# if your box is a router then you also need
iptables -A FORWARD -p icmp --icmp-type fragmentation-needed -j ACCEPT
|
Thanks to those who tried to help me on this.
EDIT)
It appears that while I was typing this post ChrisW posted the solution. Thanks.
:o) _________________ "Against stupidity, the Gods themselves contend in vain."
-- Friedrich von Schiller |
|
Back to top |
|
|
|