| View previous topic :: View next topic |
| Author |
Message |
dalu
Guru
Joined: 20 Jan 2003
Posts: 530
|
 Posted: Wed Feb 04, 2015 3:43 pm Post subject: last|head ssh entry and problems , GLSA Support pls |
 |
|
As posted on Google+, perhaps I'll be more lucky here
Right,
last|head
| Code: |
root pts/0 2a02:8070:c4c2:2 Wed Feb 4 15:52 still logged in
root ssh 2a02:8070:c4c2:2 Wed Feb 4 15:52 still logged in
root pts/0 2a02:8070:c4c2:2 Tue Feb 3 14:33 - 14:51 (00:17)
root ssh 2a02:8070:c4c2:2 Tue Feb 3 14:33 - 14:51 (00:17)
|
As you can see there's a pts/0 and a ssh line
it's the same session however.
Occasionally the ssh one died, for whatever reason.
The pts/* session remains active but no responses are sent to the client since ssh is dead.
Using systemd with a custom config, kernel 3.18.5 gentoo-sources
What might be the reason?
For instance, an Archlinux or Debian system for comparison:
| Code: |
root pts/0 2a02:8070:c4c2:2 Wed Feb 4 16:06 still logged in
root pts/0 2a02:8070:c4c2:2 Wed Feb 4 11:19 - 11:19 (00:00)
|
Last edited by dalu on Thu Feb 12, 2015 7:00 pm; edited 1 time in total |
|
| Back to top |
|
 |
Schnulli
Guru
Joined: 25 Jun 2010
Posts: 320
Location: Bremen DE
|
| Posted: Sat Feb 07, 2015 1:58 pm Post subject: |
|
|
Hi
i am wondering also for the same thing since a while and yet no answer why a dead ssh session is kept somehow alive but dead.....
just wondering 
A new Bug? |
|
| Back to top |
|
|
Hu
Administrator
Joined: 06 Mar 2007
Posts: 23070
|
| Posted: Sat Feb 07, 2015 4:16 pm Post subject: |
|
|
| Does the server system know the client has died? If the client process exited while unable to communicate with the server, then the server will not detect this until it tries and fails to send traffic to the client. Generate some output on that pty and the session should go away. |
|
| Back to top |
|
|
Schnulli
Guru
Joined: 25 Jun 2010
Posts: 320
Location: Bremen DE
|
| Posted: Sat Feb 07, 2015 5:41 pm Post subject: |
|
|
Hi Hu
dosnt matter or stresses me..... this means to me, an open pipe is kept... so... reason one... maybe a bug in ssh itself, or reason tow, ssh is not correctly configured
i for myself kill this session by myself.
i never had intentions to find out what the reason is because i screen my systems and kill pids if they are dead |
|
| Back to top |
|
|
dalu
Guru
Joined: 20 Jan 2003
Posts: 530
|
| Posted: Thu Feb 12, 2015 6:02 pm Post subject: |
|
|
Ok, it just happened again.
Funny thing is
As soon as I noticed I opened a new terminal on the client and ssh'd into the server.
did
w
| Code: |
# w
18:51:35 up 3:09, 2 users, load average: 0.10, 0.17, 0.11
USER TTY LOGIN@ IDLE JCPU PCPU WHAT
root pts/0 16:16 15.00s 0.06s 0.06s -bash
root pts/1 18:51 1.00s 0.01s 0.00s w
|
I hit [Enter] like 30 times (on pts/0) and the session re-appeared or better, the terminal became responsive again.
However I'm not sure if that's because pts/1 is active.
| Code: |
# last|head
root pts/1 2a02:8070:c4c2:2 Thu Feb 12 18:51 still logged in
root ssh 2a02:8070:c4c2:2 Thu Feb 12 18:51 still logged in
root pts/0 2a02:8070:c4c2:2 Thu Feb 12 16:16 still logged in
root ssh 2a02:8070:c4c2:2 Thu Feb 12 16:16 - 18:51 (02:35)
|
Hu,
yes when the client terminal gets closed the server detects this and "logs off" pts/0 (for instance).
It often happens when I emerge something.
edit:
I have now closed pts/1 session
| Code: |
root pts/1 2a02:8070:c4c2:2 Thu Feb 12 18:51 - 19:06 (00:14)
root ssh 2a02:8070:c4c2:2 Thu Feb 12 18:51 - 19:06 (00:14)
root pts/0 2a02:8070:c4c2:2 Thu Feb 12 16:16 still logged in
root ssh 2a02:8070:c4c2:2 Thu Feb 12 16:16 - 18:51 (02:35)
|
but pts/0 session is still active and responsive (on my client and the server) |
|
| Back to top |
|
|
Schnulli
Guru
Joined: 25 Jun 2010
Posts: 320
Location: Bremen DE
|
| Posted: Thu Feb 12, 2015 6:42 pm Post subject: |
|
|
Hi dalu
the same simptoms here.....
even killing the PID dosnt affect and its still alive......
possible a Bug and new Backdoor detected? Seems like....again ^^
Hey GLSA ! have a look at
@dalu
change ur Topic and add "GLSA Support pls"
Regards |
|
| Back to top |
|
|
dalu
Guru
Joined: 20 Jan 2003
Posts: 530
|
| Posted: Thu Feb 12, 2015 6:59 pm Post subject: |
|
|
You think, Schnulli?
Isn't that kind of drastic before doing our own investigation?
Ah well, better safe than sorry, I don't want to bug them if it isn't necessary 
They probably have more stuff to worry about.
However yeah it is the default way it works right now (sshd).
What I did so far was
check sshd_config, nothing out of the ordinary
change sshd.service sshd@.service to Archlinux ones add sshgenkeys.service, no effect
Next up:
find out which package wtmp btmp is 
emerge openssh-6.7 (no -r3, aka without the x509 patch and the other glue patch, not sure what it does) and see if that also happens
netstat doesn't show any other connections for sshd but that doesn't have to mean anything. |
|
| Back to top |
|
|
Schnulli
Guru
Joined: 25 Jun 2010
Posts: 320
Location: Bremen DE
|
| Posted: Thu Feb 12, 2015 7:06 pm Post subject: |
|
|
yep
better safe than sorry is also my way of thinking, thats one reason why Gentoo is still so clean and so much masked because dirty
A backstep is mostly this i use, in this case to risky, the old sshd version is buggy ^^
If you ntop/stat something let it run for at last a month to be sure and >> (pipe) the output it into a file..... file size dosnt matter in our linux thinking mind
by the way, why not to deep and longterm screen with Wireshark ?
regards
| dalu wrote: | You think, Schnulli?
Isn't that kind of drastic before doing our own investigation?
Ah well, better safe than sorry, I don't want to bug them if it isn't necessary
They probably have more stuff to worry about.
However yeah it is the default way it works right now (sshd).
What I did so far was
check sshd_config, nothing out of the ordinary
change sshd.service sshd@.service to Archlinux ones add sshgenkeys.service, no effect
Next up:
find out which package wtmp btmp is
emerge openssh-6.7 (no -r3, aka without the x509 patch and the other glue patch, not sure what it does) and see if that also happens
netstat doesn't show any other connections for sshd but that doesn't have to mean anything. |
|
|
| Back to top |
|
|
dalu
Guru
Joined: 20 Jan 2003
Posts: 530
|
| Posted: Thu Feb 12, 2015 7:31 pm Post subject: |
|
|
| Schnulli wrote: |
If you ntop/stat something let it run for at last a month to be sure and >> (pipe) the output it into a file..... file size dosnt matter in our linux thinking mind
|
Well my rootfs is just 384GB, the rest is dedicated to mongodb.
| Schnulli wrote: |
by the way, why not to deep and longterm screen with Wireshark ?
|
I need to write my auth server and lib (http), this wireshark logging would require 1-2 days extra, time is ticking, each day costs 4€ for running the 3 servers and they're not generating any income yet and there's still so much left to do, in short I don't really believe it's a security issue but an annoying bug and I need to get on with my plan So much to do, so little time.
Maybe I misunderstood you, I don't like the limited scrollback of screen. And it works on other distros, so it should be working here.
I'll try a modified ebuild without the x509 patches. How is the non-r3 buggy?
| Code: |
diff openssh-6.7_p1.ebuild openssh-6.7_p1-r3.ebuild
3c3
< # $Header: /var/cvsroot/gentoo-x86/net-misc/openssh/openssh-6.7_p1.ebuild,v 1.13 2014/12/31 07:40:01 vapier Exp $
---
> # $Header: /var/cvsroot/gentoo-x86/net-misc/openssh/openssh-6.7_p1-r3.ebuild,v 1.2 2014/12/31 07:29:47 vapier Exp $
14c14
< #X509_VER="8.1" X509_PATCH="${PARCH/6.7/6.6}+x509-${X509_VER}.diff.gz"
---
> X509_VER="8.2" X509_PATCH="${PARCH}+x509-${X509_VER}.diff.gz"
31,32c31,32
< KEYWORDS="alpha amd64 arm arm64 hppa ia64 m68k ~mips ppc ppc64 s390 sh sparc x86 ~amd64-fbsd ~sparc-fbsd ~x86-fbsd ~arm-linux ~x86-linux"
< IUSE="bindist ${HPN_PATCH:++}hpn kerberos ldap ldns libedit pam +pie sctp selinux skey static X X509"
---
> KEYWORDS="~alpha ~amd64 ~arm ~arm64 ~hppa ~ia64 ~m68k ~mips ~ppc ~ppc64 ~s390 ~sh ~sparc ~x86 ~amd64-fbsd ~sparc-fbsd ~x86-fbsd ~arm-linux ~x86-linux"
> IUSE="bindist ${HPN_PATCH:++}hpn kerberos kernel_linux ldap ldns libedit pam +pie sctp selinux skey static X X509"
110,111c110,111
< epatch "${FILESDIR}"/${PN}-6.6.1_p1-x509-glue.patch
< use hpn && epatch "${FILESDIR}"/${PN}-6.6.1_p1-x509-hpn14v5-glue.patch
---
> epatch "${FILESDIR}"/${P}-x509-glue.patch
> epatch "${FILESDIR}"/${P}-sctp-x509-glue.patch
191c191
< --with-pid-dir="${EPREFIX}"/var/run \
---
> --with-pid-dir="${EPREFIX}"$(usex kernel_linux '' '/var')/run \
|
Have you tried "downgrading" to the non-r3 ebuild? |
|
| Back to top |
|
|
Schnulli
Guru
Joined: 25 Jun 2010
Posts: 320
Location: Bremen DE
|
| Posted: Thu Feb 12, 2015 7:38 pm Post subject: |
|
|
yes, on other distros it works... thats why i ask myself what the heck is on Gentoo the reason.....
No income yet on ur servers? where they are located? Country? and Line speed? Any Traffic limitations?
Lets talk and open me a VirtualBox Slot & v-host remote management and i ll help paying ur bills at last a few
This is more easy instead of renting again another 4HE Rack slot somewhere
No i havent downgraded yet bec. my SSHs are running behind a firewall, thats the reason why i wonder but dont get scared
| dalu wrote: | | Schnulli wrote: |
If you ntop/stat something let it run for at last a month to be sure and >> (pipe) the output it into a file..... file size dosnt matter in our linux thinking mind
|
Well my rootfs is just 384GB, the rest is dedicated to mongodb.
| Schnulli wrote: |
by the way, why not to deep and longterm screen with Wireshark ?
|
I need to write my auth server and lib (http), this wireshark logging would require 1-2 days extra, time is ticking, each day costs 4€ for running the 3 servers and they're not generating any income yet and there's still so much left to do, in short I don't really believe it's a security issue but an annoying bug and I need to get on with my plan So much to do, so little time.
Maybe I misunderstood you, I don't like the limited scrollback of screen. And it works on other distros, so it should be working here.
I'll try a modified ebuild without the x509 patches. How is the non-r3 buggy?
| Code: |
diff openssh-6.7_p1.ebuild openssh-6.7_p1-r3.ebuild
3c3
< # $Header: /var/cvsroot/gentoo-x86/net-misc/openssh/openssh-6.7_p1.ebuild,v 1.13 2014/12/31 07:40:01 vapier Exp $
---
> # $Header: /var/cvsroot/gentoo-x86/net-misc/openssh/openssh-6.7_p1-r3.ebuild,v 1.2 2014/12/31 07:29:47 vapier Exp $
14c14
< #X509_VER="8.1" X509_PATCH="${PARCH/6.7/6.6}+x509-${X509_VER}.diff.gz"
---
> X509_VER="8.2" X509_PATCH="${PARCH}+x509-${X509_VER}.diff.gz"
31,32c31,32
< KEYWORDS="alpha amd64 arm arm64 hppa ia64 m68k ~mips ppc ppc64 s390 sh sparc x86 ~amd64-fbsd ~sparc-fbsd ~x86-fbsd ~arm-linux ~x86-linux"
< IUSE="bindist ${HPN_PATCH:++}hpn kerberos ldap ldns libedit pam +pie sctp selinux skey static X X509"
---
> KEYWORDS="~alpha ~amd64 ~arm ~arm64 ~hppa ~ia64 ~m68k ~mips ~ppc ~ppc64 ~s390 ~sh ~sparc ~x86 ~amd64-fbsd ~sparc-fbsd ~x86-fbsd ~arm-linux ~x86-linux"
> IUSE="bindist ${HPN_PATCH:++}hpn kerberos kernel_linux ldap ldns libedit pam +pie sctp selinux skey static X X509"
110,111c110,111
< epatch "${FILESDIR}"/${PN}-6.6.1_p1-x509-glue.patch
< use hpn && epatch "${FILESDIR}"/${PN}-6.6.1_p1-x509-hpn14v5-glue.patch
---
> epatch "${FILESDIR}"/${P}-x509-glue.patch
> epatch "${FILESDIR}"/${P}-sctp-x509-glue.patch
191c191
< --with-pid-dir="${EPREFIX}"/var/run \
---
> --with-pid-dir="${EPREFIX}"$(usex kernel_linux '' '/var')/run \
|
Have you tried "downgrading" to the non-r3 ebuild? |
|
|
| Back to top |
|
|
dalu
Guru
Joined: 20 Jan 2003
Posts: 530
|
| Posted: Thu Feb 12, 2015 8:21 pm Post subject: |
|
|
| Schnulli wrote: | yes, on other distros it works... thats why i ask myself what the heck is on Gentoo the reason.....
No income yet on ur servers? where they are located? Country? and Line speed? Any Traffic limitations?
Lets talk and open me a VirtualBox Slot & v-host remote management and i ll help paying ur bills at last a few
This is more easy instead of renting again another 4HE Rack slot somewhere
No i havent downgraded yet bec. my SSHs are running behind a firewall, thats the reason why i wonder but dont get scared
|
Downgrading changed nothing
I'm not doing virtualization but I can give you the Google+ Profile of someone who just started like a month ago, with 2 E5s using ganetti (and Funtoo).
The 3 servers are cheap refurbished Hetzner Xeon 1245v2 with 16GB ECC RAM and 2x3TB HGST Disks, really low cost.
I'm writing programs in Go since 1¼ years, switched from PHP and picked MongoDB for storage. The 3 cheap ones are cheaper than 1 big server.
I have
3 nginx on the front,
libreswan (IPsec) on private addrs
services listening on private addrs, for instance domain.tld service listening on s0:10000 s1:10000 s2:10000
where
s0 = 10.0.1.1
s1 = 10.0.1.2
s2 = 10.0.1.3
and in the back I have 3 replica sets which form a shard and the services talk to the mongos (mongodb shard service)
since mongodb is able to work with files (gridfs) I use this for storage
All services run as their own user with their custom, very limited shell that only accepts git pushes and "update" "build" "env" "ls".
And I have a management service listening on each server to create those services/users.
No virtualization, no containers, just systemd settings to control read/write permissions and capabilities and limits.
Since each service is a static binary that runs in its VM with safe types and 1 "GOMAXPROC" I'm not afraid of off by 1 or other attacks and the mongodb driver sanitizes by default (also because of safe Go types), so no fear of "injections".
And it all costs ~120€ / month.
That's 7.8TB single replicated storage, where 2 nodes can come down and content is still serving, but no writes can be made.
Bandwidth on paper is limited to 200mbit/s per node, real data shows it's less.
Each node is limited to 20TB outgoing / month, good enough for me, for starters
So that's 12 cores, 3x16GB ECC RAM, 7.8TB replicated storage across nodes and theoretical throughput of 600mbit/s for 120€ / month.
Aka ~5k-15k concurrent connections.
A similar offer from online.net by with just 2x3TB storage costs 155€ incl VAT.
And I don't have to deal with VMs and/or containerization.
So for all my domains I first need to write the base, authentication
then add authorization
then make it openid-connect compatible
and lastly build content sites or "apps"
but all white-hat and legal, no black-hat stuff, nothing illegal.
You know what I'm looking for? Audio ads for mp3 content. If I could monetize audio I could afford to pay people to do coding and travel the world, instead of sitting in front of my PC the whole day long And no Youtube doesn't cut it.
I can send you a PM if you want the guy's contact on Google+ |
|
| Back to top |
|
|
Schnulli
Guru
Joined: 25 Jun 2010
Posts: 320
Location: Bremen DE
|
| Posted: Thu Feb 12, 2015 10:18 pm Post subject: |
|
|
| u have a PN |
|
| Back to top |
|
|
dalu
Guru
Joined: 20 Jan 2003
Posts: 530
|
| Posted: Mon Feb 16, 2015 12:08 pm Post subject: |
|
|
Now back on topic.
| Code: |
last|head
root pts/1 2a02:8070:c4c2:2 Mon Feb 16 12:38 still logged in
root ssh 2a02:8070:c4c2:2 Mon Feb 16 12:38 still logged in
root pts/0 2a02:8070:c4c2:2 Mon Feb 16 11:59 still logged in
root ssh 2a02:8070:c4c2:2 Mon Feb 16 11:59 - 12:38 (00:38)
|
I had "less /etc/pam.d/system-auth" running and my ISP decided that it's time to reboot my router (...),
so I got disconnected. Logged in again and pts/0 was still active with "less" still running.
So I killed less and killed bash associated with pts/0.
I've noticed that there's a difference between Archlinux' and Gentoo's /etc/pam.d/systemd-user
and few others
/etc/pam.d/sshd
points to system-remote-login
system-remote-login points to system-login
Gentoo's system-login
| Code: |
auth required pam_tally2.so onerr=succeed
auth required pam_shells.so
auth required pam_nologin.so
auth include system-auth
account required pam_access.so
account required pam_nologin.so
account include system-auth
account required pam_tally2.so onerr=succeed
password include system-auth
session optional pam_loginuid.so
session required pam_env.so
session optional pam_lastlog.so silent
session include system-auth
session optional pam_motd.so motd=/etc/motd
session optional pam_mail.so
|
Archlinux' system-login
| Code: |
#%PAM-1.0
auth required pam_tally.so onerr=succeed file=/var/log/faillog
auth required pam_shells.so
auth requisite pam_nologin.so
auth include system-auth
account required pam_access.so
account required pam_nologin.so
account include system-auth
password include system-auth
session optional pam_loginuid.so
session include system-auth
session optional pam_motd.so motd=/etc/motd
session optional pam_mail.so dir=/var/spool/mail standard quiet
-session optional pam_systemd.so
session required pam_env.so
|
difference
Gentoo has
| Code: |
auth required pam_nologin.so
account required pam_tally2.so onerr=succeed
|
Archlinux has
| Code: |
auth requisite pam_nologin.so
-session optional pam_systemd.so
|
However
system-auth
Gentoo's system-auth
| Code: |
auth required pam_env.so
auth required pam_unix.so try_first_pass likeauth nullok
auth optional pam_permit.so
account required pam_unix.so
account optional pam_permit.so
password required pam_cracklib.so difok=2 minlen=8 dcredit=2 ocredit=2 retry=3
password required pam_unix.so try_first_pass use_authtok nullok sha512 shadow
password optional pam_permit.so
session required pam_limits.so
session required pam_env.so
session required pam_unix.so
session optional pam_permit.so
-session optional pam_systemd.so
|
Archlinux' system-auth
| Code: |
#%PAM-1.0
auth required pam_unix.so try_first_pass nullok
auth optional pam_permit.so
auth required pam_env.so
account required pam_unix.so
account optional pam_permit.so
account required pam_time.so
password required pam_unix.so try_first_pass nullok sha512 shadow
password optional pam_permit.so
session required pam_limits.so
session required pam_unix.so
session optional pam_permit.so
|
Difference ,amongst other things
Gentoo's has
| Code: |
-session optional pam_systemd.so
|
Now systemd-user
Gentoo's systemd-user
| Code: |
# This file is part of systemd.
#
# Used by systemd --user instances.
account include system-auth
session include system-auth
|
Archlinux' systemd-user
| Code: |
# This file is part of systemd.
#
# Used by systemd --user instances.
account include system-login
session include system-login
|
Gentoo wants system-auth
Arch wants system-login
Downloading and installing Fedora to see how they do it.
Also I should probably read pam's manual
http://www.linux-pam.org/Linux-PAM-html/Linux-PAM_SAG.html
Fedora 21 uses authconf to generate pam config files (good idea actually).
and its sshd file looks like this
| Code: |
#%PAM-1.0
auth required pam_sepermit.so
auth substack password-auth
auth include postlogin
# Used with polkit to reauthorize users in remote sessions
-auth optional pam_reauthorize.so prepare
account required pam_nologin.so
account include password-auth
password include password-auth
# pam_selinux.so close should be the first session rule
session required pam_selinux.so close
session required pam_loginuid.so
# pam_selinux.so open should only be followed by sessions to be executed in the user context
session required pam_selinux.so open env_params
session optional pam_keyinit.so force revoke
session include password-auth
session include postlogin
# Used with polkit to reauthorize users in remote sessions
-session optional pam_reauthorize.so prepare
|
and the password-auth substack
| Code: |
#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth required pam_env.so
auth sufficient pam_unix.so nullok try_first_pass
auth requisite pam_succeed_if.so uid >= 1000 quiet_success
auth required pam_deny.so
account required pam_unix.so
account sufficient pam_localuser.so
account sufficient pam_succeed_if.so uid < 1000 quiet
account required pam_permit.so
password requisite pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type=
password sufficient pam_unix.so sha512 shadow nullok try_first_pass use_authtok
password required pam_deny.so
session optional pam_keyinit.so revoke
session required pam_limits.so
-session optional pam_systemd.so
session [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session required pam_unix.so
|
|
|
| Back to top |
|
|
Ant P.
Watchman
Joined: 18 Apr 2009
Posts: 6920
|
| Posted: Mon Feb 16, 2015 6:08 pm Post subject: |
|
|
| Can you justify having PAM installed at all, when you don't understand its security implications? |
|
| Back to top |
|
|
dalu
Guru
Joined: 20 Jan 2003
Posts: 530
|
| Posted: Tue Mar 03, 2015 10:08 am Post subject: |
|
|
| Code: |
root pts/0 2a02:8070:c48f:3 Tue Mar 3 09:46 still logged in
root ssh 2a02:8070:c48f:3 Tue Mar 3 09:46 still logged in
root pts/2 2a02:8070:c48f:3 Mon Mar 2 20:44 - 20:57 (00:13)
root ssh 2a02:8070:c48f:3 Mon Mar 2 20:44 - 20:57 (00:12)
|
Where is pts/1 ?
net-misc/openssh-6.7_p1-r4::gentoo
sys-apps/systemd-219-r1:0/2::gentoo
seriously, what's going on there? |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Networking & Security
