cboldt
Veteran
Joined: 24 Aug 2005
Posts: 1046
|
 Posted: Tue May 26, 2015 12:12 pm Post subject: watch-logs : IP ban function using syslog-ng and iptables |
 |
|
After running sshguard and fail2ban for some time, I decided to reinvent the wheel and compose an alternative. sshguard and fail2ban work fine, so it's hard for me to ascribe any particular motivation to this effort, beyond a desire to learn some new bash techniques and to have a smaller (but not necessarily more efficient), tailored solution to my desire for automating firewall maintenance.
Most of the intrusion attempts here have been against sshd, and knockd (properly setup) is more secure than reacting to a failed intrusion attempt. At the same time, the knockd method is not as informative as to which IPs are attempting intrusion. So, there was a bit of curiosity factor involved too. The script started out as just watching for sshd and firewall activity, before I figured out generic coding that allows easily expanding the scope of monitoring application logging activity.
syslog-ng facilitates monitoring a variety of applications by providing a way to direct selected logging activity to a program, thereby eliminating a need to watch separate log files. syslog-ng.conf is configured to filter interesting messages, and send them to the "watch-logs" script.
Independent of the script, my firewall has a few chains that reduce the number of hits against sshd and smtpd. These might be sufficient for some systems, but I wanted a tool that imposed time-limited bans that would survive flushing and re-establishing the firewall.
| Code: | # using iptables to regulate hits against $SSHD_PORT
# on third attempt in four minutes, placed into 15 minute timeout
# timeout is restarted for attempts made during timeout
iptables -N sshd-drop
iptables -A sshd-drop -j LOG --log-prefix "$LOG_SSHD_DENY "
iptables -A sshd-drop -m recent --name SSHD-deny --set -j DROP
iptables -N sshd-persist
iptables -A sshd-persist -m limit --limit 1/hour --limit-burst 1 -j LOG --log-prefix "$LOG_SSHD_PERSIST "
iptables -A sshd-persist -j DROP
iptables -N sshd-scan
iptables -A sshd-scan -m recent --name SSHD-deny --update --reap --seconds 900 -j sshd-persist
iptables -A sshd-scan -m recent --name SSHD --update --reap --seconds 240 --hitcount 3 -j sshd-drop
iptables -A sshd-scan -m recent --name SSHD --set -j LOG --log-prefix "$LOG_SSHD_ATTEMPT "
iptables -A sshd-scan -j ACCEPT
iptables -A INPUT -p tcp --dport $SSHD_PORT -m conntrack --ctstate NEW -j sshd-scan
# using iptables to regulate hits against ports 25 and 465
# on third attempt in an hour, placed into 24 hour timeout
# timeout is restarted for attempts made during timeout
iptables -N smtp-drop
iptables -A smtp-drop -j LOG --log-prefix "$LOG_SMTP_DENY "
iptables -A smtp-drop -m recent --name SMTP-deny --set -j DROP
iptables -N smtp-scan
iptables -A smtp-scan -m recent --name SMTP-deny --update --reap --seconds 86400 -j DROP
iptables -A smtp-scan -m recent --name SMTP --update --reap --seconds 3600 --hitcount 3 -j smtp-drop
iptables -A smtp-scan -m recent --name SMTP --set -j LOG --log-prefix "$LOG_SMTP_ATTEMPT "
iptables -A smtp-scan -j ACCEPT
iptables -A INPUT -p tcp --dport 25 -m conntrack --ctstate NEW -j smtp-scan
iptables -A INPUT -p tcp --dport 465 -m conntrack --ctstate NEW -j smtp-scan |
That code isn't necessary for the watch-logs script to function, but it does provide a firewall layer that reduces the rate of intrusion attempts reaching the applications, and being logged.
Now the watch-logs script. I don't think any intrusion detection and reaction system is particularly easy to setup, and this is no exception. The user has to configure at least syslog-ng and the firewall to take advantage of the script; and may have to configure watch-logs. I tried to make that easy for both initial setup and for expanding the scope of view. I don't believe this approach scales up well, the script probably becomes overloaded somewhere around 50-200 hits per second. Another issue is that the script is not immune to being spoofed by locally-crafted logging.
| Code: | #! /bin/bash
# /usr/local/sbin/watch-logs
version=0.1 # 25 May 2015
# This script is designed to be invoked by syslog-ng
# Running the script from a shell merely gives brief instructions
# This script uses input from iptables, smtpd, imap, and sshd logging, all
# delivered directly to this script by syslog-ng. This script tests that input.
# Too many "Failed " or "authentication failed" reports result in banning
# the offending IP for a selectable period of time
# Expired bans can be removed by sending a USR1 or USR2 signal to this script
# It is suggested to `pkill -SIGUSR1 watch-logs` from /etc/cron.daily/logrotate
# and when rebuilding firewall, e.g. from /etc/iptables/establish-iptables
# Default BAN rule threshold is 10 hits in 1day, ban duration 3weeks
# BAN rule thresholds established by this script are ...
# ("smtpd" and "imap" activity is grouped under type "mail")
# ----------- ----- -------------- ----------------
# Type of hit Limit Hit-life Ban Duration
# ----------- ----- -------------- ----------------
# firewall 10 1day (default) 3weeks (default)
# sshd login 3 30minutes 5days
# mail login 4 12hours 3weeks (default)
# Firewall --recent thresholds set in /etc/iptables/establish-iptables
# ----------- ----- -------------- ----------------
# Port Limit Hit-life Ban Duration
# ----------- ----- -------------- ----------------
# sshd port 3 4minutes 15minutes
# smtp ports 3 1hour 1day
# Use `iptables-save | less` to confirm firewall function.
# Banned ip addresses should appear in "watch-banned" rule chain
# Copyright (C) 2015 Chuck Seyboldt <c.cboldt@gmail.com>
# This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License
# http://creativecommons.org/licenses/by-sa/3.0/
# ------------ GET CONFIGURATION -------------
# N.B. configuration is part of main routine, because associative arrays
# created in a subroutine aren't directly available outside that subroutine
typeset -A max_hits hit_life ban_time hit_history
shopt -s extglob
cfg=${cfg:=/etc/conf.d/iptables} # All settings, including adding new patterns,
[ -r $cfg ] && . $cfg # can be modified from optional config file
iptables=${IPTABLES:=/sbin/iptables} # To adapt to iptables6, someday
ipregex=${WATCH_LOGS_IPREGEX:='([0-9]{1,3}\.){3}[0-9]{1,3}'} # IPv4 pseudo-regex
cutter=$(which cutter) # Use `cutter` program, if it exists
verbose=${WATCH_LOGS_VERBOSE:=bans,xt_recent} # Select info to log when processing ban_log
# Options are "bans" "xt_recent" "yes" and "all"
log=${WATCH_LOGS_LOGFILEW:=/var/log/iptables/watch-logs.log} # Realtime output
ban_log=${WATCH_LOGS_BANLOG:=/var/log/iptables/watch-logs-ban.log} # Permanant record
ban_file=${WATCH_LOGS_BANFILE:=/etc/iptables/watch-logs.banned} # List of unexpired bans
hit_limit=${WATCH_LOGS_MAXHITS:=10} # This default number of hits ...
hit_expire=${WATCH_LOGS_HITLIFE:=1day} # in this default amount of time ...
ban_expire=${WATCH_LOGS_BANTIME:=3weeks} # results in a default ban this long
ban_policy=${WATCH_LOGS_BANPOLICY:=REJECT}
whitelist=${WATCH_LOGS_WHITELIST:=} # Better to whitelist by firewall, than here
drop_chain=${WATCH_LOGS_BANNED_IP_CHAIN:=watch-banned} # New iptables rule chain to hold bans
from_chain=${BLOCKED_IP_CHAIN:=blocked-ip} # Preexisting banned-ip iptables rule chain
# Incoming activity is categorized by "type" (e.g., firewall, sshd, mail)
# $watch_log_* variable names establish "types"
# $watch_log_* variable contents establish the search/watch patterns
# $watch_log_* variable contents must be bash extended pattern matching expressions
# The $LOG_* variable names are shared with /etc/iptables/establish-iptables script
# in order to coordinate iptables --log-prefix settings with pattern matching in this script
watch_log_sshd=${watch_log_sshd:='sshd.*Failed.*invalid.user.*from'}
watch_log_mail1=${watch_log_mail1:='smtpd.*authentication.failed'}
watch_log_mail2=${watch_log_mail2:='imap.*no.auth.attempts'}
watch_log_firewall1=${LOG_SSHD_ATTEMPT:=SSHD-Attempt:}
watch_log_firewall2=${LOG_SMTP_ATTEMPT:=SMTP-Attempt:}
watch_log_firewall3=${LOG_HONEYPOT:=REJECT-Honeypot:}
# Default ban threshold variables can be over-ridden from the config file
# Default thresholds set above - 10 hits per 1day results in 3week ban
# Re-start script to apply ban threshold changes
# (pkill watch-logs, syslog-ng will restart watch-logs)
max_hits[sshd]=${max_hits[sshd]:=3}
hit_life[sshd]=${hit_life[sshd]:=30minutes}
ban_time[sshd]=${ban_time[sshd]:=5days}
max_hits[mail]=${max_hits[mail]:=4}
hit_life[mail]=${hit_life[mail]:=12hours}
# Adding a new category is done by defining at least one $watch_log_* variable
# The following would add a category of "type" newtype with two matching hit strings
# Ban threshold and duration can also be configured. The example "newtype"
# is configured with 3 hits in 4hours resulting in a ban of 1month duration
# watch_log_newtype1='failed.*user.*tries'
# watch_log_newtype2='IP.address.*does.not.match'
# max_hits[newtype]=3
# hit_life[newtype]=4hours
# ban_time[newtype]=1month
# syslog-ng must send matching log_action to this script in order for the script to be useful
# A typical addition to syslog-ng.conf follows this form:
#
# filter f_newtype_warn { program(application)
# and message("failed.*user.*tries");
# or message("IP.address.*does.not.match"); };
# log { source(src); filter(f_newtype_warn); destination(watch_logs); };
# -------------------- NOW SOME SUBROUTNES ---------------------
# ----------------- EXIT AND SIGNAL HANDLING -------------------
exit_trap() {
printf "%(%s)T $HOSTNAME %s[%5s]: Exiting\n" -1 ${0##*/} $$ >> $log
exit
}
print_instructions() {
# Test to insure this script is invoked by syslog-ng
# If not, exit with installation message
if [ "$PARENT_NAME" != "syslog-ng" ]; then
echo "
Hello! ${0##*/} is designed to be invoked from and by syslog-ng.
Put the following in your syslog-ng.conf file ...
destination watch_logs { program(\"$0\" ts_format(unix)); };
filter f_iptables { facility(kern) and message(\"IN=.*OUT=.*SRC=\"); };
filter f_imap_warn { program(dovecot) and message(\"no auth attempts \"); };
filter f_mail_warn { facility(mail) and level(warn); };
filter f_sshd_warn { program(sshd) and message(\"Failed \"); };
log { source(src); filter(f_iptables); destination(watch_logs); };
log { source(src); filter(f_imap_warn); destination(watch-logs); };
log { source(src); filter(f_mail_warn); destination(watch_logs); };
log { source(src); filter(f_sshd_warn); destination(watch_logs); };
Put the following rules near the top of your iptables firewall ...
iptables -N $from_chain
iptables -A INPUT -j $from_chain
... then restart (or SIGHUP) syslog-ng.
Exiting ${0##*/} version $version now. Bye!
"
exit 1
fi
}
# ----------------- INITIALIZE FIREWALL -------------------
build_iptables_rules() {
# Create a new chain ($drop_chain) [${WATCH_LOGS_BANNED_IP_CHAIN:=watch-banned}]
# for holding iptables rules that will drop banned IPs
# $drop_chain is intended to be unique to this script - there is no need
# to create it or refer to it when building the firewall
$iptables -w -N $drop_chain > /dev/null 2>&1
$iptables -w -F $drop_chain
$iptables -w -A $drop_chain -j RETURN
# The first rule in $from_chain [${BLOCKED_IP_CHAIN:=blocked-ip}] is a jump to
# the $drop_chain list of banned IPs. The "blocked-ip" chain is tested to
# make sure it RETURNs, because INPUT should jump to "blocked-ip" early
# If there is no jump from INPUT to $from_chain (which is possible if there is
# no firewall to begin with, or firewall has no "blocked-ip" chain) then make one,
# insert as 5th rule after priority-packets, bad-flags, whitelist, and portknock
$iptables -w -N $from_chain > /dev/null 2>&1
$iptables -C $from_chain -j $drop_chain > /dev/null 2>&1 || \
$iptables -w -I $from_chain -j $drop_chain
$iptables -C $from_chain -j RETURN > /dev/null 2>&1 || \
$iptables -w -A $from_chain -j RETURN
$iptables -C INPUT -j $from_chain > /dev/null 2>&1 || \
$iptables -w -I INPUT 5 -j $from_chain
# ----------------- PROCESS BAN FILE -------------------
# This routine has multiple functions:
# - create variable $BANNED_LIST in memory to avoid repeating bans
# - trim the banlist $ban_file of expired bans
# - log unexpired bans and expiration dates to the watch-logs $log file
if [ -r $ban_file ]; then
mapfile <$ban_file ban_file_array
printf "%(%c)T $HOSTNAME : $ban_file trimmed by %s[%5s]\n" -1 ${0##*/} $$ > $ban_file
printf "%(%s)T $HOSTNAME %s[%5s]: Processing banlist $ban_file\n" -1 ${0##*/} $$ >> $log
nowtime=$(printf "%(%s)T" -1)
BANNED_LIST=(`
for (( i = 0 ; i < ${#ban_file_array[@]} ; i++ ))
do
line=( ${ban_file_array[$i]} )
if [[ "${line[@]}" =~ " : $ban_file trimmed by" ]]; then
:
elif [[ ! ${line[0]} =~ $ipregex ]]; then
printf "${ban_file_array[$i]}" >> $ban_file
elif [ -z ${line[1]} ]; then
printf "${line[0]} "
printf "${line[0]}\n" >> $ban_file
[[ "$verbose" =~ bans|yes|all ]] &&
printf "%(%s)T $HOSTNAME %s[%5s]: %-15s : BAN-LISTED : No expiration\n" -1 ${0##*/} $$ ${line[0]} >> $log
$iptables -w -I $drop_chain -s ${line[0]} -j $ban_policy
elif [ $nowtime -lt ${line[1]} ]; then
printf "${line[0]} "
printf "%-15s ${line[1]}\n" ${line[0]} >> $ban_file
[[ "$verbose" =~ bans|yes|all ]] &&
printf "%(%s)T $HOSTNAME %s[%5s]: %-15s : BAN-LISTED : Expires $(date --date @${line[1]})\n" -1 ${0##*/} $$ ${line[0]} >> $log
$iptables -w -I $drop_chain -s ${line[0]} -j $ban_policy
fi
done`)
else
printf "%(%s)T $HOSTNAME %s[%5s]: No banlist at $ban_file\n" -1 ${0##*/} $$ >> $log
fi
if [[ "$verbose" =~ xt_recent|yes|all ]]; then
for i in /proc/net/xt_recent/*
do for j in $(cat $i)
do [[ $j =~ src= ]] &&
printf "%(%s)T $HOSTNAME %s[%5s]: %-15s : $i\n" -1 ${0##*/} $$ ${j:4} >> $log
done
done
fi
}
# --------------------- PROCESS MATCHING HITS ----------------------
act_on_hit() {
# Build associative arrays ${hit_history[$type]} of "IP +date" pairs, one array for each type of hit
# Associative array hit_history[$type] is transposed into indexed array $flusher[] to trim
# aged entries, then transposed back to associative array until the next hit
# Default $hit_expire value is 1day, default $max_hits value is 10
nowtime=$(printf "%(%s)T" -1)
hit_history[$type]="${hit_history[$type]} $src_ip `date --date=+${hit_life[$type]:=$hit_expire} +%s`"
flusher=( ${hit_history[$type]} )
while [ ${flusher[1]} -lt $nowtime ]
do
flusher=( ${flusher[@]:2} )
done
hit_history[$type]="${flusher[@]}"
hit_count=0
for i in ${flusher[@]}; do [[ $i == $src_ip ]] && ((hit_count++)); done
printf "%(%s)T $HOSTNAME %s[%5s]: %-15s : %-21s %2s vs. %-2s\n" \
-1 ${0##*/} $$ $src_ip "$hit_string" $hit_count ${max_hits[$type]:=$hit_limit} >> $log
# Stop if offending IP is already in $BANNED_LIST
# If `cutter` exists, run it to sever connection with that IP
if [ $hit_count -ge ${max_hits[$type]:=$hit_limit} ]; then
if [[ "$BANNED_LIST" =~ "$src_ip" ]]; then
:
else
[ -x "$cutter" ] && $cutter $src_ip
BANNED_LIST+=$src_ip
ban_time=${ban_time[$type]:=$ban_expire}
# Increase ban_time if offending src_ip already appears in $ban_log (permanent record)
# ban_time set at 1 month for each appearance. Already in ban_log twice? 2month ban.
if [ -r $ban_log ]; then
mapfile <$ban_log ban_log_array
ban_count=0
for (( i = 0 ; i < ${#ban_log_array[@]} ; i++ ))
do
line=( ${ban_log_array[$i]} )
[[ "${line[@]}" =~ $src_ip.*BANNED ]] && ((ban_count++))
done
[[ $ban_count -ge 2 ]] && ban_time=${ban_count}months
fi
# Add offending IP to $BANNED_LIST
# Add offending IP with ban expiration to $ban_file
# Add Notice of ban and triggering $log_action line to permanent record ($ban_log)
expire_date="$(date --date=+$ban_time)"
printf "%(%s)T $HOSTNAME %s[%5s]: %-15s : BANNED : Expires $expire_date\n" -1 ${0##*/} $$ $src_ip >> $log
printf "%-15s $(date --date=+$ban_time +%s)\n" $src_ip >> $ban_file
printf "%(%s)T $HOSTNAME %s[%5s]: %-15s : BANNED : Banned from %(%c)T to $expire_date\n" -1 ${0##*/} $$ $src_ip -1 >> $ban_log
printf "$log_action\n\n" >> $ban_log
# Restore $drop_chain and its contents but only if firewall has been modified
$iptables -C $from_chain -j $drop_chain 2> /dev/null
if [ "$?" -gt "0" ]; then
printf "%(%s)T $HOSTNAME %s[%5s]: Firewall was modified, rebuilding banlist from $ban_file ...\n" -1 ${0##*/} $$ >> $log
build_iptables_rules
fi
# Insert a firewall rule to REJECT/DROP the offending IP address
$iptables -w -I $drop_chain -s $src_ip -j $ban_policy
fi # End condition of being currently banned
fi # End condition of reaching $max_hits[$type]
}
# ----------------- DETECT MATCHING HITS --------------------
# Handle activity that appears on stdin, via syslog-ng
# A variety of log messages appear, a mix of iptables.log, auth.log, mail.log and mail.warn
# Traps facilitiate interaction with firewall builder and routine flushing of expired bans
# e.g. /etc/cron.daily/logrotate sends a SIGUSR1 to flush expired bans
# /etc/iptables/establish-iptables sends a SIGUSR1 to rebuild banned IP firewall chain
read_from_stdin() {
trap "verbose=bans build_iptables_rules" SIGUSR1
trap "verbose=all build_iptables_rules" SIGUSR2
while :
do
if read log_action; then
[[ "$log_action" =~ $ipregex ]] && src_ip=$BASH_REMATCH
if [ "$src_ip" == "" ]; then
:
elif [[ "$whitelist" =~ "$src_ip" ]]; then
:
else
unset hit_string
for i in "${!watch_log_@}"
do
if [[ "$log_action" =~ ${!i} ]]; then
hit_string="$BASH_REMATCH"
type=${i:10}; type=${type//[0-9]/}
act_on_hit
break
fi
done # end loop through $watch_log_* search-for-match strings
fi # end nested conditional - $log_action contains an IP, not in whitelist
# May 28, 2015 EDIT/ERROR FIX - added quote around variable "$hit_string"
[ "$hit_string" == "" ] &&
[[ "$verbose" =~ bans|yes|all ]] &&
printf "%(%s)T $HOSTNAME %s[%5s]: %-15s : no hit_string : $log_action\n" -1 ${0##*/} $$ $src_ip >> $log
fi # end nested conditional - receipt of $log_action line from stdin
done # end wait forever loop
}
# ---------------- END OF SUBROUTNES -------------------
# ----------------- ANNOUNCE STARTUP -------------------
trap exit_trap EXIT
PARENT_PID=`ps --no-headers -o ppid --pid $$`
PARENT_NAME=`ps --no-headers -o comm $PARENT_PID`
print_instructions
printf "%(%s)T $HOSTNAME %s[%5s]: Started by ${PARENT_NAME}[%5s]\n" -1 ${0##*/} $$ $PARENT_PID >> $log
if [[ "$verbose" =~ bans|yes|all ]]; then
for i in "${!watch_log_@}"
do
i=${i:10} ; i=${i//[0-9]/} ; [[ "$types" =~ $i ]] || types="$types $i"
done
for type in $types
do
printf "%(%s)T $HOSTNAME %s[%5s]: %-15s : %2s hits per %-10s -> ${ban_time[$type]:=$ban_expire} ban\n" \
-1 ${0##*/} $$ $type ${max_hits[$type]:=$hit_limit} ${hit_life[$type]:=$hit_expire} >> $log
done
if [ -x "$cutter" ]; then
printf "%(%s)T $HOSTNAME %s[%5s]: $cutter available\n" -1 ${0##*/} $$ >> $log
fi
fi
build_iptables_rules
read_from_stdin
# -- FINI
# -------------------------------------------------------------------------------------
# Similar functionality can be obtained with only syslog-ng and iptables.
# Persistence of banned entries during restart of firewall rules can be obtained by writing
# banned IP to a file, then reading that file when rebuilding the firewall.
#
# The below syslog-ng.conf and iptables routines don't distinguish
# between port or log violation, but could.
#
# Below routines use syslog-ng parser function to drive iptables blocking, without
# resorting to any interposing script.
#
# When an entry is added to /proc/net/xt_recent/syslog-scan, by syslog-ng detecting
# failed sshd login or similar, the firewall begins tracking rate of hits.
# Adapted from https://lists.balabit.hu/pipermail/syslog-ng/2011-February/015974.html
# ---- In syslog-ng.conf ----
# destination d_syslogblock
# { pseudofile("/proc/net/xt_recent/syslog-scan" template("+${usracct.device}\n"));
# file("/var/log/syslog-block"); };
# parser pattern_db { db_parser( file("/var/lib/syslog-ng/patterndb.xml")); };
# filter f_syslogblock { tags("secevt") and match("REJECT" value("secevt.verdict")); };
# log { source(src); parser(pattern_db); filter(f_syslogblock); destination(d_syslogblock); };
# ---- In firewall ----
#iptables -N syslog-block
#iptables -A syslog-block -m recent --name syslog-block --set -j DROP
#iptables -A INPUT -m conntrack --ctstate NEW \
# -m recent --name syslog-block --update --reap --seconds 3600 -j DROP
#iptables -A INPUT -m conntrack --ctstate NEW \
# -m recent --name syslog-scan --update --reap --seconds 900 --hitcount 15 -j syslog-block |
The script has been running here on a couple of machines, for a couple of weeks. I like the persistent bans that survive restarting the firewall. What hasn't happened yet is a repeat offender triggering an extended ban, but that section of code has been tested, and appears to work.
I learned a few bash tricks while composing this, for example bash equivalents to `grep -o -E`, `wc`, and `uniq`. That saved a few subshell instances, with the aim being to make the script about as efficient as a bash script can be. No doubt the script can be improved, and even though not many (if any!) people will use it, the ideas it contains may help readers better understand iptables and syslog-ng.
Edited the script to add quotes around a variable involved in a test. The function of logging non-matching log_action strings otherwise does not work. The other functions of the script aren't affected by the bug. Change is pretty well marked in the script, noted here too. Sorry about that. |
|
Networking & Security
