SIEM with Splunk – Home SOC

Project overview

This project turns my home server into a small SOC: Splunk ingests Linux logs (auth.log, syslog, UFW, Apache, MySQL), detects SSH brute force attempts and automatically blocks the source IP in ufw, logging everything for audit.

The focus is on three pillars: visibility (dashboards), detection (SPL + Threat Intel) and response (scripts and alerts).

High-level architecture

• Inputs: linux_secure (SSH), syslog, access_combined (Apache), mysql_error_log.
• Dashboards:
  • Main Dashboard: attack overview (SSH/HTTP), banned IPs, Threat Intel.
  • Advanced Center: Splunk health, MySQL errors, top sourcetypes, audit views.
• Auto-response: brute-force alert → lookup → ban_ip.sh → UFW rule + syslog log.
• Threat Intel: lookup threatintel_by_ip.csv matches external IPs against public feeds.

SSH brute force detection (alert SPL)

Saved search that feeds the “safe” lookup of IPs to be banned:

| search index=* sourcetype=linux_secure "sshd" (Failed OR failure OR "Invalid user")
| rex field=_raw "from (?<src_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"
| where isnotnull(src_ip)
| streamstats time_window=1m count as consecutive_failures by src_ip
| where consecutive_failures >= 5
| fields src_ip
| dedup src_ip
| outputlookup ips_a_banir.csv

The search runs every minute as a scheduled alert. Instead of passing parameters to the script, it writes the IPs into ips_a_banir.csv, a simple “vault” read by the ban script.

Automatic UFW BAN (ban_ip.sh)

Script triggered by the Run a script action; it reads the vault, applies whitelist, avoids duplicates, and writes a history log:

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

DEBUG_LOG="/opt/splunk/var/log/splunk/ban_script.log"
LOOKUP_FILE="/opt/splunk/etc/apps/search/lookups/ips_a_banir.csv"
WHITELIST="/opt/splunk/etc/apps/search/lookups/ip_whitelist.csv"
HISTORY="/opt/splunk/etc/apps/search/lookups/banned_history.csv"
UFW="/usr/sbin/ufw"
LOGGER="/usr/bin/logger"

exec 2>>"$DEBUG_LOG"

echo "----------------------------------------------------" >> "$DEBUG_LOG"
echo "$(date): Start ban_ip.sh. Reading: $LOOKUP_FILE" >> "$DEBUG_LOG"

mapfile -t WL < <(awk -F',' 'NR>1{print $1}' "$WHITELIST" | tr -d '\r"')

awk -F',' 'NR>1{print $1}' "$LOOKUP_FILE" | tr -d '\r"' | while read -r IP; do
  IP="$(echo "$IP" | xargs)"
  [[ -z "${IP:-}" ]] && continue
  [[ ! "$IP" =~ ^([0-9]{1,3}\.){3}[0-9]{1,3}$ ]] && {
    echo "$(date): Ignored (invalid IPv4): '$IP'" >> "$DEBUG_LOG"; continue; }

  for W in "${WL[@]:-}"; do
    if [[ "$IP" == "$W" ]]; then
      echo "$(date): Ignored (whitelist): $IP" >> "$DEBUG_LOG"
      continue 2
    fi
  done

  if /usr/bin/sudo $UFW status numbered | grep -qE "DENY IN +$IP"; then
    echo "$(date): Already banned: $IP" >> "$DEBUG_LOG"
    continue
  fi

  echo "$(date): Banning $IP via ufw..." >> "$DEBUG_LOG"
  /usr/bin/sudo $UFW insert 1 deny from "$IP" to any comment 'Splunk-Auto-Ban-SSH-Brute-Force' || {
    echo "$(date): Failed to ban $IP" >> "$DEBUG_LOG"
    continue
  }

  $LOGGER -t BAN_SCRIPT "action=ip_banned ip=$IP reason=ssh_bruteforce"
  echo "$(date): $IP banned." >> "$DEBUG_LOG"

  echo "$(date +%F\ %T),$IP,ban,ban_script" >> "$HISTORY"
done

printf 'src_ip\n' > "$LOOKUP_FILE"
chown splunk:splunk "$LOOKUP_FILE" "$HISTORY"

echo "$(date): End ban_ip.sh. Lookup cleared." >> "$DEBUG_LOG"
echo "----------------------------------------------------" >> "$DEBUG_LOG"
exit 0

sudoers configuration

Allows the splunk user to call only the required UFW commands, passwordless:

# /etc/sudoers.d/splunk_ufw
splunk ALL=(root) NOPASSWD:/usr/sbin/ufw insert 1 deny from * to any comment Splunk-Auto-Ban-SSH-Brute-Force
splunk ALL=(root) NOPASSWD:/usr/sbin/ufw status numbered

Risk score + Threat Intel (SPL)

Search powering the “Banned IPs – Geo + Reputation + Score” panel:

| multisearch
    [ search index=* sourcetype=ban_script
      | rex max_match=1 "(?<banned_ip>\d{1,3}(?:\.\d{1,3}){3})"
      | eval via="ban_script" ]
    [ search index=* sourcetype=syslog BAN_SCRIPT
      | rex max_match=1 "ip=(?<banned_ip>\d{1,3}(?:\.\d{1,3}){3})"
      | eval via="syslog" ]
| where isnotnull(banned_ip)
| stats latest(_time) as when values(via) as via by banned_ip
| iplocation banned_ip
| eval country=coalesce(Country,"Unknown"), city=coalesce(City,"")
| lookup threatintel_by_ip ip as banned_ip OUTPUTNEW ti_desc ti_feed threat_key
| eval rep_flag = if(isnotnull(ti_feed),1,0)
| eval geo_flag = if(country!="PT",1,0)
| eval base=30, rep=rep_flag*50, geo=geo_flag*10
| eval score=base+rep+geo
| convert ctime(when) as when timeformat="%Y-%m-%d %H:%M:%S"
| sort - score - when
| table when banned_ip country city ti_feed ti_desc threat_key via score