Agent Skills: Analyzing Linux Audit Logs for Intrusion

Analyzing Linux Audit Logs for Intrusion

When to Use

• Investigating suspected unauthorized access or privilege escalation on Linux hosts
• Hunting for evidence of exploitation, backdoor installation, or persistence mechanisms
• Auditing compliance with security baselines (CIS, STIG, PCI-DSS) that require system call monitoring
• Reconstructing a timeline of attacker actions during incident response
• Detecting file tampering on critical system files such as /etc/passwd, /etc/shadow, or SSH keys

Do not use for network-level intrusion detection; use Suricata or Zeek for network traffic analysis. Auditd operates at the kernel level on individual hosts.

Prerequisites

• Linux system with auditd package installed and the audit daemon running (systemctl status auditd)
• Root or sudo access to configure audit rules and query logs
• Audit rules deployed via /etc/audit/rules.d/*.rules or loaded with auditctl
• Recommended: Neo23x0/auditd ruleset from GitHub for comprehensive baseline coverage
• Familiarity with Linux syscalls (execve, open, connect, ptrace, etc.)
• Log storage with sufficient retention (default location: /var/log/audit/audit.log)

Workflow

Step 1: Verify Audit Daemon Status and Configuration

Confirm the audit system is running and check the current rule set:

# Check auditd service status
systemctl status auditd

# Show current audit rules loaded in the kernel
auditctl -l

# Show audit daemon configuration
cat /etc/audit/auditd.conf | grep -E "log_file|max_log_file|num_logs|space_left_action"

# Check if the audit backlog is being exceeded (dropped events)
auditctl -s

If the backlog limit is being reached, increase it:

auditctl -b 8192

Step 2: Deploy Intrusion-Focused Audit Rules

Add rules that target common intrusion indicators. Place these in /etc/audit/rules.d/intrusion.rules:

# Monitor credential files for unauthorized reads or modifications
-w /etc/passwd -p wa -k credential_access
-w /etc/shadow -p rwa -k credential_access
-w /etc/gshadow -p rwa -k credential_access
-w /etc/sudoers -p wa -k privilege_escalation
-w /etc/sudoers.d/ -p wa -k privilege_escalation

# Monitor SSH configuration and authorized keys
-w /etc/ssh/sshd_config -p wa -k sshd_config_change
-w /root/.ssh/authorized_keys -p wa -k ssh_key_tampering

# Monitor user and group management commands
-w /usr/sbin/useradd -p x -k user_management
-w /usr/sbin/usermod -p x -k user_management
-w /usr/sbin/groupadd -p x -k user_management

# Detect process injection via ptrace
-a always,exit -F arch=b64 -S ptrace -F a0=0x4 -k process_injection
-a always,exit -F arch=b64 -S ptrace -F a0=0x5 -k process_injection
-a always,exit -F arch=b64 -S ptrace -F a0=0x6 -k process_injection

# Monitor execution of programs from unusual directories
-a always,exit -F arch=b64 -S execve -F exe=/tmp -k exec_from_tmp
-a always,exit -F arch=b64 -S execve -F exe=/dev/shm -k exec_from_shm

# Detect kernel module loading (rootkit installation)
-a always,exit -F arch=b64 -S init_module -S finit_module -k kernel_module_load
-a always,exit -F arch=b64 -S delete_module -k kernel_module_remove
-w /sbin/insmod -p x -k kernel_module_tool
-w /sbin/modprobe -p x -k kernel_module_tool

# Monitor network socket creation for reverse shells
-a always,exit -F arch=b64 -S socket -F a0=2 -k network_socket_created
-a always,exit -F arch=b64 -S connect -F a0=2 -k network_connection

# Detect cron job modifications (persistence)
-w /etc/crontab -p wa -k cron_persistence
-w /etc/cron.d/ -p wa -k cron_persistence
-w /var/spool/cron/ -p wa -k cron_persistence

# Monitor log deletion or tampering
-w /var/log/ -p wa -k log_tampering

Reload rules after editing:

augenrules --load
auditctl -l | wc -l   # Confirm rule count

Step 3: Search for Intrusion Indicators with ausearch

Use ausearch to query the audit log for specific events:

# Search for all failed login attempts in the last 24 hours
ausearch -m USER_LOGIN --success no -ts recent

# Search for commands executed by a specific user
ausearch -ua 1001 -m EXECVE -ts today

# Search for all file access events on /etc/shadow
ausearch -f /etc/shadow -ts this-week

# Search for privilege escalation via sudo
ausearch -m USER_CMD -ts today

# Search for kernel module loading events
ausearch -k kernel_module_load -ts this-month

# Search for processes executed from /tmp (common attack staging)
ausearch -k exec_from_tmp -ts this-week

# Search for SSH key modifications
ausearch -k ssh_key_tampering -ts this-month

# Search for a specific event by audit event ID
ausearch -a 12345

# Search events in a specific time range
ausearch -ts 03/15/2026 08:00:00 -te 03/15/2026 18:00:00

# Interpret syscall numbers and format output readably
ausearch -k credential_access -i -ts today

Step 4: Generate Summary Reports with aureport

Use aureport to produce aggregate summaries for triage:

# Summary of all authentication events
aureport -au -ts this-week --summary

# Report of all failed events (login, access, etc.)
aureport --failed --summary -ts today

# Report of executable runs
aureport -x --summary -ts today

# Report of all anomaly events (segfaults, promiscuous mode, etc.)
aureport --anomaly -ts this-week

# Report of file access events
aureport -f --summary -ts today

# Report of all events by key (maps to your custom rule keys)
aureport -k --summary -ts this-month

# Report of all system calls
aureport -s --summary -ts today

# Report of events grouped by user
aureport -u --summary -ts this-week

# Detailed time-based event report for timeline building
aureport -ts 03/15/2026 08:00:00 -te 03/15/2026 18:00:00 --summary

Step 5: Reconstruct the Attack Timeline

Combine ausearch queries to build a chronological narrative:

# Step 5a: Identify the initial access timestamp
ausearch -m USER_LOGIN -ua 0 --success yes -ts this-week -i | head -50

# Step 5b: Trace what the attacker did after gaining access
# Get all events from the compromised account within the incident window
ausearch -ua <UID> -ts "03/15/2026 14:00:00" -te "03/15/2026 18:00:00" -i \
  | aureport -f -i

# Step 5c: Extract all commands executed during the incident window
ausearch -m EXECVE -ts "03/15/2026 14:00:00" -te "03/15/2026 18:00:00" -i

# Step 5d: Check for persistence mechanisms installed
ausearch -k cron_persistence -ts "03/15/2026 14:00:00" -i
ausearch -k ssh_key_tampering -ts "03/15/2026 14:00:00" -i

# Step 5e: Check for lateral movement (outbound connections)
ausearch -k network_connection -ts "03/15/2026 14:00:00" -i

Step 6: Forward Audit Logs to SIEM

Configure audisp-remote or auditbeat to ship logs to a central SIEM for correlation:

# Option A: Using audisp-remote plugin
# Edit /etc/audit/plugins.d/au-remote.conf
active = yes
direction = out
path = /sbin/audisp-remote
type = always

# Configure remote target in /etc/audit/audisp-remote.conf
remote_server = siem.internal.corp
port = 6514
transport = tcp

# Option B: Using Elastic Auditbeat
# Install auditbeat and configure /etc/auditbeat/auditbeat.yml
# Auditbeat reads directly from the kernel audit framework

Key Concepts

| Term | Definition | |------|------------| | auditd | The Linux Audit daemon that receives audit events from the kernel and writes them to /var/log/audit/audit.log | | auditctl | Command-line utility to control the audit system: add/remove rules, check status, set backlog size | | ausearch | Query tool that searches audit logs by message type, user, file, key, time range, or event ID | | aureport | Reporting tool that generates aggregate summaries of audit events for triage and compliance | | audit rule key (-k) | A user-defined label attached to an audit rule, enabling fast filtering of related events with ausearch and aureport | | syscall auditing | Kernel-level monitoring of system calls (execve, open, connect, ptrace) that captures process and file activity | | augenrules | Utility that merges all files in /etc/audit/rules.d/ into /etc/audit/audit.rules and loads them into the kernel |

Verification

• [ ] auditd is running and rules are loaded (auditctl -l returns expected rule count)
• [ ] No audit backlog overflow (auditctl -s shows backlog: 0 or low value, lost: 0)
• [ ] ausearch returns events for each custom key (ausearch -k <key> -ts today returns results)
• [ ] aureport generates non-empty summaries for authentication, executable, and file events
• [ ] Timeline reconstruction produces a coherent chronological sequence of attacker actions
• [ ] Critical file watches trigger alerts on test modifications (touch /etc/shadow generates an event)
• [ ] Logs are forwarding to central SIEM (verify with a test event and confirm receipt)
• [ ] Audit rules persist across reboot (rules in /etc/audit/rules.d/, not only via auditctl)