MASTER QUICK REFERENCE
Splunk's Search Processing Language (SPL) is the language of SIEM. Every SOC analyst, threat hunter, and IR professional needs to know it cold. This is your god-level reference for interviews and real-world investigations.
// GURU TIP — HOW SPL PIPELINES WORK
SPL works like Unix pipes. Each command passes results to the next via | . The search always starts with index= or sourcetype= to narrow data first — then transform, filter, and visualize. Time range is your most powerful filter — always set it. Never run a wildcard search against all time on production.
BASIC SEARCH STRUCTURE
index=Target data index
sourcetype=Data source format
host=Filter by hostname
source=Filter by log file/input
earliest= latest=Time range bounds
| head NFirst N results
| tail NLast N results
MOST USED COMMANDS
| statsAggregate statistics
| evalCreate/transform fields
| tableSelect fields to display
| whereFilter results
| sortOrder results
| dedupRemove duplicates
| rexExtract fields via regex
STATS FUNCTIONS
countCount events
dc(field)Distinct count
sum(field)Sum values
avg(field)Average
max(field)Maximum value
min(field)Minimum value
values(field)List distinct values
TIME MODIFIERS
earliest=-24hLast 24 hours
earliest=-7d@dLast 7 days
earliest=-1h latest=nowLast hour
earliest=@dStart of today
earliest=@wStart of week
earliest=@monStart of month
span=1h / span=5mBucket by time
BOOLEAN & WILDCARDS
AND OR NOTBoolean operators
field=value*Wildcard suffix
field=*value*Contains match
NOT field=valueExclude field value
field IN (a,b,c)List membership
"exact phrase"Phrase search
field!=valueNot equal
KEY SOURCETYPES
WinEventLog:SecurityWindows security logs
WinEventLog:SystemWindows system logs
syslogLinux/Unix syslog
access_combinedApache/web access logs
pan:trafficPalo Alto firewall
cisco:asaCisco ASA firewall
suricataSuricata IDS/IPS
SPL BASICS — SEARCH FUNDAMENTALS
The foundational commands every Splunk user must know. These form the backbone of every investigation query.
// GURU TIP
Always narrow with index= and sourcetype= before piping to transforms. Searching all indexes on all time is a performance disaster and will get you noticed by Splunk admins for the wrong reasons.
| COMMAND | SYNTAX | EXAMPLE | PURPOSE |
|---|---|---|---|
| Basic Search | index=[idx] sourcetype=[type] [keyword] | index=main sourcetype=syslog error index=windows EventCode=4624 | CRITICAL Foundation of every SPL query. Always specify index first for performance. Keywords are implicit AND. Splunk searches raw _raw field by default. |
| | search | | search [condition] | index=main | search status=failed index=web | search src_ip="10.*" | CRITICAL Post-pipeline filter. Filters results mid-pipeline. Unlike where, it can use wildcards and string patterns. Use after transforms to filter aggregated results. |
| | table | | table [field1] [field2] ... | index=windows EventCode=4625 | table _time, src_ip, user, host | CRITICAL Select and order fields for display. Removes all other fields from output. Essential for clean, readable results in dashboards and reports. Order of fields matters. |
| | where | | where [expression] | | where count > 10 | where src_ip != "192.168.1.1" | where isnotnull(username) | CRITICAL Filter using expressions and eval functions. More powerful than search for numeric comparisons and null checks. Use isnotnull() to exclude empty fields. |
| | sort | | sort [+/-][field] | sort -count | | sort -count | sort +_time | sort -bytes_out limit=20 | CRITICAL Sort results. - = descending, + = ascending. Default limit is 10,000 — add limit=0 to remove limit. Always sort your final table for readability. |
| | dedup | | dedup [field] | dedup [field1] [field2] | | dedup src_ip | dedup user host | dedup 3 src_ip | SOC Remove duplicate events. dedup 3 field keeps 3 copies. Use to get unique lists of IPs, users, or hosts without stat overhead. |
| | top | | top [limit=N] [field] | | top limit=10 src_ip | top limit=20 dest_port | top limit=5 user showperc=f | CRITICAL Top N most frequent values with count and percentage. showperc=f hides percentage column. Quick way to find most active IPs, users, or ports without a full stats command. |
| | rare | | rare [limit=N] [field] | | rare limit=10 user_agent | rare limit=5 dest_port | HUNT Least frequent values — the opposite of top. Gold for threat hunting: rare user agents, unusual ports, infrequent login times. Anomalies hide in the rare, not the common. |
| | head / tail | | head [N] | tail [N] | | head 100 | sort _time | tail 50 | Return first/last N events. tail returns most recent when sorted by time. Use for quick sampling without full result set loading. |
| | rename | | rename [old] AS [new] | | rename src_ip AS "Source IP" | rename EventCode AS "Event ID" | Rename fields for cleaner display. Essential when building dashboards or reports for non-technical stakeholders. Human-readable field names in tables. |
| | fields | | fields [+/-] [field1] [field2] | | fields src_ip, dest_ip, action | fields - _raw, _indextime | PRO Include (+) or exclude (-) fields. Faster than table — doesn't reorder, just filters. Removes _raw from results to reduce display size in large searches. |
STATS, TIMECHART & AGGREGATION
The most powerful SPL commands for analysis, trending, and detection. If you can write stats and timechart queries fluently, you can answer almost any SOC question.
// 🎯 INTERVIEW GOLD
"Write a SPL query to detect brute force logins." Answer: index=windows EventCode=4625 | stats count by src_ip, user | where count > 10 | sort -count. Follow with: "In production I'd add a timechart to see if it's sustained, and a lookup against a known-good IP list to reduce false positives."
| COMMAND | SYNTAX | EXAMPLE | PURPOSE |
|---|---|---|---|
| | stats count | | stats count by [field] | stats count as [name] by [field] | index=main | stats count by host index=windows EventCode=4625 | stats count as failures by src_ip | CRITICAL Count events grouped by field. The single most-used SPL command. as renames the count column. Multiple by fields for multi-dimensional grouping. |
| | stats dc() | | stats dc([field]) as [name] by [field] | index=vpn | stats dc(src_ip) as unique_ips by user index=dns | stats dc(query) as unique_domains by src_ip | CRITICALHUNT Distinct count — count unique values. Threat hunting gold: a user accessing many unique IPs = lateral movement. One IP querying many unique domains = C2 beaconing or DGA. |
| | stats values() | | stats values([field]) as [name] by [field] | | stats values(dest_ip) as destinations by src_ip | stats values(user) as users by src_ip | SOC List all distinct values per group. See all IPs a host connected to, or all users who logged into a machine. Essential for lateral movement and account sharing detection. |
| | stats sum/avg | | stats sum([field]) by [field] | stats avg([field]) by [field] | | stats sum(bytes_out) as total_bytes by src_ip | stats avg(duration) as avg_dur by dest_ip | SOC Sum and average numeric fields. Exfiltration detection: sum bytes_out per src_ip, sort descending — top talkers are your suspects. Average duration anomalies reveal long-lived C2 connections. |
| | stats multiple | | stats count, dc(field), sum(field) by [field] | index=firewall | stats count, dc(dest_ip) as targets, sum(bytes) as total_bytes by src_ip | sort -total_bytes | PROCRITICAL Multiple aggregations in one stats command. Single query to get count + unique targets + total bytes per source. This is how pros write efficient investigation queries. |
| | timechart | | timechart span=[time] [func] by [field] | index=windows EventCode=4625 | timechart span=1h count by host index=firewall action=blocked | timechart span=5m count | CRITICALALERT Time-series aggregation for line charts. Shows trends over time. Attack detection pattern: spikes in failed logins, blocked connections, or DNS queries over time = automated attack. Add to dashboards. |
| | chart | | chart [func] over [field] by [field] | | chart count over hour by action | chart sum(bytes) over dest_country by protocol | PRO Two-dimensional chart — value over one field, split by another. More control than timechart for non-time axes. Use for protocol breakdown by country, action breakdown by hour. |
| | eventstats | | eventstats [func] as [name] by [field] | | eventstats avg(bytes) as avg_bytes by dest_ip | eval anomaly=if(bytes > avg_bytes*3, "YES","NO") | PROHUNT Like stats but adds results back to each event as new fields without removing original events. Enables per-event anomaly scoring — compare each event to its group average. |
| | streamstats | | streamstats [func] by [field] | | sort _time | streamstats count as login_count by user | where login_count > 5 | PRO Running statistics computed on events as they flow through the pipeline in time order. Running totals, session counting, detecting Nth occurrence. Advanced analytics for behavioral detection. |
EVAL, REX & FIELD MANIPULATION
Create calculated fields, extract data with regex, and transform values. These commands turn raw log data into actionable intelligence.
| COMMAND | SYNTAX | EXAMPLE | PURPOSE |
|---|---|---|---|
| | eval (basic) | | eval [newfield]=[expression] | | eval bytes_mb=bytes/1048576 | eval full_name=first_name." ".last_name | eval is_admin=if(group="admins",1,0) | CRITICAL Create new fields or modify existing ones. Math, string concatenation, conditionals. The most versatile SPL command — used in almost every advanced query. |
| | eval if() | | eval [field]=if([condition],[true],[false]) | | eval severity=if(count>100,"CRITICAL","LOW") | eval internal=if(cidrmatch("10.0.0.0/8",src_ip),"YES","NO") | CRITICALALERT Conditional field creation. cidrmatch() checks IP against CIDR range — key for internal vs external traffic classification. Build risk scoring with if() chains. |
| | eval case() | | eval [field]=case([cond1],[val1],[cond2],[val2],true(),[default]) | | eval risk=case( severity="critical",10, severity="high",7, severity="medium",4, true(),1) | CRITICALALERT Multi-condition evaluation. Risk scoring gold — assign numeric scores based on severity, category, or any field. The true() at the end is the default/else case. |
| | eval math | | eval [field]=[numeric expression] | | eval mb=round(bytes/1024/1024,2) | eval age_days=round((now()-_time)/86400,0) | eval pct=round(count/total*100,1) | SOC Math operations: +, -, *, /, pow(), round(), floor(), ceiling(). Convert bytes to MB/GB for readable exfiltration reports. Calculate event age from epoch timestamps. |
| | eval string | | eval [field]=upper/lower/substr/len/replace | | eval user=lower(username) | eval domain=substr(url,8,50) | eval clean=replace(field,"[^a-zA-Z0-9]","") | PRO String manipulation. Normalize usernames to lowercase before stats (avoids ADMIN vs admin counting twice). Extract domain from URL. Strip special chars for comparison. |
| | rex | | rex field=[field] "(?P<name>pattern)" | | rex field=_raw "user=(?P<username>\w+)" | rex field=url "https?://(?P<domain>[^/]+)" | rex field=cmd "(?P<exe>[^\s]+\.exe)" | CRITICALPRO Extract fields from raw text using named capture groups. When Splunk hasn't parsed a field you need. Extracts usernames, domains, executable names from unstructured logs. Named groups become field names. |
| | rex mode=sed | | rex mode=sed field=[field] "s/[old]/[new]/g" | | rex mode=sed field=username "s/\\\\[^\\\\]+\\\\//g" | rex mode=sed field=url "s/\?.*$//" | PRO Sed-style substitution. Remove domain prefixes from usernames (DOMAIN\user → user). Strip URL query strings for cleaner domain analysis. Normalize data before grouping. |
| | spath | | spath input=[field] path=[jsonpath] | | spath input=_raw path=user.name | spath input=json_field path=events{0}.action | PRO Extract fields from JSON or XML log data. Modern security tools (EDR, cloud logs) emit JSON — spath navigates the structure. Essential for AWS CloudTrail, CrowdStrike, and modern SIEM data. |
| | convert | | convert [func]([field]) as [name] | | convert ctime(_time) as readable_time | convert num(bytes) as bytes_num | convert auto(field) | Type conversion and formatting. ctime() converts epoch to human-readable time string. num() forces string-to-number conversion for math operations. |
LOOKUP, ENRICH & CORRELATION
Enrich events with threat intelligence, asset data, and user context. Lookups turn raw IPs into threat actor names, and usernames into department/role context.
// GURU TIP — LOOKUPS IN PRODUCTION SOC
The most powerful Splunk analysts aren't the ones who know the most commands — they're the ones who build the best lookup tables. A threat intel lookup that tags malicious IPs automatically makes every other query smarter. Build once, use everywhere.
| COMMAND | SYNTAX | EXAMPLE | PURPOSE |
|---|---|---|---|
| | lookup | | lookup [lookup_name] [input_field] OUTPUT [output_field] | | lookup threat_intel src_ip OUTPUT threat_category, reputation | lookup asset_list hostname OUTPUT owner, department, criticality | CRITICAL Enrich events from a lookup table (CSV or KV store). Match field value against lookup table, add context columns. Essential for threat intel enrichment, asset context, and user info. |
| | lookup OUTPUTNEW | | lookup [name] [field] OUTPUTNEW [field] | | lookup geo_ip src_ip OUTPUTNEW country, city, lat, lon | SOC OUTPUTNEW only adds the field if it doesn't already exist — prevents overwriting existing data. Use for GeoIP lookups where some IPs might already have location data from another source. |
| | inputlookup | | inputlookup [lookup_name] | inputlookup [lookup_name] where [condition] | | inputlookup threat_ip_list | inputlookup watchlist.csv where risk_score > 8 | HUNT Search a lookup table directly as a data source. Start a query FROM a threat intel list, not from events. Use to build watchlist-driven hunting: "find all events involving known-bad IPs." |
| | outputlookup | | outputlookup [lookup_name] | index=windows EventCode=4625 | stats count by src_ip | where count > 50 | outputlookup brute_force_ips.csv | PRO Write search results to a lookup table. Create dynamic threat lists from search results — automatically populate a blocklist of IPs that exceeded brute-force thresholds. |
| | append | | append [subsearch] | index=windows EventCode=4624 | append [search index=linux sourcetype=auth_log action=success] | stats count by user | PRO Combine results from two different searches. Correlate Windows and Linux logins in one unified user activity view. Slower than union — use | union for large datasets. |
| | join | | join [type=] [field] [subsearch] | index=firewall action=allowed | join src_ip [search index=threat_feeds | rename ip as src_ip] | table src_ip, dest_ip, threat_type | HUNT SQL-style join between two searches on a common field. Join firewall allowed traffic with threat intel to find connections to known-bad IPs that weren't blocked. Use sparingly — expensive operation. |
| | union | | union [subsearch1] [subsearch2] | | union [search index=windows EventCode=4624] [search index=linux sourcetype=secure action=accepted] | PRO More efficient than append for combining large result sets. All results combined into one pipeline. Use for cross-platform correlation — Windows + Linux + cloud authentication events in one view. |
| tstats | | tstats [func] from datamodel=[model] where [filter] by [field] | | tstats count from datamodel=Network_Traffic where All_Traffic.action=blocked by All_Traffic.src_ip, All_Traffic.dest_port | PROCRITICAL Accelerated stats using data models (CIM). 10-100x faster than regular stats on large datasets. Use on CIM-compliant data models: Network_Traffic, Authentication, Endpoint, Web. Production SOC standard. |
THREAT HUNTING QUERIES
Ready-to-run SPL queries for proactive threat hunting. These are the searches that find attackers who bypassed your preventive controls.
// 🎯 INTERVIEW GOLD
"How do you hunt for lateral movement in Splunk?" Answer: index=windows EventCode=4624 Logon_Type=3 | stats dc(host) as unique_hosts by user | where unique_hosts > 5 | sort -unique_hosts. Type 3 = network logon. One user hitting many hosts = lateral movement or compromised credential misuse.
| HUNT | SPL QUERY | THRESHOLD / INDICATOR | WHAT IT DETECTS |
|---|---|---|---|
| Brute Force Login | index=windows EventCode=4625 | stats count as failures by src_ip, user | where failures > 10 | sort -failures | failures > 10 in time window Same src_ip, many usernames = spray Same username, many src_ips = stuffing | CRITICAL Failed logins (4625) grouped by source. High count = brute force. Many users from one IP = password spray. Many IPs for one user = credential stuffing. Tune threshold per environment baseline. |
| Successful Login After Failures | index=windows (EventCode=4625 OR EventCode=4624) | stats count(eval(EventCode=4625)) as fails, count(eval(EventCode=4624)) as success by user, src_ip | where fails > 5 AND success > 0 | sort -fails | Failures followed by success = compromise High fail:success ratio | CRITICALIR The most important authentication query — brute force succeeded. Many 4625s followed by a 4624 from same source = account compromise. Correlates fail and success in one query. |
| Lateral Movement | index=windows EventCode=4624 Logon_Type=3 | stats dc(host) as unique_hosts, values(host) as host_list by user | where unique_hosts > 5 | sort -unique_hosts | Logon Type 3 = network logon unique_hosts > baseline = suspicious | CRITICALHUNT Network logons (Type 3) to many different hosts by single user = lateral movement. Normal users log into 1-3 hosts. Admins maybe 10. Compromised account or attacker moving laterally = many hosts in short window. |
| Pass-the-Hash Detection | index=windows EventCode=4624 Logon_Type=3 Logon_Process=NtLmSsp WorkstationName!="-" | stats count by user, src_ip, host | where count > 3 | NTLM auth + Logon Type 3 No Kerberos = no ticket = hash used | HUNT Pass-the-Hash uses NTLM auth (not Kerberos). NTLM network logons to multiple machines = PtH indicator. Legitimate domain auth uses Kerberos. NTLM at scale is anomalous in modern AD environments. |
| Impossible Travel | index=auth action=success | lookup geo_ip src_ip OUTPUT country | sort user _time | streamstats current=f last(country) as prev_country, last(_time) as prev_time by user | eval time_diff=(_time-prev_time)/3600 | where country!=prev_country AND time_diff < 2 | table user, prev_country, country, time_diff | Login from two countries within 2 hours Physical travel impossible | HUNTPRO Two logins from different countries within time window that makes physical travel impossible. Classic compromised credential or VPN/proxy usage indicator. Requires GeoIP lookup. |
| DNS Beaconing | index=dns | stats count, dc(query) as unique_q, avg(bytes) as avg_bytes by src_ip, dest_ip | where count > 100 AND unique_q < 5 | sort -count | High frequency, low unique domains Regular time intervals Long random subdomain names | HUNTCRITICAL C2 beaconing via DNS: high query count, few unique domains (repetitive callback), regular intervals. Combine with: | where len(query) > 50 to catch DNS tunneling with long encoded subdomains. |
| Unusual Process Execution | index=endpoint sourcetype=sysmon EventCode=1 | stats count by ParentImage, Image | where count < 5 | sort count | Rare parent-child process pairs cmd.exe spawning from Word/Excel PowerShell from unusual parents | HUNT Sysmon Event 1 = process creation. Rare parent-child combos = malicious execution. winword.exe → cmd.exe = macro malware. excel.exe → powershell.exe = spear phishing execution. |
| Large Data Exfiltration | index=firewall | stats sum(bytes_out) as total_out, sum(bytes_in) as total_in by src_ip, dest_ip | eval ratio=round(total_out/total_in,2) | where total_out > 100000000 OR ratio > 10 | sort -total_out | bytes_out > 100MB to external IP Out:In ratio > 10:1 = sending much more than receiving | HUNTCRITICAL Outbound/inbound ratio anomaly. Normal web browsing = more IN than OUT. Exfiltration = much more OUT than IN. Total bytes threshold catches bulk transfer. High ratio catches slow-and-low exfil. |
| New Admin Account | index=windows EventCode=4728 OR EventCode=4732 Group_Name IN ("Administrators","Domain Admins","Enterprise Admins") | table _time, user, Group_Name, SubjectUserName, host | sort -_time | EventCode 4728 = domain group add EventCode 4732 = local group add Privileged group membership changes | IRCRITICAL User added to privileged group = privilege escalation or persistence mechanism. Attackers create admin accounts for persistence. Alert on ANY addition to Administrators, Domain Admins, or Enterprise Admins. |
| PowerShell Encoded Commands | index=windows sourcetype="WinEventLog:Microsoft-Windows-PowerShell/Operational" EventCode=4104 | search ScriptBlockText="*-enc*" OR ScriptBlockText="*encodedcommand*" OR ScriptBlockText="*[Convert]::FromBase64*" | table _time, host, user, ScriptBlockText | -EncodedCommand flag = base64 payload FromBase64String = encoded execution Obfuscation to bypass detection | CRITICALHUNT Encoded PowerShell = almost always malicious. Legitimate admins don't encode commands. Event 4104 = PS script block logging (must be enabled). One of the highest-fidelity detections in Windows environments. |
CRITICAL WINDOWS EVENT IDs
Windows Event IDs are the SOC analyst's alphabet. Know these cold — they appear in every interview and every real investigation.
AUTHENTICATION (4600s)
4624Successful logon
4625Failed logon
4634Account logoff
4647User initiated logoff
4648Logon with explicit credentials
4672Special privileges assigned
4768Kerberos TGT request
4769Kerberos service ticket
4771Kerberos pre-auth failed
4776NTLM auth attempt
ACCOUNT MANAGEMENT (4700s)
4720User account created
4722User account enabled
4723Password change attempt
4724Password reset attempt
4725User account disabled
4726User account deleted
4728User added to global group
4732User added to local group
4756User added to universal group
4767User account unlocked
PROCESS & EXECUTION
4688Process created (native)
4689Process terminated
Sysmon 1Process creation (detailed)
Sysmon 3Network connection
Sysmon 7DLL loaded
Sysmon 8Remote thread created
Sysmon 11File created
PS 4103PS module logging
PS 4104PS script block logging
OBJECT ACCESS & POLICY
4663Object access attempt
4698Scheduled task created
4699Scheduled task deleted
4702Scheduled task updated
4719Audit policy changed
4946Firewall rule added
7045New service installed
4657Registry value modified
LOGON TYPE CODES
Type 2Interactive (local console)
Type 3Network (lateral movement!)
Type 4Batch (scheduled tasks)
Type 5Service (service accounts)
Type 7Unlock (screen unlock)
Type 8NetworkCleartext (dangerous!)
Type 9NewCredentials (runas)
Type 10RemoteInteractive (RDP)
Type 11CachedInteractive
SPLUNK QUERIES BY EVENT ID
index=win EventCode=4624 Logon_Type=10RDP logins
index=win EventCode=4698New scheduled tasks
index=win EventCode=7045New services (persistence)
index=win EventCode=4719Audit policy tampering
index=win EventCode=4769 Ticket_Options=0x40810000Kerberoasting
index=win EventCode=4771Kerberos pre-auth fail (AS-REP)
NETWORK & FIREWALL ANALYSIS
SPL queries for network traffic analysis, firewall log investigation, and detecting network-based attacks.
| USE CASE | SPL QUERY | EXAMPLE / THRESHOLD | WHAT IT FINDS |
|---|---|---|---|
| Top Talkers | index=firewall | stats sum(bytes_out) as bytes by src_ip | sort -bytes | head 20 | eval gb=round(bytes/1073741824,2) | Top 20 by outbound bytes GB-level transfers flagged | CRITICAL Find hosts sending most data outbound. Exfiltration candidates are at the top. Convert to GB for readability. Cross-reference with business justification — unauthorized bulk transfer = incident. |
| Port Scan Detection | index=firewall action=blocked | stats dc(dest_port) as ports_hit by src_ip | where ports_hit > 20 | sort -ports_hit | One src scanning >20 ports = scan High port count in short window | HUNTCRITICAL Distinct count of destination ports per source. Normal hosts hit a handful of ports. Scanners hit dozens or hundreds. Blocked traffic is most telling — attacker scanning perimeter will be blocked. |
| C2 Beaconing | index=firewall action=allowed | bucket span=1h _time | stats count by _time, src_ip, dest_ip | stats stdev(count) as jitter, avg(count) as rate by src_ip, dest_ip | where jitter < 2 AND rate > 5 | sort jitter | Low standard deviation = regular intervals Regular connections to external IP = C2 | HUNTPRO C2 beacons fire at regular intervals — low standard deviation in hourly counts. Low jitter + sustained rate to external IP = automated beacon. One of the most sophisticated network hunting queries. |
| Blocked Traffic Spike | index=firewall action=blocked | timechart span=5m count as blocked_count | where blocked_count > 1000 | Sudden spike in blocked traffic DDoS or scanning campaign start | SOC Sudden spike in blocked connections = attack start. DDoS, port scan campaign, or vulnerability exploitation attempt. Timechart reveals the pattern — sustained vs spike. |
| Outbound to Unusual Ports | index=firewall direction=outbound NOT dest_port IN (80,443,53,22,25,465,587,993,995) | stats count, dc(dest_ip) as unique_dest by src_ip, dest_port | where count > 5 | sort -count | Internal host connecting outbound on non-standard port Reverse shells often use 4444, 1234, 8888 | HUNT Outbound on unusual ports = C2, reverse shell, or policy violation. Exclude common legitimate ports. Find internal hosts beaconing out on Metasploit defaults (4444) or custom C2 ports. |
| DNS Exfiltration | index=dns query_type=A | eval domain_len=len(query) | where domain_len > 50 | stats count, avg(domain_len) as avg_len by src_ip, query | sort -count | Query length > 50 chars Subdomain is base64/hex encoded data | HUNTCRITICAL DNS tunneling uses long subdomains to encode exfiltrated data. Normal FQDNs rarely exceed 30-40 chars. 50+ character queries to one domain = data smuggling via DNS. |
| Firewall Rule Changes | index=firewall sourcetype=cisco:asa OR sourcetype=pan:config message_type="config-change" | table _time, user, host, change_detail | sort -_time | Any firewall rule modification Especially rules that open inbound access | IR Attackers modify firewall rules to allow inbound access for persistence or to open RDP/SSH. Any unexpected firewall change = potential compromise indicator. Alert on all firewall config changes. |
ALERT ENGINEERING & SAVED SEARCHES
Building production-grade detections in Splunk. The difference between a junior and senior analyst is building alerts that are precise — high signal, low noise.
// GURU TIP — ALERT QUALITY FRAMEWORK
Every alert you build should answer four questions: 1) What does it detect? 2) What's the false positive rate? 3) What's the baseline threshold? 4) What's the response action? Interviewers love when you talk about tuning, not just writing the initial query.
| DETECTION | PRODUCTION SPL | TUNE / THRESHOLD | DEPLOYMENT NOTES |
|---|---|---|---|
| Brute Force Alert | index=windows EventCode=4625 earliest=-15m latest=now | stats count as failures by src_ip, user, host | where failures >= 10 | eval alert="BRUTE_FORCE" | table _time, src_ip, user, host, failures, alert | Run every 5 min, window 15 min Threshold: 10 failures / 15 min Tune: exclude service accounts, exclude known scanners | CRITICAL Schedule as Splunk Alert: every 5 minutes, rolling 15-min window. Trigger condition: number of results > 0. Add lookup to whitelist known pentest IPs. Suppress for 1 hour after first fire to avoid alert fatigue. |
| Admin Group Change | index=windows EventCode IN (4728,4732,4756) Group_Name IN ("Administrators","Domain Admins","Enterprise Admins","Schema Admins") earliest=-5m latest=now | table _time, SubjectUserName, MemberName, Group_Name, host | Run every 5 min Any result = alert (zero tolerance) No tuning — every change must be reviewed | CRITICALALERT Zero-tolerance detection. Any addition to privileged groups should fire immediately. Severity: CRITICAL. Alert destination: SOC + CISO. This detection has essentially no false positives in well-managed environments. |
| New Local Admin | index=windows EventCode=4732 Group_Name="Administrators" NOT SubjectUserName IN ("SYSTEM","*$") earliest=-5m | lookup authorized_admins SubjectUserName OUTPUT approved | where isnull(approved) | table _time, SubjectUserName, MemberName, host | Filter out machine accounts (*$) Lookup against authorized admin list Alert on unapproved additions only | PRO Enhanced version using lookup to compare against approved admin list. Only alerts when unauthorized account is added. Shows lookup-driven alert engineering — reduces noise without missing real threats. |
| High-Volume DNS | index=dns earliest=-1h | stats count as queries, dc(query) as unique_q by src_ip | where queries > 1000 AND unique_q > 500 | eval threat=if(unique_q/queries > 0.9,"DGA_SUSPECT","HIGH_VOLUME") | sort -queries | 1000+ queries/hr = high volume unique_q/count > 0.9 = DGA suspect (random domains = each one unique) | ALERTPRO High ratio of unique-to-total queries = Domain Generation Algorithm (DGA) malware. DGA generates random domains to find C2 — nearly every domain is unique. The ratio calculation is a sophisticated DGA indicator. |
| After-Hours Activity | index=windows EventCode=4624 Logon_Type=2 earliest=-1h latest=now | eval hour=tonumber(strftime(_time,"%H")) | eval day=strftime(_time,"%A") | where (hour < 7 OR hour > 22) OR day IN ("Saturday","Sunday") | lookup vip_users user OUTPUT is_vip | where isnull(is_vip) | table _time, user, host, hour, day | Interactive logins outside 7am-10pm Weekend logins for non-VIP users Tune: exclude on-call staff, exclude VIPs | HUNT After-hours interactive logons for non-exempt users. Attackers often operate during off-hours to avoid detection. strftime extracts hour and day from timestamp. VIP lookup excludes legitimate after-hours workers. |
| Scheduled Task Created | index=windows EventCode=4698 earliest=-5m latest=now NOT TaskName IN ("\\Microsoft\\*","\\Adobe\\*","\\Google\\*") | eval cmd=lower(TaskContent) | where match(cmd,"powershell|cmd\.exe|wscript|cscript|mshta|regsvr32|rundll32") | table _time, user, TaskName, TaskContent, host | Non-Microsoft scheduled tasks Tasks running suspicious LOLBins Persistence via scheduled task | CRITICAL Persistence detection. Attackers create scheduled tasks to survive reboots. Filter out known-good Microsoft/vendor tasks. Flag any task running LOLBins (Living Off The Land Binaries) like PowerShell, mshta, regsvr32. |
🎯 INTERVIEW SCENARIOS
Real SPL questions from SOC analyst, threat hunter, and Splunk admin interviews. With full answers, reasoning, and production-grade query patterns.
| QUESTION | ANSWER QUERY | WHY THIS IS THE RIGHT ANSWER |
|---|---|---|
| "Write SPL to find all failed logins in the last 24 hours, grouped by user" | index=windows EventCode=4625 earliest=-24h | stats count as failures by user | sort -failures | head 20 | CRITICAL Shows index+EventCode filtering, time range, stats aggregation, sort, and result limiting. Mention you'd also add | where failures > 5 in production to cut noise, and potentially cross-reference with 4624 (successful) events. |
| "How would you detect a compromised account being used for lateral movement?" | index=windows EventCode=4624 Logon_Type=3 earliest=-24h | stats dc(host) as unique_hosts, values(host) as accessed_hosts by user | where unique_hosts > 5 | sort -unique_hosts | CRITICAL Logon Type 3 = network logon. One user accessing many hosts = lateral movement. dc() for count, values() for list. Mention establishing a baseline (normal users = 1-3 hosts) to tune the threshold appropriately per environment. |
| "What's the difference between stats and tstats?" | -- stats: searches raw events index=main | stats count by src_ip -- tstats: searches accelerated data models | tstats count from datamodel=Network_Traffic by All_Traffic.src_ip | CRITICAL tstats operates on pre-indexed, accelerated data models (CIM). 10-100x faster on large datasets. Requires data to be CIM-mapped. Use tstats for dashboards and production alerts. Use stats for ad-hoc searches on unstructured data. |
| "How do you build an alert for unauthorized after-hours access?" | index=windows EventCode=4624 Logon_Type=2 earliest=-1h | eval hour=tonumber(strftime(_time,"%H")) | where hour < 7 OR hour > 22 | lookup approved_afterhours user OUTPUT exempt | where isnull(exempt) | table _time, user, host | PRO Shows strftime for time extraction, eval for derived fields, lookup for whitelisting. Mention: schedule every 30 min, suppress for 4 hours after first fire, route to on-call SOC team. Shows you think about operationalizing, not just writing the query. |
| "Walk me through your process when you get an alert for suspicious PowerShell" | -- 1. Get full context index=windows EventCode=4104 host=[host] earliest=-1h | table _time, user, ScriptBlockText -- 2. Check parent process index=windows Sysmon EventCode=1 host=[host] Image="*powershell*" earliest=-1h | table _time, ParentImage, CommandLine, User | IRCRITICAL Shows IR methodology: get the full script (4104), check parent process (Sysmon 1 — what launched PowerShell?), check network connections (Sysmon 3), check files created (Sysmon 11). Shows you investigate, not just acknowledge alerts. |
| "How do you reduce false positives in a brute force detection?" | index=windows EventCode=4625 earliest=-15m | stats count as failures by src_ip, user | where failures >= 10 -- Add these to reduce FPs: | lookup service_accounts user OUTPUT is_service | where isnull(is_service) | lookup known_scanners src_ip OUTPUT is_scanner | where isnull(is_scanner) | where NOT match(user,"^\$$") | PROALERT Three FP reduction techniques: 1) Exclude service accounts (they fail auth legitimately), 2) Whitelist known pentest/scanner IPs, 3) Exclude machine accounts (end in $). Mention: review alert history, adjust threshold based on baseline, add asset context with lookup. |
| "What Splunk data models do you use and why?" | -- Authentication events | tstats from datamodel=Authentication -- Network traffic | tstats from datamodel=Network_Traffic -- Endpoint (processes, files, registry) | tstats from datamodel=Endpoint -- Web (HTTP proxy/IDS) | tstats from datamodel=Web -- Alerts (IDS/IPS/AV) | tstats from datamodel=Alerts | PROCRITICAL CIM (Common Information Model) data models normalize data across different sources. Authentication works for Windows, Linux, VPN, cloud. Network_Traffic works for Palo Alto, Cisco, pfSense. Mention: tstats is faster because data models are pre-indexed/accelerated by Splunk. |
| "How do you investigate a potential data exfiltration incident?" | -- Step 1: Find top outbound talkers index=firewall src_ip=[suspect] | stats sum(bytes_out) as total_out by dest_ip | sort -total_out -- Step 2: Timeline the activity index=firewall src_ip=[suspect] | timechart span=1h sum(bytes_out) -- Step 3: Identify destination | lookup geo_ip dest_ip OUTPUT country, org | where country != "US" | IRCRITICAL Show methodology: quantify (how much?), timeline (when did it start?), destination (where did it go?). Mention: check for staging (internal to internal large transfer before external), check for compression/archive activity on endpoint (Sysmon), check email for large attachments. |
// GURU — THE COMPLETE SPLUNK INVESTIGATION WORKFLOW
When given a Splunk environment to investigate: 1) Check index inventory — | eventcount summarize=false index=* → 2) Understand sourcetypes — index=* | stats count by sourcetype | sort -count → 3) Establish timeline — timechart to see when events started/spiked → 4) Pivot on affected entities (user/host/IP) → 5) Correlate across sourcetypes → 6) Build timeline → 7) Document with saved searches and notable events. That's how a senior analyst works.