Alert Triage: Real Threats vs False Positives

Picture your first week in a real SOC. The queue shows 847 open alerts. Your lead drops a single sentence before walking away: "Clear what you can, escalate what you can't." No further instructions. The clock is running.

That scenario is not an edge case. It is Tuesday morning in most tier-1 SOCs around the world. The skill that separates analysts who thrive from those who burn out in six months is not knowing every MITRE technique by heart. It is the ability to triage: to look at an alert, gather the right evidence in the right order, and reach a defensible verdict before the next alert lands.

What Alert Triage Actually Is

Triage comes from the French verb trier, to sort. In medicine it means assigning priority to patients before any treatment begins. In a SOC it means the same thing: sorting the signal from the noise before you act.

The process has four stages, and they are sequential for a reason.

Stage one: queue intake. Before you open any individual alert, you scan the queue for patterns. Are there clusters of the same rule firing across multiple hosts? Is one source IP appearing in a dozen separate alerts? Pattern recognition at the queue level is faster than investigating each alert in isolation and often reveals whether you are looking at coordinated activity or a single misconfigured scanner.

Stage two: evidence assessment. Once you pick an alert to investigate, you pull all available context: the asset and its owner, the user account involved, the raw log payload, and historical baseline for that entity. You are building a picture, not confirming a hypothesis. The goal at this stage is to gather before you judge.

Stage three: verdict. Based on the evidence, you assign one of three labels. True positive (TP): the activity is malicious or clearly policy-violating. False positive (FP): the rule fired correctly but the activity is not malicious. Benign true positive (BTP): the activity is exactly what the rule describes, it is real, and it is not an attack. We will spend more time on that distinction below because it matters enormously.

Stage four: disposition. Your verdict determines the action. True positives escalate to tier 2 or trigger an incident. False positives get closed with a note that feeds back into rule tuning. Benign true positives get closed with documentation of why the activity is expected and authorized.

Missing any stage collapses the whole structure. Analysts who skip queue intake waste time investigating individual alerts that would have been obviously linked at the queue level. Analysts who skip evidence assessment jump straight to verdict and miss context that would have changed it. Analysts who conflate verdict categories create noise downstream and undermine escalation credibility.

A Practical Triage Framework

Good triage is not intuition. At tier 1, it is a structured checklist you run every time, and the checklist has four axes.

Severity. What is the detection rule's confidence and impact rating? A SIGMA rule tagged critical on a domain controller is not the same as a low rule on a developer workstation. Severity from the rule is a starting weight, not a final verdict, but it calibrates where you spend the most scrutiny.

Asset criticality. Who or what generated the event? A failed login on a shared printer differs from the same failed login on a database server that stores payment records. Your organization's CMDB or asset inventory maps hostnames and service accounts to their business function. If you do not have that mapping, your first week's side project should be building a cheat-sheet version of it.

IOC corroboration. Does the indicator of compromise appear in more than one data source? A suspicious domain that appears only in a DNS log is interesting. The same domain appearing in DNS, proxy, and firewall logs simultaneously is significant. Your threat intel feeds and the indicator lists published with CISA advisories give you a fast sanity-check on whether an IP address, hash, or domain has documented malicious history, and MITRE ATT&CK tells you which technique the behavior maps to. When the destination was registered three days ago none of that history exists yet, and the corroboration has to come from the shape of the request instead. That is what data encoding hands you for nothing: base64 padding sitting inside a DNS label, or a User-Agent string that decodes cleanly into a browser version, are properties of the traffic itself and stay readable whether or not anyone has ever reported the far end.

Baseline deviation. Is this activity normal for this user or host? A login at 3 AM from a European IP for an account that has only ever authenticated from Chicago during business hours is anomalous. A login at 3 AM from a developer who routinely works overnight is not. Baseline requires data: thirty days of history for the entity involved. Most SIEMs make this trivial to query. Analysts who skip the baseline check generate the most false escalations.

Run all four axes on every alert. Not in four hours per alert. In three to five minutes. The goal is not exhaustive investigation; that is what tier 2 is for. The goal is a calibrated preliminary verdict backed by evidence you can articulate.

True Positive vs. False Positive vs. Benign True Positive

These three labels are the vocabulary of triage, and confusing them is the most consequential mistake a new analyst makes.

A true positive is an alert that fired because something genuinely malicious or policy-violating happened. An attacker ran Mimikatz. A user exfiltrated files to a personal cloud drive in violation of policy. The detection worked correctly and the activity is a real problem.

A false positive is an alert that fired but the underlying activity is not what the rule was designed to catch. A vulnerability scanner doing authorized work trips a port-scan detection. A backup job generates "brute force" level authentication noise. The rule's logic is correct, but the specific instance is not an actual threat. False positives are where most of your queue lives, and learning to recognize them quickly without rubber-stamping them carelessly is the central tier-1 skill.

A benign true positive is the subtle one. The rule fired correctly. The activity is exactly what the rule describes. And it is not an attack. An employee using an approved remote access tool generates a legitimate-looking lateral movement alert. A sysadmin running an authorized credential audit trips a credential-dumping detection. A scheduled decommissioning job clears a server's Security audit log and fires a tier-1 log-tampering alert. The detection is technically correct. The activity is real. It just has a valid business explanation.

Getting these three categories right matters beyond the individual alert. Your escalation credibility with tier 2 depends on the precision of your TP calls. If you escalate ten alerts per shift and eight turn out to be BTPs, the tier-2 team will start ignoring your escalations. If you close everything that looks like a BTP and miss a real TP using the same pattern, the consequences are obvious.

Alert Fatigue: Why It Happens and How Pros Survive It

Staffing is where the strain shows first: in the SANS 2025 SOC Survey, 62% of respondents said their organization is not doing enough to retain its SOC staff. That is not a failure of individual analysts.

Alert fatigue has a structural cause: base rates. In most enterprise environments, the vast majority of alerts generated by detection rules are not true positives. This is true even in well-tuned SOCs with mature detection engineering. In Vectra AI's 2023 State of Threat Detection study of 2,000 SOC analysts, teams received an average of 4,484 alerts a day and 67% of them went uninvestigated. Analysts are not lazy; genuine investigation of every alert does not scale.

Understanding base rates means internalizing this: if your environment generates 200 alerts per shift and historically 3% are true positives, you should expect six real threats today. That expectation shapes how you allocate attention. It does not mean you close everything carelessly. It means you develop rapid, systematic filters that let you spend deep investigation time on the 6 that matter rather than equal time on all 200.

Experienced analysts survive alert fatigue through a combination of deliberate triage structure, aggressive feedback loops on false positives (every FP you close should feed a tuning ticket), and a strategy borrowed from signal detection theory called criterion shifting: when the queue is overwhelming, you raise the threshold of evidence required before escalation rather than escalating everything or closing everything. That is a conscious adaptation, not capitulation.

A Worked Example: Suspicious Login Alert from Queue to Verdict

The best way to understand triage in practice is to walk one alert through the full process. Here is a realistic alert payload as it might appear in a Splunk SIEM:

Alert: Suspicious Authentication Event
Severity: Medium
Rule: Multiple Failed Logins Followed by Success ([T1110.001](https://attack.mitre.org/techniques/T1110/001/))
Time: 2026-04-21 02:47:33 UTC
User: j.hartley@corp.internal
Source IP: 185.220.101.47
Destination: CORPDC01.corp.internal (Domain Controller)
Failed attempts: 14
Outcome: Authentication Succeeded on attempt 15

Stage one: queue intake. You scan around it. Three other alerts in the same window also show 185.220.101.47 as a source. One is a proxy alert for an access to an external file-sharing site. Two more are authentication failures against different accounts. That IP appearing across four separate detections in a twelve- minute window is a pattern. You flag this cluster for investigation. The file-sharing hit is the one to name, because exfiltration over a web service rides a sanctioned domain over a normal port and never looks wrong on volume alone. In this cluster it is corroborating detail; arriving alone it closes as noise.

Stage two: evidence assessment. You pull context on the user j.hartley:

User profile: Jessica Hartley, Accounts Payable
Manager: M.Torres
Normal auth pattern: Mon-Fri 08:00-18:00, source IPs in 10.44.0.0/16 (Chicago VPN)
Last login before this event: 2026-04-20 17:22 UTC (Chicago VPN, 10.44.18.22)
Current event: 02:47 UTC, IP 185.220.101.47
 
IP 185.220.101.47:
  ASN: Tor exit node operator (AS201814)
  Threat intel: Listed in AbuseIPDB with 847 recent reports
  CISA KEV match: None
  Geolocation: Netherlands

The source IP resolves to a Tor exit node with documented abuse history. The authentication time is outside every historical pattern for this user. The target is a domain controller, not a workstation. Fourteen failed attempts followed by success is consistent with a low-rate password spray that eventually matched. The fourteen failures are what made this catchable: replay authentication material you already stole, a hash or a Kerberos ticket, and the success arrives with no failures in front of it.

Stage three: verdict. This is a true positive. Four corroborating signals: Tor exit node source, time anomaly, historical pattern break, and domain controller target. Any one of these in isolation might be an FP. All four together produce a confident TP.

Stage four: disposition. Escalate immediately. Your escalation note documents the cluster pattern, the Tor exit node context, the geolocation deviation, and the exact query you used to surface the corroborating alerts. Tier 2 can act immediately without re-investigating ground you already covered.

This entire process, from opening the alert to submitting the escalation, should take roughly ten minutes for a clear TP with good context. If you are spending forty minutes on a single alert at tier 1, you are investigating, not triaging.

Common Rookie Triage Mistakes

Every analyst makes these. Knowing them in advance shortens the learning curve.

Investigating in isolation. Opening one alert and treating it as independent without first checking the queue for related activity. Correlation at the queue level is faster and surfaces incident-level patterns that individual alert investigation misses.

Trusting severity labels without context. A critical alert on a decommissioned server that still sends logs is not more urgent than a medium alert on a production database. Rule severity is calibrated to general environments, not your specific asset landscape.

Closing without documenting. Closing a false positive with no notes means the next analyst (or future you) has no record of why that activity was benign. Good triage notes are an institutional memory asset. NIST SP 800-61 Rev. 3 builds incident response on CSF 2.0, whose outcome RS.AN-06 asks that actions performed during an investigation be recorded, with their integrity and provenance preserved. A triage verdict is one of those actions.

Escalation to avoid the decision. Escalating alerts you have not triaged because you are uncertain feels conservative but is actually a failure mode. Tier-2 analysts have a higher skill ceiling, not a different job function. Your role is to filter noise with the tools and data available to you. An unexplained escalation consumes tier-2 capacity without adding value.

Confusing speed with sloppiness. Fast triage is good. Fast triage that skips the baseline check is not fast. It creates rework downstream when the verdict is wrong. A three-minute triage with all four checklist axes covered beats a thirty-second gut-feel close every time.

Where the Framework Becomes Judgment

A framework tells you which axes to check. Judgment tells you how much weight to put on each one when they disagree, and that calibration only develops by seeing the same patterns resolve, correctly and incorrectly, across hundreds of alerts.

One BTP that reliably catches new analysts off-guard is a sysadmin running PingCastle against their own Active Directory. The tool issues LDAP queries at volume, touches high-value group memberships, and generates a stream of events that maps almost perfectly onto a T1069.002 domain-enumeration pattern. Teams that have not documented the quarterly audit cycle will escalate it. Teams that have escalated it twice, and twice received "authorized scan" back from tier 2, eventually write the BTP entry. After that, the third occurrence is a five-second recognition rather than a forty-minute investigation. That is the whole mechanism: document the first occurrence properly, ideally as a reusable playbook, and every recurrence becomes faster. The habit only forms through reps on realistic material.

A live production SOC is a poor place to acquire those reps: the alerts are real, a missed true positive has consequences, and the clock does not pause for you to think. SOCSimulator's shift mode closes that gap: realistic SIEM, XDR, and firewall alerts at realistic volume and sequence, scored against the same verdict framework above, so you meet the base rate and build the baseline-check reflex before it can cost you anything.

The detections feeding that queue are themselves worth understanding from the other side. Our breakdown of 10 SIEM use cases every SOC runs shows the logic behind the rules you will be triaging. Phishing reports are one of the most common alert types you will triage, and learning to read the lures pays off fast: 15 phishing email examples breaks down the recurring tells side by side. From there, the same triage confidence carries into a tier-1 SOC role, a SOC analyst interview, and a phishing email analysis investigation alike.

The framework is here. The queue is waiting.