Network Monitoring

SNMP Versions:
  v1: No security (community string in cleartext) — DEPRECATED
  v2c: Community string still in cleartext — DISCOURAGED
  v3: Authentication + encryption — RECOMMENDED

SNMP v3 Configuration (Cisco IOS):
  snmp-server group ADMIN v3 priv read ADMINVIEW write ADMINVIEW
  snmp-server user monitoring ADMIN v3 auth sha s3cur3Pa55 priv aes 256 An0th3rPa55
  snmp-server host 10.0.0.50 version 3 priv monitoring
  snmp-server enable traps syslog
  snmp-server enable traps bgp
  snmp-server enable traps config

Alert Severity Levels:

  P1 — Critical (Immediate response, 24/7):
    └─ Firewall down (no traffic flowing)
    └─ Core switch CPU > 90%
    └─ Link saturation > 95% for > 5 minutes
    └─ SIEM correlation triggered (indicator of compromise)

  P2 — High (Respond within 1 hour):
    └─ Backup link active (primary failed)
    └─ BGP session down
    └─ DNS resolution failure rate > 10%
    └─ Authentication failure spike > 5x baseline

  P3 — Medium (Respond within 8 hours):
    └─ Link utilization > 80% (capacity planning)
    └─ TLS cert expiring within 30 days
    └─ Device with high error rate > 1%
    └─ DHCP pool > 90% exhausted

  P4 — Low (Respond within 1 week):
    └─ Device firmware outdated
    └─ Non-critical device unreachable
    └─ SNMP polling errors
    └─ Small authentication failures

Mistake 1: Alerting on everything
  └─ "Interface Gi0/1 went up" — Who cares? This is noise.
  └─ Fix: Only alert on state changes that require action.

Mistake 2: Static thresholds that never adjust
  └─ "CPU > 80%" might be normal during business hours
  └─ Fix: Use dynamic baselines (e.g., Zabbix, Prometheus adaptive thresholds)

Mistake 3: No de-duplication
  └─ Down switch triggers alerts for all 48 connected devices
  └─ Fix: Alert correlation (parent-child dependency mapping)

Mistake 4: Alert fatigue
  └─ 1,000 alerts per day → analysts ignore them all
  └─ Fix: Tune relentlessly. Every alert should require action.

Mistake 5: No runbook
  └─ Alert fires. Nobody knows what to do.
  └─ Fix: Every alert has a linked runbook with response steps.

Weekly Traffic Baseline:
  └─ Peak hours vs off-peak (e.g., 2 Gbps at noon vs 200 Mbps at 2 AM)
  └─ Protocol distribution (80% HTTPS, 10% DNS, 5% SSH, 5% other)
  └─ Top talkers (which IPs consume the most bandwidth)
  └─ Top conversations (client-server pairs)
  └─ New connection rate per second
  └─ DNS query rate per second

Monthly Baseline:
  └─ Business day pattern vs weekend pattern
  └─ Patterns around month-end (financial reporting increases traffic)
  └─ Backup window traffic patterns

Quarterly Baseline:
  └─ Organic growth rate (traffic increases 5% per quarter)
  └─ New application deployment traffic signatures

What Happened:
  └─ FireEye (Target's security vendor) detected the malware in September 2013
  └─ FireEye alerted Target's security team in Bangalore, India
  └─ The security team evaluated the alert and decided it was not a threat
  └─ FireEye continued alerting for weeks — each alert ignored
  └─ No automated action or escalation was triggered

Why Monitoring Failed:
  └─ No SIEM correlation (alerts were isolated, not aggregated)
  └─ Alert fatigue (too many alerts, too few analysts)
  └─ No escalation if alert not acknowledged within time window
  └─ No integration between detection tools and response workflow
  └─ Management had no visibility into alert volume or response metrics

What Should Have Happened:
  └─ Automated response: FireEye alerts → SIEM correlation → block traffic → page on-call
  └─ Escalation: Alert not reviewed within 15 minutes → escalate to senior analyst → then CISO
  └─ Metrics: Track time-to-acknowledge and time-to-respond for every alert
  └─ Runbooks: Every alert type has a documented response procedure

Network Devices ──SNMP──┐
Switches       ──NetFlow┼───> Monitoring Server ──> Alerting (PagerDuty/OpsGenie)
Firewalls      ──Syslog─┤       (Zabbix/Prometheus/  ──> Dashboard (Grafana)
Routers        ──SNMP──┘        LibreNMS)             ──> SIEM (Splunk/ELK)
Wireless APs   ──SNMP──┐                              ──> Ticketing (Jira/ServiceNow)
Load Balancers ──Syslog─┼───> Log Aggregator
Cloud APIs     ──API───┘       (ELK/Loki)