Shannon Entropy in URL Analysis: How Security Engines Detect DGA Domains and Obfuscated Malware

Signature-based blocklists fail against zero-day threats — a domain registered this morning has no blocklist entry yet. Domain Generation Algorithms (DGAs) exploit this window by producing thousands of algorithmically generated, short-lived domain names that botnet malware uses for command-and-control communication. To detect these domains before blocklists catch up, modern threat intelligence systems apply information theory: specifically, Shannon Entropy, which measures how random a string is and therefore how likely it is to be machine-generated rather than human-named.

1. Shannon Entropy — The Mathematical Foundation

Claude Shannon formalized information entropy in his 1948 paper "A Mathematical Theory of Communication." Shannon Entropy H(X) measures the average information (unpredictability) per character in a string:

H(X) = -∑ [ P(xᵢ) × log₂ P(xᵢ) ]

Where:

• xᵢ is each distinct character in the string
• P(xᵢ) is that character's frequency divided by string length
• log₂ gives the result in bits

A string where every character is identical (e.g., aaaaaa) has entropy of 0 — maximum predictability. A string where every character appears exactly once and no pattern exists approaches the theoretical maximum entropy of log₂(character_set_size). For lowercase alphanumeric domains (36 possible characters), that maximum is approximately 5.17 bits per character. Human-readable domain names cluster between 2.0 and 3.5 bits. DGA-generated names cluster between 3.6 and 4.8 bits.

2. Worked Examples — Calculating Entropy by Hand

Example 1: google.com (9 characters: g,o,o,g,l,e,.,c,o,m)

Character frequencies: g=2, o=3, l=1, e=1, .=1, c=1, m=1. Total=10 characters (including the dot).

H = -(3/10)×log₂(3/10) - (2/10)×log₂(2/10) - (1/10)×log₂(1/10) × 5
  = -(0.3×−1.737) - (0.2×−2.322) - (0.1×−3.322)×5
  ≈ 0.521 + 0.464 + 1.661
  ≈ 2.64 bits/char

Example 2: x9k2m7qzp1w8a.com (DGA candidate)

High character diversity, no repeated patterns. Entropy calculation yields approximately 3.78 bits/char — well above the human-language cluster.

Example 3: aHR0cHM6Ly9ldmlsLnh5ei9sb2dpbg== (Base64 encoded URL)

Base64 uses 64 characters with near-uniform distribution when encoding binary data. Entropy calculation: approximately 4.45 bits/char — the highest range, indicating obfuscated/encoded content.

3. Entropy Thresholds Used in Production Security Systems

A note on false positives: content delivery networks (CDNs) and cloud storage providers sometimes use high-entropy subdomains as edge cache keys (e.g., d1q9x8k2w7a3b.cloudfront.net). Production systems apply entropy analysis only to the registered domain label (the part before the TLD), not to CDN-assigned subdomains. They also whitelist known CDN apex domains before scoring.

4. Domain Generation Algorithms — How They Work

A DGA is an algorithm baked into malware that generates a large list of domain names from a shared seed — typically a date, a hardcoded value, or both. The malware tries each generated domain as a potential command-and-control (C2) server. The botnet operator registers one or a few of these domains on a given day, knowing the malware will eventually reach that domain in its list.

This model has three properties that make traditional blocklisting ineffective:

• Volume: A single DGA can generate 50,000+ domains per day. Registering them all to blocklist them costs more than the attacker pays to register one.
• Temporal evasion: The C2 domain rotates daily. Yesterday's blocklist entry is useless today.
• Plausible deniability: Newly registered domains have no reputation history — threat feeds have nothing to report on them until they appear in an incident.

Major DGA Malware Families and Their Entropy Profiles

TrickBot's word-list DGA is a notable exception — by concatenating common English words, it produces domains with natural-language entropy that evades pure entropy scoring. Modern detectors combine entropy with a lexical model trained on English word distribution to catch these.

5. Entropy Analysis on URL Paths and Query Parameters

Domain hostname entropy is only one application. Security scanners also apply entropy to:

• URL path segments:/login/auth/secure has low entropy (natural words). /9f2a1b7c8d3e4f5a has high entropy — likely a session token or hash used to identify specific victims in targeted phishing.
• Query string parameter values:?id=12345 is low entropy. ?token=aHR0cHM6Ly9ldmlsLnh5ei9sb2dpbg== is high entropy — the value is base64-encoded, potentially containing an encoded redirect URL.
• Subdomain labels: Attackers generate high-entropy subdomains for fast-flux DNS infrastructure. a9b2c7d1.attacker.com is a common pattern for per-victim phishing URLs that expire after first use.

6. JavaScript Implementation for Edge Workers

This is the entropy function as deployed in IncogSay's Cloudflare Workers edge runtime:

function calculateShannonEntropy(str) {
  if (!str || str.length === 0) return 0;

  const len = str.length;
  const charFreq = new Map();

  for (let i = 0; i < len; i++) {
    const char = str[i];
    charFreq.set(char, (charFreq.get(char) || 0) + 1);
  }

  let entropy = 0;
  for (const count of charFreq.values()) {
    const probability = count / len;
    entropy -= probability * Math.log2(probability);
  }

  return entropy;
}

// Evaluate only the registered domain label (not subdomains or TLD)
function scoreDomainEntropy(hostname) {
  // Extract registered domain: "api.evil-payload.com" -> "evil-payload"
  const parts = hostname.replace(/\.$/, '').split('.');
  const registeredLabel = parts.length >= 2 ? parts[parts.length - 2] : parts[0];

  const entropy = calculateShannonEntropy(registeredLabel);

  if (entropy >= 4.2)  return { risk: 'HIGH',   score: 85, reason: 'Base64/obfuscated payload signature' };
  if (entropy >= 3.8)  return { risk: 'MEDIUM',  score: 55, reason: 'DGA-candidate entropy range' };
  if (entropy >= 3.2)  return { risk: 'LOW',     score: 20, reason: 'Slightly elevated randomness' };
  return               { risk: 'CLEAN',  score: 0,  reason: 'Natural language entropy' };
}

7. Entropy Is One Signal, Not a Complete Verdict

Entropy analysis catches patterns that signature-based systems miss, but it has two important limitations that prevent it from being used as a standalone verdict:

• False positives on CDN and UUID-based domains: A domain like a1b2c3d4.vercel.app has high entropy but is a legitimate deployment URL. Entropy scoring must be combined with apex domain whitelisting and domain age data.
• Low-entropy DGAs: TrickBot and similar families use word-list concatenation to produce low-entropy domain names that evade entropy analysis. Catching these requires n-gram language models or machine learning trained on DGA family patterns.

IncogSay's URL Scanner combines Shannon entropy with Levenshtein distance brand matching, domain registration age lookup, SSL certificate age, threat intelligence feed queries, and redirect chain analysis. No single signal drives the verdict — each contributes a weighted component to the 0–100 threat score, and the final result shows which factors contributed so you can evaluate the reasoning, not just the number.