LLM-Powered Security Diff Analysis of Dependency Version Upgrades

The Problem

Renovate and Dependabot raise hundreds of dependency version bump PRs per month across a typical organisation’s repositories. The standard automation is: if CI passes, merge. Tests validate behaviour, security scanners catch known CVEs, and timely updates reduce exposure. The gap is in what neither CI nor vulnerability scanners examine: the semantic content of what changed.

curl’s change to require TLS 1.2 as a minimum did not break any tests in most consuming applications — but it changed the security posture of every HTTPS connection they made. OpenSSL’s padding oracle fix altered certificate verification error handling in ways that broke some TLS-pinning implementations. The axios SSRF patch added a hostname allowlist check that was absent before; integration tests caught the functional change but not its security significance. None of these changes appeared in CVE databases at merge time. All of them changed the security behaviour of applications that auto-merged the update.

Volume compounds the problem. An engineer reviewing 50 dependency bump PRs per week is not reading changelogs. The changelog is often buried behind feature announcements or absent entirely for libraries maintained without structured release notes.

LLM-based security diff analysis addresses this at the point where it is actionable — on the PR, before the merge — with structured output that distinguishes a low-risk patch bump from a version that silently changes cryptographic defaults. The LLM reads the changelog and optionally the commit diff, extracts security-relevant changes, and posts a structured comment in seconds.

Threat Model

Silent security-relevant default change. A library ships a new minor version that changes a default value for a security parameter: minimum TLS version, certificate verification depth, hash algorithm selection, HTTP redirect following behaviour. The change is intentional and beneficial — the maintainer is improving the library’s out-of-the-box security posture — but consuming applications may not expect it. Tests pass because the new default is behaviorally equivalent in the test environment (which may use the default TLS version, never issue redirects, or use simplified certificate chains). Production breaks, or more dangerously, production’s security model silently changes and nobody notices.

New transitive dependency with a known CVE. A library update adds a new transitive dependency. The new dependency has a published vulnerability that is not yet in the scanner database used by the CI pipeline (scanner databases lag behind CVE publication by hours to days). The CI pipeline’s vulnerability scan passes. The application ships with a vulnerable transitive dependency that was introduced by an auto-merged update.

Compromised minor version release. A library maintainer’s package registry account is compromised. The attacker publishes a new patch or minor version that is functionally identical to the previous version plus a small addition: an extra HTTP call to an attacker-controlled endpoint during library initialisation, or a secondary exfiltration path that triggers on a specific environment variable being present. Tests pass because the happy path is unchanged. A diff of the library’s source between the old and new version would show the addition, but no automated process is reading it.

Version upgrade introduces newly disclosed vulnerability. A dependency bump PR upgrades a package from a version that was published before a vulnerability was known to a version that includes the fix, but the update also introduces a regression that creates a new, unfiled vulnerability. The CVE scanner approves the new version (it contains no known CVEs), CI passes, and the new vulnerability ships. An LLM reviewing the commit diff between versions can flag the changed code for human review.

Hardening Configuration

Step 1: GitHub Actions Workflow for Dependency Bump PRs

# .github/workflows/dependency-security-review.yml
name: Dependency Security Review

on:
  pull_request:
    types: [opened, synchronize]

jobs:
  security-diff-analysis:
    # Only run on dependency bump PRs from Renovate or Dependabot
    if: |
      github.actor == 'renovate[bot]' ||
      github.actor == 'dependabot[bot]' ||
      contains(github.event.pull_request.labels.*.name, 'dependencies')
    runs-on: ubuntu-24.04

    permissions:
      pull-requests: write
      contents: read

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Install dependencies
        run: pip install anthropic requests packaging

      - name: Detect changed packages
        id: detect
        run: |
          python3 .github/scripts/detect-dep-changes.py \
            --base "${{ github.base_ref }}" \
            --head "${{ github.sha }}" \
            --output /tmp/changed-packages.json

          echo "packages=$(cat /tmp/changed-packages.json | base64 -w0)" >> "$GITHUB_OUTPUT"

      - name: Run security diff analysis
        id: analysis
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          python3 .github/scripts/security-diff-analysis.py \
            --packages /tmp/changed-packages.json \
            --output /tmp/security-analysis.json \
            --repo-token "$GITHUB_TOKEN"

          RISK=$(jq -r '.overall_risk' /tmp/security-analysis.json)
          echo "risk_level=$RISK" >> "$GITHUB_OUTPUT"

      - name: Post analysis comment to PR
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const analysis = JSON.parse(fs.readFileSync('/tmp/security-analysis.json', 'utf8'));

            const riskEmoji = {
              'low': ':white_check_mark:',
              'medium': ':warning:',
              'high': ':rotating_light:'
            }[analysis.overall_risk] || ':question:';

            let body = `## Dependency Security Review ${riskEmoji}\n\n`;
            body += `**Risk Level:** \`${analysis.overall_risk.toUpperCase()}\`\n\n`;
            body += `**Summary:** ${analysis.summary}\n\n`;

            for (const pkg of analysis.packages) {
              body += `### \`${pkg.name}\` ${pkg.old_version} → ${pkg.new_version}\n\n`;

              if (pkg.crypto_changes.length > 0) {
                body += `**Cryptographic Changes:**\n`;
                pkg.crypto_changes.forEach(c => { body += `- ${c}\n`; });
                body += '\n';
              }
              if (pkg.network_changes.length > 0) {
                body += `**Network Changes:**\n`;
                pkg.network_changes.forEach(c => { body += `- ${c}\n`; });
                body += '\n';
              }
              if (pkg.input_validation_changes.length > 0) {
                body += `**Input Validation Changes:**\n`;
                pkg.input_validation_changes.forEach(c => { body += `- ${c}\n`; });
                body += '\n';
              }
              if (pkg.permission_changes.length > 0) {
                body += `**Permission/Capability Changes:**\n`;
                pkg.permission_changes.forEach(c => { body += `- ${c}\n`; });
                body += '\n';
              }
            }

            await github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body
            });

      - name: Set PR status check
        if: steps.analysis.outputs.risk_level == 'high'
        run: |
          echo "[FAIL] High-risk dependency changes require security engineer review"
          exit 1

Step 2: Package Change Detection

#!/usr/bin/env python3
# .github/scripts/detect-dep-changes.py
# Detect which packages changed between base and head commits
import argparse
import json
import subprocess
import sys
import re
from pathlib import Path

LOCK_FILE_PARSERS = {
    "package-lock.json": "npm",
    "yarn.lock": "yarn",
    "Pipfile.lock": "pipenv",
    "poetry.lock": "poetry",
    "go.sum": "gomod",
    "Cargo.lock": "cargo",
    "Gemfile.lock": "bundler",
}

def parse_npm_diff(diff_text: str) -> list[dict]:
    changes = []
    # npm package-lock format: "node_modules/pkgname": {"version": "x.y.z"}
    pattern = re.compile(
        r'"node_modules/([^"]+)"[^}]*?"version":\s*"([^"]+)"',
        re.DOTALL
    )
    # Parse old (- lines) and new (+ lines) versions
    old_versions = {}
    new_versions = {}

    for line in diff_text.splitlines():
        if line.startswith('-') and '"node_modules/' in line:
            m = re.search(r'"node_modules/([^"]+)"', line)
            if m:
                pkg = m.group(1)
                v = re.search(r'"version":\s*"([^"]+)"', line)
                if v:
                    old_versions[pkg] = v.group(1)
        elif line.startswith('+') and '"node_modules/' in line:
            m = re.search(r'"node_modules/([^"]+)"', line)
            if m:
                pkg = m.group(1)
                v = re.search(r'"version":\s*"([^"]+)"', line)
                if v:
                    new_versions[pkg] = v.group(1)

    for pkg in set(old_versions) & set(new_versions):
        if old_versions[pkg] != new_versions[pkg]:
            changes.append({
                "name": pkg,
                "old_version": old_versions[pkg],
                "new_version": new_versions[pkg],
                "ecosystem": "npm"
            })

    return changes

def get_changed_packages(base_ref: str, head_sha: str) -> list[dict]:
    changed_files = subprocess.run(
        ["git", "diff", "--name-only", f"origin/{base_ref}...{head_sha}"],
        capture_output=True, text=True
    ).stdout.splitlines()

    packages = []
    for changed_file in changed_files:
        lock_name = Path(changed_file).name
        if lock_name not in LOCK_FILE_PARSERS:
            continue

        diff = subprocess.run(
            ["git", "diff", f"origin/{base_ref}...{head_sha}", "--", changed_file],
            capture_output=True, text=True
        ).stdout

        ecosystem = LOCK_FILE_PARSERS[lock_name]
        if ecosystem == "npm":
            packages.extend(parse_npm_diff(diff))
        # Additional parsers for other ecosystems follow the same pattern

    # Deduplicate
    seen = set()
    unique = []
    for pkg in packages:
        key = (pkg["name"], pkg["old_version"], pkg["new_version"])
        if key not in seen:
            seen.add(key)
            unique.append(pkg)

    return unique

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--base", required=True)
    parser.add_argument("--head", required=True)
    parser.add_argument("--output", required=True)
    args = parser.parse_args()

    packages = get_changed_packages(args.base, args.head)
    with open(args.output, "w") as f:
        json.dump(packages, f, indent=2)

    print(f"[*] Detected {len(packages)} changed packages")
    for pkg in packages:
        print(f"  {pkg['name']}: {pkg['old_version']} -> {pkg['new_version']}")

Step 3: Core LLM Security Analysis with Structured Output

#!/usr/bin/env python3
# .github/scripts/security-diff-analysis.py
import argparse
import json
import requests
import sys
import anthropic
from typing import Optional

client = anthropic.Anthropic()

SECURITY_REVIEW_PROMPT = """You are a security engineer reviewing a library upgrade for security-relevant changes.

Library: {name}
Old version: {old_version}
New version: {new_version}
Ecosystem: {ecosystem}

Changelog / release notes between versions:
---
{changelog}
---

{diff_section}

Review this upgrade and identify:

1. **crypto_changes**: Any changes to cryptographic algorithms, TLS settings, minimum protocol versions, certificate validation, hash functions, or random number generation. Include both strengthening and weakening changes.

2. **network_changes**: Any new outbound HTTP/HTTPS connections added, changes to allowed hostnames, proxy handling, redirect following, or timeout defaults.

3. **input_validation_changes**: Any changes to input sanitisation, HTML encoding, SQL escaping, path traversal checks, or size limits.

4. **permission_changes**: Any new OS-level permissions required, new file paths accessed, new environment variables read, or new system calls made.

5. **risk_level**: Overall risk assessment:
   - "low": Patch version, dependency bumps, documentation only, test-only changes
   - "medium": Minor version with new features, changed defaults, new optional capabilities
   - "high": Security-relevant default changes, new network access, removed validation, or anything that could change security behaviour silently

6. **summary**: One sentence describing the most security-relevant aspect of this upgrade.

Respond with valid JSON matching this exact schema:
{{
  "name": "{name}",
  "old_version": "{old_version}",
  "new_version": "{new_version}",
  "crypto_changes": ["list of changes, empty array if none"],
  "network_changes": ["list of changes, empty array if none"],
  "input_validation_changes": ["list of changes, empty array if none"],
  "permission_changes": ["list of changes, empty array if none"],
  "risk_level": "low|medium|high",
  "summary": "one sentence"
}}"""

def fetch_changelog(name: str, old_version: str, new_version: str,
                    ecosystem: str, github_token: str) -> str:
    """Attempt to fetch changelog between versions from GitHub releases API."""
    headers = {"Authorization": f"Bearer {github_token}",
               "Accept": "application/vnd.github+json"}

    # For npm packages, look up GitHub repo via registry
    if ecosystem == "npm":
        try:
            npm_meta = requests.get(
                f"https://registry.npmjs.org/{name}",
                timeout=10
            ).json()
            repo_url = npm_meta.get("repository", {}).get("url", "")
            github_path = repo_url.replace("git+https://github.com/", "").replace(".git", "")
        except Exception:
            return f"[Changelog unavailable for {name} — no GitHub repository found]"

        try:
            releases = requests.get(
                f"https://api.github.com/repos/{github_path}/releases",
                headers=headers, timeout=10
            ).json()

            changelog_parts = []
            for release in releases:
                tag = release.get("tag_name", "").lstrip("v")
                if tag == new_version or tag == old_version:
                    changelog_parts.append(
                        f"### {release['tag_name']}\n{release.get('body', 'No release notes')}"
                    )

            return "\n\n".join(changelog_parts) if changelog_parts else "[No matching GitHub releases found]"
        except Exception as e:
            return f"[GitHub API error: {e}]"

    return "[Changelog fetch not implemented for this ecosystem]"

def fetch_git_diff(name: str, old_version: str, new_version: str,
                   ecosystem: str, github_token: str) -> Optional[str]:
    """Fetch abbreviated git diff between tagged versions when changelog is absent."""
    if ecosystem != "npm":
        return None

    try:
        npm_meta = requests.get(f"https://registry.npmjs.org/{name}", timeout=10).json()
        repo_url = npm_meta.get("repository", {}).get("url", "")
        github_path = repo_url.replace("git+https://github.com/", "").replace(".git", "")

        headers = {"Authorization": f"Bearer {github_token}",
                   "Accept": "application/vnd.github+json"}

        # Get compare diff summary (not full diff — too large for LLM)
        compare = requests.get(
            f"https://api.github.com/repos/{github_path}/compare/v{old_version}...v{new_version}",
            headers=headers, timeout=15
        ).json()

        if "files" not in compare:
            return None

        # Summarise which files changed
        changed_files = [f["filename"] for f in compare.get("files", [])[:30]]
        file_summary = "\n".join(f"  - {f}" for f in changed_files)
        additions = compare.get("ahead_by", "?")
        deletions = compare.get("behind_by", "?")

        return f"Git diff summary ({additions} commits ahead):\nChanged files:\n{file_summary}"
    except Exception:
        return None

def analyse_package(pkg: dict, github_token: str) -> dict:
    name = pkg["name"]
    old_version = pkg["old_version"]
    new_version = pkg["new_version"]
    ecosystem = pkg.get("ecosystem", "unknown")

    changelog = fetch_changelog(name, old_version, new_version, ecosystem, github_token)
    diff_info = fetch_git_diff(name, old_version, new_version, ecosystem, github_token)

    diff_section = ""
    if diff_info:
        diff_section = f"Git commit comparison:\n---\n{diff_info}\n---"

    # Truncate changelog to avoid token limits
    changelog_truncated = changelog[:6000] if len(changelog) > 6000 else changelog

    prompt = SECURITY_REVIEW_PROMPT.format(
        name=name,
        old_version=old_version,
        new_version=new_version,
        ecosystem=ecosystem,
        changelog=changelog_truncated,
        diff_section=diff_section,
    )

    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}]
    )

    try:
        result = json.loads(response.content[0].text)
    except json.JSONDecodeError:
        # Extract JSON from response if wrapped in prose
        import re
        match = re.search(r'\{.*\}', response.content[0].text, re.DOTALL)
        result = json.loads(match.group(0)) if match else {
            "name": name, "old_version": old_version, "new_version": new_version,
            "crypto_changes": [], "network_changes": [], "input_validation_changes": [],
            "permission_changes": [], "risk_level": "medium",
            "summary": "Analysis failed — manual review required"
        }

    return result

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--packages", required=True)
    parser.add_argument("--output", required=True)
    parser.add_argument("--repo-token", required=True)
    args = parser.parse_args()

    with open(args.packages) as f:
        packages = json.load(f)

    # Skip patch-version bumps for non-security packages (cost optimisation)
    def should_analyse(pkg: dict) -> bool:
        old = pkg["old_version"].split(".")
        new = pkg["new_version"].split(".")
        # Always analyse major and minor bumps; skip patch for common safe packages
        safe_patch_only = {"lodash", "typescript", "eslint", "prettier"}
        if pkg["name"] in safe_patch_only and old[:2] == new[:2]:
            return False
        return True

    to_analyse = [p for p in packages if should_analyse(p)]
    print(f"[*] Analysing {len(to_analyse)}/{len(packages)} packages (skipping safe patch bumps)")

    results = []
    for pkg in to_analyse:
        print(f"[*] Analysing {pkg['name']} {pkg['old_version']} -> {pkg['new_version']}")
        result = analyse_package(pkg, args.repo_token)
        results.append(result)

    # Overall risk is the maximum risk across all packages
    risk_order = {"low": 0, "medium": 1, "high": 2}
    overall_risk = max(
        (r.get("risk_level", "low") for r in results),
        key=lambda r: risk_order.get(r, 0),
        default="low"
    )

    output = {
        "overall_risk": overall_risk,
        "packages": results,
        "summary": (
            f"{len([r for r in results if r.get('risk_level') == 'high'])} high-risk, "
            f"{len([r for r in results if r.get('risk_level') == 'medium'])} medium-risk, "
            f"{len([r for r in results if r.get('risk_level') == 'low'])} low-risk packages analysed."
        )
    }

    with open(args.output, "w") as f:
        json.dump(output, f, indent=2)

    print(f"[*] Analysis complete. Overall risk: {overall_risk}")

Step 4: Diff-Based Analysis When Changelog Is Absent

When a library publishes no release notes, the git commit log between tagged versions provides the raw material. Shallow-clone the repo, extract git log --oneline {old_tag}..{new_tag} and git diff --name-only, then send both to the LLM with a targeted prompt:

You are a security engineer. Library '{name}' was bumped from {old} to {new}.
No changelog is available. Here are commit messages and changed files.

Commit log: {log[:3000]}
Changed files: {files[:1000]}

Assess:
1. Security-related commits (CVE, fix, vuln, security, auth, tls, crypto, overflow, injection)
2. Security-sensitive changed paths (crypto/, auth/, tls/, parser/, validator/)
3. Your confidence level given limited information

Respond as JSON: {security_relevant_commits, security_sensitive_files, risk_level, confidence, summary}

The fetch_git_diff function in security-diff-analysis.py (Step 3) already fetches the GitHub compare API summary for npm packages as a fallback when fetch_changelog returns no content, so this path is automatically used for libraries without GitHub releases.

Step 5: Rate Limiting and Cost Controls

#!/usr/bin/env python3
# cost-control-filter.py — filter packages to analyse based on cost rules
import json
import sys
from packaging.version import Version, InvalidVersion

def classify_bump_type(old: str, new: str) -> str:
    """Classify version bump as major, minor, or patch."""
    try:
        old_v = Version(old)
        new_v = Version(new)
        if new_v.major > old_v.major:
            return "major"
        if new_v.minor > old_v.minor:
            return "minor"
        return "patch"
    except InvalidVersion:
        return "unknown"

# Packages where patch bumps are virtually always safe (build tools, type stubs)
SAFE_PATCH_PACKAGES = {
    "@types/node", "@types/react", "typescript", "eslint", "prettier",
    "black", "flake8", "mypy", "pytest", "go-mock", "testify",
}

# Packages that always require analysis regardless of bump type
ALWAYS_ANALYSE = {
    "openssl", "cryptography", "pyca/cryptography", "rustls",
    "axios", "node-fetch", "urllib3", "requests",
    "jsonwebtoken", "passport", "jwt",
    "django", "express", "fastapi", "gin",
}

def should_analyse(pkg: dict) -> tuple[bool, str]:
    name = pkg["name"].lower()
    bump_type = classify_bump_type(pkg["old_version"], pkg["new_version"])

    if name in ALWAYS_ANALYSE:
        return True, f"always-analyse list ({bump_type} bump)"

    if bump_type == "major":
        return True, "major version bump — always analyse"

    if bump_type == "minor":
        return True, "minor version bump — analyse by default"

    # Patch bump: skip safe packages
    if name in SAFE_PATCH_PACKAGES:
        return False, f"safe-patch-packages allowlist — skipping"

    # Patch bump for unknown package: analyse
    return True, f"patch bump for non-allowlisted package — analyse"

if __name__ == "__main__":
    with open(sys.argv[1]) as f:
        packages = json.load(f)

    filtered = []
    skipped = []

    for pkg in packages:
        analyse, reason = should_analyse(pkg)
        if analyse:
            pkg["analysis_reason"] = reason
            filtered.append(pkg)
        else:
            pkg["skip_reason"] = reason
            skipped.append(pkg)

    print(f"[*] Analysing {len(filtered)}/{len(packages)} packages", file=sys.stderr)
    for s in skipped:
        print(f"  [skip] {s['name']}: {s['skip_reason']}", file=sys.stderr)

    print(json.dumps(filtered))

Step 6: Renovate postUpgradeTasks Integration

// renovate.json — attach LLM security analysis to every dependency bump PR
{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:recommended"],
  "postUpgradeTasks": {
    "commands": [
      "python3 .github/scripts/detect-dep-changes.py --base {{{baseBranch}}} --head HEAD --output /tmp/dep-changes.json",
      "python3 .github/scripts/security-diff-analysis.py --packages /tmp/dep-changes.json --output /tmp/security-analysis.json --repo-token {{{env.GITHUB_TOKEN}}}"
    ],
    "fileFilters": ["**/package-lock.json", "**/Pipfile.lock", "**/go.sum", "**/Cargo.lock"],
    "executionMode": "branch"
  },
  "prBodyTemplate": "{{{prBodyDefinitions.table}}}\n\n---\n\n**Security Analysis:** See the `dependency-security-review` CI check for LLM-powered security diff analysis.",
  "packageRules": [
    {
      "description": "Auto-merge only low-risk packages after CI passes",
      "matchUpdateTypes": ["patch"],
      "matchPackageNames": ["eslint", "prettier", "black", "pytest"],
      "automerge": true,
      "automergeType": "pr"
    },
    {
      "description": "Require security review for security-critical packages",
      "matchPackageNames": ["cryptography", "openssl", "axios", "urllib3", "requests", "jsonwebtoken"],
      "reviewers": ["team:security"],
      "additionalBranchPrefix": "security-review/",
      "labels": ["security-review-required"]
    }
  ]
}

Expected Behaviour After Hardening

When Renovate raises a PR bumping axios from 1.6.7 to 1.7.2, the workflow runs within two minutes. The LLM flags network_changes: ["Added SSRF protection via allowlist for redirect targets in 1.7.0"] and input_validation_changes: ["URL validation now rejects private IP ranges by default"] with risk_level: "medium". Because axios is on the ALWAYS_ANALYSE list, a required status check blocks auto-merge until a team member confirms whether the application makes internal HTTP calls via axios. For a patch bump of typescript from 5.4.2 to 5.4.5, the cost filter skips analysis and CI auto-merges after tests pass.

Over a month across a typical service repository, the workflow analyses 140 of 200 dependency PRs (skipping 60 safe patch bumps). It generates 12 "high" findings, 45 "medium" findings, and 83 "low" findings. All 12 "high" findings are routed to security review; 8 are confirmed genuine security-relevant changes that receive manual approval with implementation notes.

Trade-offs and Operational Considerations

Failure Modes

Related Articles

• Renovate and Dependabot Security
• AI SAST in CI/CD Vulnerability Discovery
• Dependency Pinning
• Go Module Supply Chain Security
• SBOM Supply Chain Compromise Detection