AI-Assisted Security Audit of Open-Source Shared Libraries Before Production Use

The Problem

When a Linux package manager installs a shared library, it drops a .so file into /usr/lib or /usr/local/lib, runs ldconfig, and the library immediately enters the trusted path for every process on the host. From that moment, any process that dlopen()s the library or links against it at startup executes its code with the privileges of the calling process. There is no sandbox, no permission gate, no runtime verification that the binary you received matches what the project published as source.

Traditional security practice for shared libraries is to trust the distribution package: if the package was signed by the distribution’s key and the repository metadata was verified, the library is assumed safe. This model worked well when the threat was primarily a network attacker intercepting package downloads. It breaks down against the threats that have become dominant: upstream compromise (the package in the registry is tampered before distribution signing), trojanised builds (the source is clean but the CI system was compromised during the build), and dependency confusion (a maliciously named package that shadow-resolves ahead of the legitimate one).

The source-binary alignment problem is particularly acute. A project publishes libcurl 8.7.1 source code on GitHub. The apt repository serves libcurl4 8.7.1-1ubuntu1. Unless someone has performed a reproducible build of the source and diffed the result against the distributed binary, nobody has verified that those two things are the same. For the vast majority of libraries in production, they have not been verified.

AI-powered binary analysis changes the cost equation. Until recently, binary-level analysis of shared libraries — lifting disassembly to intermediate representation, performing taint analysis across function call graphs, identifying anomalous capabilities — required dedicated security researchers and weeks of work per library. Tools like Ghidra, Binary Ninja, and angr made the mechanics available but still demanded expert users. In 2026, LLM-assisted scripting has reduced the entry cost dramatically. An AI-generated Ghidra headless script can extract all external function calls from a library, group them by capability class, and flag any execve, socket, or /proc read in a library that declares itself a pure math utility. An LLM-generated angr harness can set up taint analysis on library input functions in minutes instead of hours.

The goal is not to replace expert reverse engineering for high-value targets. The goal is to apply a structured, automatable pre-production gate that catches the most common tampering signatures and unexpected capability additions before a library ever reaches a production host.

Threat Model

Upstream compromise. An attacker with access to a package registry — via stolen maintainer credentials, a compromised build server, or a malicious package mirror — modifies the binary that gets distributed without modifying the published source. The binary diverges from the source in ways that are not visible to anyone performing source code review. The XZ utils backdoor (CVE-2024-3094) was an extreme example: the malicious code was introduced in the compressed release tarball, not the git repository, specifically to evade source-based review. Binary-level analysis detects the divergence.

Dependency confusion. A package named libinternal-auth exists on an internal package server at version 1.2.3. An attacker publishes a package with the same name to PyPI, npm, or a public Cargo registry at version 9.9.9. Build systems that check public registries before private ones pull the attacker’s version. The malicious .so or .a is linked into the application and loaded at runtime. Capability analysis flags a library that should implement only authentication logic but instead opens sockets or reads environment variables containing credentials.

Trojanised build. The project’s CI/CD pipeline is compromised. Source code in the repository is clean and passes review. During the CI build, a malicious build dependency, a compromised build tool, or an injected pipeline step modifies the compiled output. The resulting binary has additional functionality not present in any source file. Reproducible build comparison — rebuilding from source and diffing against the distributed binary — is the primary detection mechanism.

Runtime injection. An attacker with local access to the host (initial access via web shell, SSRF, or credential theft) places a malicious .so into a library search path ahead of the legitimate library. LD_LIBRARY_PATH, /etc/ld.so.conf.d/ entries, or RPATH entries in executables can all be abused to redirect library loading. An LD_AUDIT hook that monitors the runtime linker’s resolution decisions detects unexpected library loads and alerts before the malicious code executes in a sensitive process.

Hardening Configuration

Step 1: Checksec and Readelf Baseline Audit

Before any deeper analysis, confirm that installed libraries have the expected binary security properties. A library missing RELRO, stack canaries, or with an executable stack is a red flag independent of its functional behaviour.

#!/usr/bin/env bash
# /usr/local/bin/lib-security-baseline.sh
# Run as root or with sudo to reach all library paths
set -euo pipefail

REPORT_DIR="/var/log/lib-security-audit"
mkdir -p "$REPORT_DIR"
TIMESTAMP=$(date +%Y%m%dT%H%M%S)
REPORT="$REPORT_DIR/baseline-$TIMESTAMP.json"

LIB_PATHS=(
  /usr/lib
  /usr/lib/x86_64-linux-gnu
  /usr/local/lib
  /lib/x86_64-linux-gnu
)

echo '{"timestamp":"'"$TIMESTAMP"'","libraries":[' > "$REPORT"
FIRST=1

for DIR in "${LIB_PATHS[@]}"; do
  find "$DIR" -maxdepth 2 -name "*.so*" -type f 2>/dev/null | while read -r lib; do
    # checksec outputs structured data with --format=json
    CHECKSEC_OUT=$(checksec --format=json --file="$lib" 2>/dev/null || echo '{}')

    # readelf for additional properties
    HAS_RELRO="none"
    if readelf -d "$lib" 2>/dev/null | grep -q 'GNU_RELRO'; then
      HAS_RELRO="partial"
    fi
    if readelf -d "$lib" 2>/dev/null | grep -q 'BIND_NOW'; then
      HAS_RELRO="full"
    fi

    NX_STACK=$(readelf -W -l "$lib" 2>/dev/null | \
      awk '/GNU_STACK/ { print ($6=="RWE" || $6=="RW") ? "disabled" : "enabled" }')

    # Flag libraries with missing protections
    RISK="low"
    if [[ "$HAS_RELRO" == "none" ]] || [[ "$NX_STACK" == "disabled" ]]; then
      RISK="high"
    elif [[ "$HAS_RELRO" == "partial" ]]; then
      RISK="medium"
    fi

    [[ "$FIRST" -eq 1 ]] || echo "," >> "$REPORT"
    FIRST=0

    jq -n \
      --arg path "$lib" \
      --arg relro "$HAS_RELRO" \
      --arg nx "$NX_STACK" \
      --arg risk "$RISK" \
      --argjson cs "$CHECKSEC_OUT" \
      '{path: $path, relro: $relro, nx_stack: $nx, risk: $risk, checksec: $cs}' >> "$REPORT"
  done
done

echo ']}' >> "$REPORT"

# Emit high-risk findings to stdout for CI/alerting integration
jq '.libraries[] | select(.risk == "high") | .path' "$REPORT"
echo "Full report: $REPORT"

Run this after every package installation as part of a post-install hook:

# /etc/apt/apt.conf.d/99-lib-security-hook
DPkg::Post-Invoke {"if [ -x /usr/local/bin/lib-security-baseline.sh ]; then /usr/local/bin/lib-security-baseline.sh >> /var/log/lib-security-audit/apt-hook.log 2>&1; fi";};

Step 2: AI-Assisted Ghidra Headless Analysis

Ghidra’s headless analyser can be scripted to perform automated function-level analysis. The following script uses Ghidra to extract external calls from a library and then passes the results to an LLM to classify them by capability.

# ghidra_capability_audit.py — run with: ghidra_headless.py
# Usage: $GHIDRA_HOME/support/analyzeHeadless /tmp/ghidra_proj LibAudit \
#        -import /usr/lib/x86_64-linux-gnu/libtarget.so \
#        -postScript ghidra_capability_audit.py \
#        -scriptPath /opt/audit-scripts

from ghidra.app.util.headless import HeadlessScript
from ghidra.program.model.symbol import SymbolType
import json

def run():
    fm = currentProgram.getFunctionManager()
    external_calls = []

    for func in fm.getFunctions(True):
        for ref in func.getCalledFunctions(monitor):
            if ref.isExternal():
                external_calls.append({
                    "caller": func.getName(),
                    "callee": ref.getName(),
                    "address": str(func.getEntryPoint())
                })

    # Write for LLM analysis
    output_path = "/tmp/ghidra_external_calls.json"
    with open(output_path, "w") as f:
        json.dump(external_calls, f, indent=2)

    print(f"[+] Extracted {len(external_calls)} external calls to {output_path}")

After Ghidra extraction, pass the results to an LLM for capability classification:

#!/usr/bin/env python3
# llm_capability_classifier.py
import json
import sys
import anthropic

SUSPICIOUS_CAPABILITIES = {
    "network": ["socket", "connect", "bind", "sendto", "recvfrom", "getaddrinfo"],
    "execution": ["execve", "execvp", "system", "popen", "posix_spawn"],
    "proc_access": ["open", "read", "readlink"],  # flagged when path is /proc
    "credential_access": ["getenv", "getpwuid", "getpwnam"],
    "privilege": ["setuid", "setgid", "capset", "prctl"],
}

def classify_with_llm(lib_name: str, calls: list) -> dict:
    client = anthropic.Anthropic()

    calls_text = json.dumps(calls[:500], indent=2)  # cap at 500 entries

    prompt = f"""You are auditing a shared library named '{lib_name}' for unexpected capabilities.

Below is a list of external function calls extracted from the library binary via Ghidra.
Each entry shows which internal function makes a call to which external (libc or syscall wrapper) function.

External calls:
{calls_text}

For this library, identify:
1. Any network capability calls (socket, connect, bind, sendto, etc.) — flag as HIGH risk if present in a library that should have no network function
2. Any execution capability calls (execve, system, popen) — flag as CRITICAL
3. Any /proc filesystem access patterns — flag as MEDIUM
4. Any credential or environment variable access (getenv, getpwuid) — flag as MEDIUM
5. Any privilege manipulation (setuid, capset, prctl) — flag as HIGH

Respond as JSON with this exact structure:
{{
  "capabilities_found": ["list of capability categories detected"],
  "high_risk_calls": [
    {{"function": "name", "called_by": "caller", "risk": "HIGH/CRITICAL/MEDIUM", "reason": "..."}}
  ],
  "overall_risk": "LOW/MEDIUM/HIGH/CRITICAL",
  "summary": "one paragraph summary",
  "recommendation": "APPROVE/REVIEW/REJECT"
}}"""

    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=2048,
        messages=[{"role": "user", "content": prompt}]
    )

    return json.loads(response.content[0].text)

if __name__ == "__main__":
    calls_file = sys.argv[1]
    lib_name = sys.argv[2] if len(sys.argv) > 2 else "unknown"

    with open(calls_file) as f:
        calls = json.load(f)

    result = classify_with_llm(lib_name, calls)
    print(json.dumps(result, indent=2))

    if result["overall_risk"] in ("HIGH", "CRITICAL"):
        sys.exit(1)

Step 3: Taint Analysis with angr and LLM-Generated Harnesses

angr performs symbolic execution and taint analysis. LLM assistance dramatically reduces harness authoring time.

#!/usr/bin/env python3
# angr_taint_harness.py — LLM-generated harness template, then customised
# Checks: does attacker-controlled input to library entry points reach exec/socket/write?

import angr
import claripy
import sys

LIB_PATH = sys.argv[1]       # e.g. /usr/lib/x86_64-linux-gnu/libfoo.so
ENTRY_FUNC = sys.argv[2]     # e.g. foo_parse_packet

# Load binary — use full_init_state for library analysis
proj = angr.Project(LIB_PATH, load_options={
    "auto_load_libs": False,
    "main_opts": {"base_addr": 0x400000}
})

# Create symbolic input buffer (attacker-controlled network data)
INPUT_SIZE = 1024
sym_input = claripy.BVS("attacker_input", INPUT_SIZE * 8)

# Find the entry function address
entry_sym = proj.loader.find_symbol(ENTRY_FUNC)
if not entry_sym:
    print(f"[!] Symbol '{ENTRY_FUNC}' not found in {LIB_PATH}")
    sys.exit(1)

entry_addr = entry_sym.rebased_addr

# Initial state: call entry function with symbolic buffer pointer
state = proj.factory.call_state(
    entry_addr,
    sym_input,
    claripy.BVV(INPUT_SIZE, 64),
    add_options={angr.options.SYMBOL_FILL_UNCONSTRAINED_MEMORY,
                 angr.options.SYMBOL_FILL_UNCONSTRAINED_REGISTERS}
)

# Danger addresses: resolve libc functions we want to detect reaching
DANGER_FUNCS = ["execve", "system", "socket", "connect", "write", "sendto"]
danger_addrs = []
for func_name in DANGER_FUNCS:
    sym = proj.loader.find_symbol(func_name)
    if sym:
        danger_addrs.append((func_name, sym.rebased_addr))
        print(f"[*] Tracking {func_name} at {hex(sym.rebased_addr)}")

simgr = proj.factory.simulation_manager(state)

# Run for bounded steps to avoid infinite analysis
simgr.run(n=2000)

print(f"[*] Analysis complete: {len(simgr.deadended)} paths explored")
for stash in simgr.stashes:
    if simgr.stashes[stash]:
        print(f"  {stash}: {len(simgr.stashes[stash])} states")

# Check if any path reaches a danger function with tainted (symbolic) arguments
for stash_name, states in simgr.stashes.items():
    for s in states:
        for func_name, addr in danger_addrs:
            # If the instruction pointer can reach this address
            if s.solver.satisfiable(extra_constraints=[s.ip == addr]):
                print(f"[CRITICAL] Tainted input can reach {func_name}!")

Step 4: Reproducible Build Comparison with diffoscope

#!/usr/bin/env bash
# repro-build-check.sh — compare installed library against source rebuild
# Requires: diffoscope, docker (for hermetic build environment)
set -euo pipefail

PACKAGE_NAME="$1"   # e.g. libcurl4
PACKAGE_VERSION="$2" # e.g. 8.7.1-1ubuntu1

WORK_DIR=$(mktemp -d)
INSTALLED_LIB=$(dpkg -L "$PACKAGE_NAME" | grep '\.so$' | head -1)

echo "[*] Comparing installed $INSTALLED_LIB against reproducible build"
cp "$INSTALLED_LIB" "$WORK_DIR/installed.so"

# Pull source and rebuild in hermetic container
docker run --rm \
  -v "$WORK_DIR:/out" \
  -e PACKAGE="$PACKAGE_NAME" \
  -e VERSION="$PACKAGE_VERSION" \
  ubuntu:24.04 bash -c '
    apt-get update -qq
    apt-get build-dep -y "$PACKAGE" 2>/dev/null
    apt-get source "$PACKAGE=$VERSION" 2>/dev/null
    cd $(ls -d */); dpkg-buildpackage -b -uc -us 2>/dev/null
    cp ../*.so* /out/rebuilt.so 2>/dev/null || true
  '

if [[ ! -f "$WORK_DIR/rebuilt.so" ]]; then
  echo "[WARN] Could not produce rebuilt binary; skipping diffoscope"
  exit 0
fi

# Run diffoscope — non-zero exit means differences found
if diffoscope \
    --html "$WORK_DIR/diff-report.html" \
    --json "$WORK_DIR/diff-report.json" \
    "$WORK_DIR/installed.so" \
    "$WORK_DIR/rebuilt.so"; then
  echo "[PASS] Installed binary matches reproducible build"
else
  echo "[FAIL] Binary divergence detected — review $WORK_DIR/diff-report.html"
  # Extract section-level differences for quick triage
  jq '.[] | select(.unified_diff != null) | .path' "$WORK_DIR/diff-report.json" 2>/dev/null || true
  exit 1
fi

Step 5: LD_AUDIT Runtime Library Load Monitoring

LD_AUDIT is a glibc mechanism that lets a monitoring library intercept all dynamic linker decisions. This detects runtime injection of unexpected .so files.

/* ld_audit_monitor.c — compile as shared library */
/* gcc -shared -fPIC -o /usr/local/lib/ld_audit_monitor.so ld_audit_monitor.c */
#define _GNU_SOURCE
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <syslog.h>
#include <link.h>

/* Allowlist of expected library paths — loaded from environment or file */
static const char *EXPECTED_PREFIXES[] = {
    "/usr/lib/",
    "/usr/lib/x86_64-linux-gnu/",
    "/lib/x86_64-linux-gnu/",
    "/lib64/",
    NULL
};

unsigned int la_version(unsigned int version) {
    openlog("ld_audit_monitor", LOG_PID | LOG_NDELAY, LOG_SECURITY);
    return LAV_CURRENT;
}

char *la_objsearch(const char *name, uintptr_t *cookie, unsigned int flag) {
    return (char *)name;  /* pass through search unchanged */
}

unsigned int la_objopen(struct link_map *map, Lmid_t lmid, uintptr_t *cookie) {
    const char *path = map->l_name;

    if (!path || path[0] == '\0') return LA_FLG_BINDFROM | LA_FLG_BINDTO;

    int path_ok = 0;
    for (int i = 0; EXPECTED_PREFIXES[i] != NULL; i++) {
        if (strncmp(path, EXPECTED_PREFIXES[i], strlen(EXPECTED_PREFIXES[i])) == 0) {
            path_ok = 1;
            break;
        }
    }

    if (!path_ok) {
        syslog(LOG_CRIT,
            "UNEXPECTED_LIBRARY_LOAD path=%s lmid=%ld — possible injection",
            path, (long)lmid);

        /* In enforcement mode, kill the process rather than load unknown library */
        const char *enforce = getenv("LD_AUDIT_ENFORCE");
        if (enforce && strcmp(enforce, "1") == 0) {
            syslog(LOG_CRIT, "ENFORCEMENT: aborting process to prevent load of %s", path);
            abort();
        }
    }

    return LA_FLG_BINDFROM | LA_FLG_BINDTO;
}

Deploy this system-wide for sensitive processes:

# /etc/systemd/system/nginx.service.d/ld-audit.conf
[Service]
Environment="LD_AUDIT=/usr/local/lib/ld_audit_monitor.so"
Environment="LD_AUDIT_ENFORCE=1"

Step 6: Automated Pre-Deployment Pipeline

# .github/workflows/library-security-gate.yml
name: Shared Library Security Gate

on:
  workflow_dispatch:
    inputs:
      package_name:
        description: 'apt package name (e.g. libcurl4)'
        required: true
      package_version:
        description: 'Version string (e.g. 8.7.1-1ubuntu1)'
        required: true

jobs:
  library-audit:
    runs-on: ubuntu-24.04
    container:
      image: ubuntu:24.04
      options: --privileged

    steps:
      - name: Install audit tooling
        run: |
          apt-get update -qq
          apt-get install -y --no-install-recommends \
            checksec readelf-tools binutils \
            python3-pip diffoscope docker.io
          pip3 install angr anthropic

      - name: Install target package
        run: |
          apt-get install -y "${{ inputs.package_name }}=${{ inputs.package_version }}"

      - name: Checksec baseline
        id: checksec
        run: |
          /usr/local/bin/lib-security-baseline.sh 2>&1 | tee /tmp/checksec-output.txt
          # Fail if any high-risk libraries found
          if grep -q '"risk":"high"' /tmp/checksec-output.txt; then
            echo "high_risk=true" >> "$GITHUB_OUTPUT"
          fi

      - name: Ghidra capability extraction
        run: |
          LIB_PATH=$(dpkg -L "${{ inputs.package_name }}" | grep '\.so$' | head -1)
          $GHIDRA_HOME/support/analyzeHeadless /tmp/ghidra_proj LibAudit \
            -import "$LIB_PATH" \
            -postScript ghidra_capability_audit.py \
            -scriptPath .github/scripts

      - name: LLM capability classification
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          python3 .github/scripts/llm_capability_classifier.py \
            /tmp/ghidra_external_calls.json \
            "${{ inputs.package_name }}" | tee /tmp/llm-analysis.json
          RISK=$(jq -r '.overall_risk' /tmp/llm-analysis.json)
          echo "llm_risk=$RISK" >> "$GITHUB_OUTPUT"

      - name: Reproducible build comparison
        run: |
          bash .github/scripts/repro-build-check.sh \
            "${{ inputs.package_name }}" \
            "${{ inputs.package_version }}"

      - name: Generate approval record
        run: |
          jq -n \
            --arg pkg "${{ inputs.package_name }}" \
            --arg ver "${{ inputs.package_version }}" \
            --arg ts "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
            --argjson llm "$(cat /tmp/llm-analysis.json)" \
            '{package: $pkg, version: $ver, audited_at: $ts, llm_analysis: $llm, status: "APPROVED"}' \
            > "approved-libs/${{ inputs.package_name }}-${{ inputs.package_version }}.json"

      - name: Upload audit artifacts
        uses: actions/upload-artifact@v4
        with:
          name: library-audit-${{ inputs.package_name }}
          path: /tmp/
          retention-days: 90

Expected Behaviour After Hardening

After implementing this pipeline, a library must pass all gates before being permitted into the production library allowlist. A new version of libpq5 (PostgreSQL client library) arriving via a routine apt upgrade is held until the automated workflow completes. If the reproducible build check passes and the LLM capability classifier returns "overall_risk": "LOW" with no flagged external calls beyond malloc, memcpy, and PostgreSQL protocol functions, the library is added to the approved manifest and permitted. If the Ghidra analysis flags an unexpected socket() call in a newly added function, the workflow fails, blocks the upgrade, and creates a GitHub issue with the LLM’s structured explanation.

The LD_AUDIT monitor in observation mode (non-enforcement) logs every library load for 30 days, building a baseline of expected load patterns per process. After the baseline period, enforcement mode activates and processes are terminated if they attempt to load a library not on the approved list.

Security properties checked:

• All production libraries carry Full RELRO, stack canaries, and NX stack
• No library with unexpected network, execution, or privilege capabilities is deployed without manual security review
• Checksums of deployed libraries are recorded in an immutable audit log and verified daily
• The LD_AUDIT monitor generates a SECURITY syslog alert for any out-of-allowlist library load within 5 seconds of the event

Trade-offs and Operational Considerations

Failure Modes

Related Articles

• Linux Shared Library Security
• Malicious Dependency Runtime Detection
• Reproducible Builds Security
• Linux Developer Workstation Supply Chain
• SBOM Generation and Consumption