A Composite AI Trust Scoring Framework for Open-Source Projects

The Problem

When an engineering team evaluates a new OSS library, the question “should we adopt this?” rarely receives a structured answer. The informal evaluation is a quick scan of GitHub metrics — star count, recent commit activity, number of open issues, a glance at the README. These signals are better than nothing but are not a security evaluation. A popular project with high star count can have catastrophic security hygiene. A small, low-activity project written by a single security researcher might have excellent hygiene and a clean CVE history.

OpenSSF Scorecard addresses this gap with a systematic automated assessment covering code review practices, branch protection, dependency pinning, CI/CD security controls, and more. Scorecard is genuinely valuable — it provides objective evidence about practices that correlate with security outcomes. But Scorecard alone has blind spots. It does not assess the semantic security quality of the code itself. It does not weight the project’s actual CVE history by exploitability. It does not assess whether the maintainers are known entities with a track record. And once a Scorecard result is generated, there is no standard mechanism for it to be automatically re-evaluated when circumstances change — when a new CVE is published, when the primary maintainer changes, or when the project is acquired.

The result is that OSS adoption decisions remain partly informal even in organisations that use Scorecard. The security team’s answer to “can we use this?” is a mix of Scorecard number, gut feel, and a quick search for recent CVEs. The composite signal is not documented, not repeatable across different engineers making similar decisions, and not automatically updated as the project evolves.

A composite trust score that combines Scorecard with EPSS-weighted CVE history, an LLM-assisted code review summary of security-sensitive modules, maintainer behaviour signals, and dependency health — each with explicit weights and a consistent aggregation method — turns this into a defensible, auditable decision. The score can be stored in a registry, automatically re-triggered when relevant inputs change, and surfaced to engineers at the point of adoption.

Threat Model

Adoption of a library with poor security hygiene that is later exploited. An engineering team adopts a library that has a Scorecard of 4/10 and a history of three CVEs in the past two years, two of which had public exploits. The informal evaluation noted the low star count but concluded the library was small and well-scoped. Eighteen months later, a new CVE is published with an active exploit in the wild. The organisation’s scanner flags the vulnerability and a critical patch cycle begins. A composite trust score would have flagged this library as Yellow or Red at adoption time, requiring security team sign-off.

Rejection of a legitimate library due to misleading individual signals. A well-maintained, security-focused cryptography library has a low Scorecard because it is a single-maintainer project that does not use automated dependency update tools (a Scorecard check). The Scorecard score of 5/10 triggers automatic rejection in an organisation that uses Scorecard as the sole criterion. But the maintainer is a well-known cryptographer, the code has been extensively reviewed by third parties, and the CVE history is clean. A composite score with appropriate weighting for code quality and maintainer identity would produce a Green result with a note about the single-maintainer concentration risk.

Trust scores becoming stale after a project event. A library receives a Green composite score and is adopted. Two years later, the primary maintainer transfers the project to a new organisation. The new maintainers’ GitHub activity patterns are different. A large new dependency with its own vulnerabilities is added. No re-evaluation is triggered. The library is still tagged Green in the organisation’s registry. When the new dependency’s vulnerability is exploited, the trust score was technically valid at adoption but stale for 18 months.

Over-reliance on the composite score. A library scores 82/100 and receives a Green classification. An engineering team uses this as permission to skip detailed code review of the library’s network-handling module, which implements a custom HTTP parser with a subtle but exploitable buffer handling issue. The trust score represents average signal quality; it does not substitute for code review of high-risk subsystems in libraries that will have elevated privilege in the application.

Hardening Configuration

The Five-Component Composite Model

The composite score is constructed from five component categories, each producing a normalised 0–100 sub-score. The component weights are tuned to reflect a production web service risk profile:

#!/usr/bin/env python3
"""
oss_trust_score.py — Composite OSS project trust scoring.

Queries OpenSSF Scorecard, OSV, EPSS, GitHub, and deps.dev APIs to compute
a composite trust score for an OSS project. Outputs a structured JSON score
suitable for storage in a trust registry.
"""

import json
import math
import time
import hashlib
import requests
import anthropic
from dataclasses import dataclass, field, asdict
from datetime import datetime, timezone
from pathlib import Path
from typing import Optional

client = anthropic.Anthropic()

SCORECARD_API = "https://api.securityscorecards.dev"
OSV_API = "https://api.osv.dev/v1"
EPSS_API = "https://api.first.org/data/v1/epss"
DEPS_DEV_API = "https://api.deps.dev/v3alpha"
GITHUB_API = "https://api.github.com"


@dataclass
class ComponentScore:
    name: str
    raw_score: float           # 0.0–100.0
    weight: float              # 0.0–1.0
    weighted_score: float = 0.0
    evidence: dict = field(default_factory=dict)
    notes: list[str] = field(default_factory=list)

    def __post_init__(self):
        self.weighted_score = self.raw_score * self.weight


@dataclass
class TrustScore:
    package_name: str
    ecosystem: str
    github_repo: str
    composite_score: float     # 0.0–100.0
    classification: str        # Green / Yellow / Red
    components: list[ComponentScore] = field(default_factory=list)
    evaluated_at: str = ""
    score_version: str = "1.0"
    next_review_trigger: list[str] = field(default_factory=list)

    def __post_init__(self):
        self.evaluated_at = datetime.now(timezone.utc).isoformat()


# ─── Component 1: OpenSSF Scorecard ──────────────────────────────────────────

def fetch_scorecard(github_repo: str) -> ComponentScore:
    """Fetch OpenSSF Scorecard result for a GitHub repo."""
    url = f"{SCORECARD_API}/projects/github.com/{github_repo}"
    try:
        resp = requests.get(url, timeout=15)
        resp.raise_for_status()
        data = resp.json()
        score = float(data.get("score", 0)) * 10  # Scorecard is 0–10, normalise to 0–100

        evidence = {
            "scorecard_score": data.get("score"),
            "checks": {
                c["name"]: c.get("score", 0)
                for c in data.get("checks", [])
            },
        }
        notes = [
            f"Check '{c['name']}' scored {c.get('score', 0)}/10"
            for c in data.get("checks", [])
            if c.get("score", 10) < 5
        ]

        return ComponentScore(
            name="OpenSSF Scorecard",
            raw_score=score,
            weight=0.25,
            evidence=evidence,
            notes=notes[:5],  # Top 5 weak checks
        )
    except requests.RequestException as e:
        return ComponentScore(
            name="OpenSSF Scorecard",
            raw_score=50.0,  # Neutral fallback if API unavailable
            weight=0.25,
            notes=[f"API unavailable: {e}"],
        )


# ─── Component 2: EPSS-Weighted CVE History ───────────────────────────────────

def fetch_cve_history(package_name: str, ecosystem: str) -> ComponentScore:
    """
    Query OSV for all historical CVEs affecting this package, fetch EPSS scores,
    and compute a penalty score: more exploitable CVEs = lower score.
    """
    payload = {"package": {"name": package_name, "ecosystem": ecosystem}}
    try:
        resp = requests.post(f"{OSV_API}/query", json=payload, timeout=15)
        resp.raise_for_status()
        vulns = resp.json().get("vulns", [])
    except requests.RequestException:
        vulns = []

    if not vulns:
        return ComponentScore(
            name="CVE History (EPSS-weighted)",
            raw_score=95.0,  # Clean history = near-perfect score
            weight=0.20,
            evidence={"cve_count": 0},
            notes=["No CVEs found in OSV database"],
        )

    # Fetch EPSS for all CVE IDs
    cve_ids = [
        alias
        for v in vulns
        for alias in v.get("aliases", [])
        if alias.startswith("CVE-")
    ]
    epss_data = {}
    if cve_ids:
        try:
            params = {"cve": ",".join(cve_ids[:100])}
            resp = requests.get(EPSS_API, params=params, timeout=10)
            resp.raise_for_status()
            for item in resp.json().get("data", []):
                epss_data[item["cve"]] = float(item.get("epss", 0))
        except requests.RequestException:
            pass

    # Penalty: sum of EPSS scores, capped at 1.0 per CVE
    total_epss = sum(min(epss_data.get(c, 0.01), 1.0) for c in cve_ids)
    # Sigmoid-style mapping: 0 EPSS → 95, 1.0 total EPSS → 50, 5.0+ → 5
    raw_score = max(5.0, 95.0 - (total_epss * 18.0))

    high_epss = {c: s for c, s in epss_data.items() if s > 0.05}

    return ComponentScore(
        name="CVE History (EPSS-weighted)",
        raw_score=raw_score,
        weight=0.20,
        evidence={
            "total_cves": len(vulns),
            "cves_with_epss": len(epss_data),
            "total_epss_sum": round(total_epss, 4),
            "high_epss_cves": high_epss,
        },
        notes=[f"{len(vulns)} CVEs found; {len(high_epss)} with EPSS > 0.05"],
    )


# ─── Component 3: LLM Code Review Summary ────────────────────────────────────

CODE_REVIEW_PROMPT = """You are a security code reviewer evaluating an open-source library for safe adoption.

Package: {package_name} ({ecosystem})
GitHub repo: https://github.com/{github_repo}

The following files are the security-sensitive modules from this project (input validation,
authentication, cryptography, network handling, deserialization):

{code_samples}

Evaluate the security quality of this code on a 0–100 scale:
- 90–100: Excellent security hygiene, consistent input validation, safe crypto use, no obvious issues
- 70–89: Good overall, minor concerns that don't represent immediate risk
- 50–69: Mixed quality, some patterns that could lead to vulnerabilities
- 30–49: Significant concerns: missing validation, unsafe patterns, questionable crypto
- 0–29: Multiple serious security issues evident in the code

Output valid JSON only:
{{
  "security_score": <integer 0-100>,
  "key_concerns": ["list of specific concerns with file/line references where possible"],
  "positive_signals": ["list of positive security patterns observed"],
  "high_risk_modules": ["module names that warrant manual review before adoption"],
  "summary": "Two sentences summarising the overall security posture of the code"
}}"""


def fetch_code_samples(github_repo: str, github_token: str = "") -> str:
    """
    Fetch security-sensitive source files from the repository for LLM analysis.
    Targets files likely to contain input handling, auth, and crypto logic.
    """
    headers = {"Accept": "application/vnd.github+json"}
    if github_token:
        headers["Authorization"] = f"Bearer {github_token}"

    # Search for security-relevant files
    security_patterns = [
        "auth", "crypto", "validate", "sanitize", "parse", "deserializ",
        "token", "session", "permission", "security",
    ]

    samples = []
    total_chars = 0
    MAX_CHARS = 12000  # Keep within LLM context limits

    try:
        # Get repo tree
        resp = requests.get(
            f"{GITHUB_API}/repos/{github_repo}/git/trees/HEAD?recursive=1",
            headers=headers, timeout=20
        )
        resp.raise_for_status()
        tree = resp.json().get("tree", [])

        # Find Python/Go/JS source files matching security patterns
        candidate_files = [
            item["path"] for item in tree
            if item["type"] == "blob"
            and any(pat in item["path"].lower() for pat in security_patterns)
            and item["path"].endswith((".py", ".go", ".js", ".ts", ".rs", ".java"))
            and item.get("size", 0) < 50000  # Skip very large files
        ][:8]  # Max 8 files

        for file_path in candidate_files:
            if total_chars >= MAX_CHARS:
                break
            try:
                resp = requests.get(
                    f"{GITHUB_API}/repos/{github_repo}/contents/{file_path}",
                    headers=headers, timeout=10
                )
                resp.raise_for_status()
                import base64
                content = base64.b64decode(resp.json()["content"]).decode("utf-8", errors="replace")
                excerpt = content[:2000]  # First 2000 chars per file
                samples.append(f"=== {file_path} ===\n{excerpt}")
                total_chars += len(excerpt)
            except (requests.RequestException, KeyError):
                pass

    except requests.RequestException:
        return "Unable to fetch code samples — API unavailable"

    return "\n\n".join(samples) if samples else "No security-relevant files found"


def llm_code_review(package_name: str, ecosystem: str,
                    github_repo: str, github_token: str = "") -> ComponentScore:
    """Run LLM-assisted security code review of the project's critical modules."""
    code_samples = fetch_code_samples(github_repo, github_token)

    message = client.messages.create(
        model="claude-opus-4-5",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": CODE_REVIEW_PROMPT.format(
                package_name=package_name,
                ecosystem=ecosystem,
                github_repo=github_repo,
                code_samples=code_samples,
            ),
        }],
        system=[{
            "type": "text",
            "text": "You are a precise security code reviewer. Output only valid JSON.",
            "cache_control": {"type": "ephemeral"},
        }],
    )

    try:
        result = json.loads(message.content[0].text)
        raw_score = float(result.get("security_score", 50))
        return ComponentScore(
            name="LLM Code Review",
            raw_score=raw_score,
            weight=0.20,
            evidence=result,
            notes=result.get("key_concerns", [])[:3],
        )
    except (json.JSONDecodeError, ValueError) as e:
        return ComponentScore(
            name="LLM Code Review",
            raw_score=50.0,
            weight=0.20,
            notes=[f"LLM review parse error: {e}"],
        )


# ─── Component 4: Maintainer Behaviour Signals ───────────────────────────────

def fetch_maintainer_signals(github_repo: str, github_token: str = "") -> ComponentScore:
    """Assess maintainer health signals from GitHub API."""
    headers = {"Accept": "application/vnd.github+json"}
    if github_token:
        headers["Authorization"] = f"Bearer {github_token}"

    score = 70.0  # Start neutral
    evidence = {}
    notes = []

    try:
        # Repo metadata
        repo_resp = requests.get(f"{GITHUB_API}/repos/{github_repo}", headers=headers, timeout=10)
        repo_resp.raise_for_status()
        repo = repo_resp.json()

        # Branch protection
        branch = repo.get("default_branch", "main")
        bp_resp = requests.get(
            f"{GITHUB_API}/repos/{github_repo}/branches/{branch}/protection",
            headers=headers, timeout=10
        )
        has_branch_protection = bp_resp.status_code == 200
        if has_branch_protection:
            score += 10
        else:
            notes.append("No branch protection on default branch")

        # Contributor count
        contrib_resp = requests.get(
            f"{GITHUB_API}/repos/{github_repo}/contributors?per_page=30",
            headers=headers, timeout=10
        )
        contributors = len(contrib_resp.json()) if contrib_resp.status_code == 200 else 1
        if contributors >= 5:
            score += 10
        elif contributors == 1:
            score -= 15
            notes.append("Single maintainer — concentration risk")

        # Last commit recency
        last_push = repo.get("pushed_at", "")
        if last_push:
            days_since = (datetime.now(timezone.utc) -
                          datetime.fromisoformat(last_push.replace("Z", "+00:00"))).days
            if days_since > 365:
                score -= 20
                notes.append(f"Last commit {days_since} days ago — possibly unmaintained")
            elif days_since < 30:
                score += 5

        # SECURITY.md presence
        sec_resp = requests.get(
            f"{GITHUB_API}/repos/{github_repo}/contents/SECURITY.md",
            headers=headers, timeout=10
        )
        has_security_md = sec_resp.status_code == 200
        if has_security_md:
            score += 5
        else:
            notes.append("No SECURITY.md — no documented vulnerability reporting process")

        evidence = {
            "contributors": contributors,
            "has_branch_protection": has_branch_protection,
            "has_security_md": has_security_md,
            "days_since_last_push": days_since if last_push else None,
            "open_issues": repo.get("open_issues_count", 0),
            "archived": repo.get("archived", False),
        }

        if repo.get("archived"):
            score = min(score, 20)
            notes.append("ARCHIVED — project is no longer maintained")

    except requests.RequestException as e:
        notes.append(f"GitHub API error: {e}")

    return ComponentScore(
        name="Maintainer Behaviour",
        raw_score=max(0.0, min(100.0, score)),
        weight=0.20,
        evidence=evidence,
        notes=notes,
    )


# ─── Component 5: Dependency Health ──────────────────────────────────────────

def fetch_dependency_health(package_name: str, ecosystem: str) -> ComponentScore:
    """Query deps.dev for the project's own dependency health."""
    try:
        url = f"{DEPS_DEV_API}/projects/{ecosystem.lower()}/{package_name}"
        resp = requests.get(url, timeout=10)
        if resp.status_code == 404:
            return ComponentScore(
                name="Dependency Health",
                raw_score=60.0,
                weight=0.15,
                notes=["Package not found in deps.dev"],
            )
        resp.raise_for_status()
        data = resp.json()

        # deps.dev scorecards and metrics
        scorecard = data.get("scorecard", {})
        dep_score = float(scorecard.get("score", 5.0)) * 10  # normalise to 0–100

        return ComponentScore(
            name="Dependency Health",
            raw_score=dep_score,
            weight=0.15,
            evidence={"deps_dev_score": scorecard.get("score"), "version_count": data.get("versionCount", 0)},
        )
    except requests.RequestException:
        return ComponentScore(
            name="Dependency Health",
            raw_score=60.0,  # Neutral fallback
            weight=0.15,
            notes=["deps.dev API unavailable"],
        )


# ─── Composite Score and Classification ──────────────────────────────────────

def compute_composite(components: list[ComponentScore]) -> tuple[float, str]:
    """Aggregate weighted component scores into composite with classification."""
    composite = sum(c.weighted_score for c in components)

    if composite >= 75:
        classification = "Green"
    elif composite >= 50:
        classification = "Yellow"
    else:
        classification = "Red"

    return round(composite, 1), classification


def evaluate_project(package_name: str, ecosystem: str,
                     github_repo: str, github_token: str = "") -> TrustScore:
    """Run the full five-component trust evaluation for an OSS project."""
    print(f"Evaluating {package_name} ({ecosystem}) — {github_repo}")

    components = [
        fetch_scorecard(github_repo),
        fetch_cve_history(package_name, ecosystem),
        llm_code_review(package_name, ecosystem, github_repo, github_token),
        fetch_maintainer_signals(github_repo, github_token),
        fetch_dependency_health(package_name, ecosystem),
    ]

    composite, classification = compute_composite(components)

    return TrustScore(
        package_name=package_name,
        ecosystem=ecosystem,
        github_repo=github_repo,
        composite_score=composite,
        classification=classification,
        components=components,
        next_review_trigger=[
            "New CVE published affecting this package",
            "New major version release",
            "Maintainer change detected (GitHub owner transfer)",
            "Quarterly scheduled re-evaluation",
        ],
    )


# ─── Trust Registry Storage ───────────────────────────────────────────────────

def store_score(score: TrustScore, registry_path: str = "trust-registry.json"):
    """Store the trust score in a local JSON registry."""
    registry_file = Path(registry_path)
    registry = {}
    if registry_file.exists():
        registry = json.loads(registry_file.read_text())

    key = f"{score.ecosystem}:{score.package_name}"
    registry[key] = asdict(score)
    registry_file.write_text(json.dumps(registry, indent=2))
    print(f"Score stored: {key} → {score.composite_score} ({score.classification})")


if __name__ == "__main__":
    import sys
    import os

    if len(sys.argv) < 4:
        print("Usage: oss_trust_score.py <package> <ecosystem> <github_repo>")
        print("Example: oss_trust_score.py requests PyPI psf/requests")
        sys.exit(1)

    result = evaluate_project(
        package_name=sys.argv[1],
        ecosystem=sys.argv[2],
        github_repo=sys.argv[3],
        github_token=os.environ.get("GITHUB_TOKEN", ""),
    )

    print(f"\nComposite Score: {result.composite_score}/100 ({result.classification})")
    for c in result.components:
        print(f"  {c.name}: {c.raw_score:.0f}/100 × {c.weight} = {c.weighted_score:.1f}")
    if result.components:
        for c in result.components:
            for note in c.notes:
                print(f"    ⚠ {note}")

    store_score(result)

Score Thresholds and Governance Workflow

# trust-policy.yaml — governs how trust scores map to adoption decisions

thresholds:
  green:
    min_score: 75
    action: "Approve for adoption; no additional security review required"
    re_evaluation: "Quarterly or on new CVE"
  yellow:
    min_score: 50
    max_score: 74
    action: "Conditional approval; requires security team sign-off via JIRA ticket"
    additional_review:
      - "Manual review of high_risk_modules identified in LLM code review"
      - "Compensating control documentation if single-maintainer"
    re_evaluation: "Monthly or on new CVE or maintainer change"
  red:
    max_score: 49
    action: "Requires security exception; compensating controls mandatory"
    exception_process:
      - "CISO sign-off required"
      - "Documented compensating controls"
      - "6-month sunset plan or re-evaluation trigger"
    re_evaluation: "Every new CVE; monthly scheduled"

# Automatic re-evaluation triggers (checked by CI/CD webhook handlers)
re_evaluation_triggers:
  - event: "new_cve_in_osv"
    condition: "package matches any CVE affected list"
    urgency: "within_24h"
  - event: "new_release"
    condition: "major version bump"
    urgency: "within_1_week"
  - event: "github_owner_transfer"
    condition: "repo transferred to new org"
    urgency: "immediate"
  - event: "scheduled"
    condition: "quarterly"
    urgency: "within_sprint"

Example Evaluation: HTTP Client Libraries

Running the composite framework against three common Python HTTP client libraries illustrates how the scoring differentiates projects that informal evaluation might treat similarly:

The urllib3 Yellow classification reflects its CVE track record rather than poor code quality — appropriate behaviour. The security team sign-off for urllib3 would note that the CVEs were patched promptly and the current version is clean, likely resulting in conditional approval with a note to maintain at the latest patch version.

Expected Behaviour After Hardening

With the composite framework operational and integrated into the OSS adoption workflow, every library request from an engineering team triggers an automated evaluation against the five-component model. Results are cached in the trust registry and returned within minutes for previously-evaluated packages.

Security engineers stop being the bottleneck for routine adoption decisions. Green-classified libraries are approved without additional process. Yellow and Red classifications surface automatically, prompting a targeted security review focused on the specific concerns identified in the LLM code review component rather than a generic evaluation from scratch.

The trust registry becomes a living inventory of the organisation’s dependency assessments, automatically updated when CVEs are published or when scheduled re-evaluations run. Post-incident reviews can query the registry to understand what the trust score was at the time a vulnerable library was adopted, providing evidence for retrospective programme improvement.

Trade-offs and Operational Considerations

Failure Modes

Related Articles

• OpenSSF Scorecard for Risk Scoring
• OpenSSF Scorecard and Supply Chain Security
• Supply Chain Risk Management
• Software Supply Chain Third-Party Risk
• SBOM and VEX Exploitability