Detecting and Preventing Cloud Audit Log Tampering

Detecting and Preventing Cloud Audit Log Tampering

Problem

Cloud audit log tampering is one of the first actions an attacker takes after obtaining elevated IAM credentials. The logic is consistent across attack patterns: audit logs are the primary evidence used to detect, investigate, and attribute an intrusion. Disabling or deleting them buys time. On AWS, this means stopping a CloudTrail trail. On GCP, it means disabling Cloud Audit Logs or modifying the log export sink to redirect logs to an attacker-controlled destination. On Azure, it means deleting Diagnostic Settings or disabling the Activity Log export.

This is not theoretical. Several major breach post-mortems from 2023–2025 documented that attackers with stolen IAM credentials performed log tampering within minutes of gaining access, specifically to frustrate incident response. In the Capital One 2019 breach, the attacker’s AWS API calls were logged — but only because CloudTrail happened to be active and correctly configured. Subsequent attackers have been more sophisticated about eliminating that trail.

The structural problem is that cloud audit logging is configured per-account, and the configuration itself is mutable by anyone with IAM permissions to modify it. This creates a circular dependency: you need logs to detect unauthorized changes to your logging configuration, but an attacker can disable the logging before making the changes they want to conceal.

Common tampering techniques by platform:

AWS:

  • cloudtrail:StopLogging — pauses CloudTrail event delivery without deleting history
  • cloudtrail:DeleteTrail — removes the trail configuration entirely
  • cloudtrail:UpdateTrail — redirects log delivery to an attacker-controlled S3 bucket
  • logs:DeleteLogGroup / logs:DeleteLogStream — deletes CloudWatch Logs groups
  • s3:DeleteObject / s3:DeleteBucket — destroys the S3 bucket containing archived logs
  • kms:DisableKey / kms:ScheduleKeyDeletion — removes the KMS key that decrypts encrypted logs

GCP:

  • logging.sinks.delete — removes a log export sink, stopping log archival
  • logging.sinks.update — redirects sink destination to attacker-controlled bucket
  • logging.exclusions.create — creates a log exclusion to filter out specific API calls
  • iam.serviceAccounts.delete — removes the service account that the log sink uses

Azure:

  • microsoft.insights/diagnosticSettings/delete — removes diagnostic settings for resource logs
  • microsoft.operationalinsights/workspaces/delete — deletes the Log Analytics workspace
  • microsoft.storage/storageaccounts/delete — destroys the storage account holding log archives

The secondary problem is that even when logging is active, many organizations do not monitor their logging configuration for changes. The detection gap between when tampering occurs and when it is noticed is often measured in hours or days — long enough for an attacker to complete their objective and exit.

Target systems: AWS accounts with CloudTrail; GCP projects with Cloud Audit Logs; Azure subscriptions with Activity Log and Diagnostic Settings; any multi-cloud environment using centralized SIEM with cloud audit log integration.


Threat Model

Adversary 1 — Compromised IAM admin credential. Access level: AWS IAM user or role with cloudtrail:*, logs:*, and s3:* permissions. Objective: stop CloudTrail, delete the last 24 hours of CloudWatch Logs, then proceed with data exfiltration or ransomware deployment — confident that no forensic evidence will link the intrusion to specific API calls.

Adversary 2 — Insider threat with cloud access. Access level: legitimate employee with elevated AWS/GCP/Azure permissions. Objective: disable audit logging before performing unauthorized data access, then re-enable it — relying on the assumption that no alert fires for temporary disablement.

Adversary 3 — Compromised CI/CD pipeline role. Access level: AWS role assumed by a CI/CD system with overly broad permissions inherited during setup. Objective: use the role to stop CloudTrail and exfiltrate secrets before the compromise is detected.

Adversary 4 — Ransomware operator. Access level: domain admin or cloud admin credentials obtained via phishing. Objective: disable all logging and monitoring before deploying ransomware, maximizing the dwell time before detection and preventing accurate forensic reconstruction of the attack path.

Without controls: log tampering succeeds; audit trail is destroyed; incident response relies on memory and incomplete evidence. With controls: immutable WORM archives prevent retroactive deletion; cross-account monitoring detects tampering in seconds; SCPs prevent disablement even by root.


Configuration / Implementation

Step 1 — Implement immutable S3 log archival with Object Lock

S3 Object Lock in WORM mode prevents log objects from being deleted or overwritten for a defined retention period — even by the bucket owner, even by the root account:

# Create a dedicated log archive bucket with Object Lock enabled
# Object Lock must be enabled at bucket creation — cannot be added later
aws s3api create-bucket \
  --bucket org-cloudtrail-immutable-$(date +%Y) \
  --region us-east-1 \
  --object-lock-enabled-for-bucket \
  --create-bucket-configuration LocationConstraint=us-east-1

# Set default Object Lock retention: 365 days GOVERNANCE mode
# (COMPLIANCE mode prevents even root from deleting; GOVERNANCE allows S3:BypassGovernanceRetention)
aws s3api put-object-lock-configuration \
  --bucket org-cloudtrail-immutable-$(date +%Y) \
  --object-lock-configuration '{
    "ObjectLockEnabled": "Enabled",
    "Rule": {
      "DefaultRetention": {
        "Mode": "COMPLIANCE",
        "Days": 365
      }
    }
  }'

# Enable versioning (required for Object Lock)
aws s3api put-bucket-versioning \
  --bucket org-cloudtrail-immutable-$(date +%Y) \
  --versioning-configuration Status=Enabled

# Block public access
aws s3api put-public-access-block \
  --bucket org-cloudtrail-immutable-$(date +%Y) \
  --public-access-block-configuration \
    "BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true"

Configure CloudTrail to deliver to the locked bucket:

aws cloudtrail update-trail \
  --name org-management-trail \
  --s3-bucket-name org-cloudtrail-immutable-$(date +%Y) \
  --include-global-service-events \
  --is-multi-region-trail \
  --enable-log-file-validation

aws cloudtrail start-logging --name org-management-trail

Step 2 — Deploy cross-account log archival

The primary defence against account-level tampering is storing logs in a separate, independent account that the primary account cannot modify:

# In the LOG ARCHIVE account (separate from the primary account):

# Create the archive bucket with cross-account write-only access
ARCHIVE_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
PRIMARY_ACCOUNT_ID="123456789012"

aws s3api put-bucket-policy \
  --bucket org-audit-archive \
  --policy "{
    \"Version\": \"2012-10-17\",
    \"Statement\": [
      {
        \"Sid\": \"AWSCloudTrailWrite\",
        \"Effect\": \"Allow\",
        \"Principal\": {
          \"Service\": \"cloudtrail.amazonaws.com\"
        },
        \"Action\": \"s3:PutObject\",
        \"Resource\": \"arn:aws:s3:::org-audit-archive/AWSLogs/${PRIMARY_ACCOUNT_ID}/*\",
        \"Condition\": {
          \"StringEquals\": {
            \"s3:x-amz-acl\": \"bucket-owner-full-control\"
          }
        }
      },
      {
        \"Sid\": \"DenyDeleteFromPrimary\",
        \"Effect\": \"Deny\",
        \"Principal\": {
          \"AWS\": \"arn:aws:iam::${PRIMARY_ACCOUNT_ID}:root\"
        },
        \"Action\": [
          \"s3:DeleteObject\",
          \"s3:DeleteObjectVersion\",
          \"s3:DeleteBucket\"
        ],
        \"Resource\": [
          \"arn:aws:s3:::org-audit-archive\",
          \"arn:aws:s3:::org-audit-archive/*\"
        ]
      }
    ]
  }"

Step 3 — Create real-time alerts for logging tampering

CloudWatch Events + SNS provides sub-minute alerting when logging is modified:

# Create CloudWatch Events rule for CloudTrail tampering events
aws events put-rule \
  --name "detect-cloudtrail-tampering" \
  --event-pattern '{
    "source": ["aws.cloudtrail"],
    "detail-type": ["AWS API Call via CloudTrail"],
    "detail": {
      "eventName": [
        "StopLogging",
        "DeleteTrail",
        "UpdateTrail",
        "DeleteEventDataStore",
        "StopEventDataStoreIngestion"
      ]
    }
  }' \
  --state ENABLED

# Create SNS topic for critical security alerts
aws sns create-topic --name security-critical-alerts
aws sns subscribe \
  --topic-arn "arn:aws:sns:us-east-1:${ACCOUNT_ID}:security-critical-alerts" \
  --protocol email \
  --notification-endpoint "security-team@example.com"

# Attach SNS to the CloudWatch Events rule
aws events put-targets \
  --rule "detect-cloudtrail-tampering" \
  --targets "[{
    \"Id\": \"notify-security\",
    \"Arn\": \"arn:aws:sns:us-east-1:${ACCOUNT_ID}:security-critical-alerts\"
  }]"
# Also alert on S3 log bucket modifications
aws events put-rule \
  --name "detect-log-bucket-tampering" \
  --event-pattern '{
    "source": ["aws.s3"],
    "detail-type": ["AWS API Call via CloudTrail"],
    "detail": {
      "eventName": [
        "DeleteBucket",
        "DeleteBucketPolicy",
        "PutBucketPolicy",
        "DeleteObject",
        "DeleteObjects"
      ],
      "requestParameters": {
        "bucketName": ["org-cloudtrail-immutable-2026", "org-audit-archive"]
      }
    }
  }' \
  --state ENABLED

Step 4 — Implement AWS SCP to deny logging disablement

AWS Service Control Policies enforce constraints that no IAM policy can override, including root account actions within member accounts:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyCloudTrailTampering",
      "Effect": "Deny",
      "Action": [
        "cloudtrail:StopLogging",
        "cloudtrail:DeleteTrail",
        "cloudtrail:UpdateTrail",
        "cloudtrail:DeleteEventDataStore",
        "cloudtrail:StopEventDataStoreIngestion"
      ],
      "Resource": "*",
      "Condition": {
        "ArnNotLike": {
          "aws:PrincipalArn": [
            "arn:aws:iam::*:role/SecurityBreakGlassRole",
            "arn:aws:iam::*:role/OrganizationAccountAccessRole"
          ]
        }
      }
    },
    {
      "Sid": "DenyLogBucketDeletion",
      "Effect": "Deny",
      "Action": [
        "s3:DeleteBucket",
        "s3:DeleteBucketPolicy",
        "s3:PutBucketPolicy"
      ],
      "Resource": "arn:aws:s3:::*cloudtrail*",
      "Condition": {
        "ArnNotLike": {
          "aws:PrincipalArn": "arn:aws:iam::*:role/SecurityBreakGlassRole"
        }
      }
    },
    {
      "Sid": "DenyKMSKeyDeletion",
      "Effect": "Deny",
      "Action": [
        "kms:DisableKey",
        "kms:ScheduleKeyDeletion",
        "kms:DeleteImportedKeyMaterial"
      ],
      "Resource": "*",
      "Condition": {
        "StringLike": {
          "kms:RequestAlias": "*cloudtrail*"
        }
      }
    }
  ]
}

Apply the SCP at the Organization root or to the OU containing production accounts:

aws organizations create-policy \
  --name "ProtectAuditLogging" \
  --type SERVICE_CONTROL_POLICY \
  --content file://protect-audit-logging-scp.json

POLICY_ID=$(aws organizations list-policies --filter SERVICE_CONTROL_POLICY \
  --query "Policies[?Name=='ProtectAuditLogging'].Id" --output text)

aws organizations attach-policy \
  --policy-id "$POLICY_ID" \
  --target-id r-xxxx  # Organization root ID

Step 5 — GCP equivalent controls

# GCP: Create an organization-level log export sink to a protected bucket
# in a separate project

# Create protected project for log archive
gcloud projects create org-log-archive-prod \
  --organization=$ORG_ID \
  --name="Log Archive"

# Create log sink at org level with protected destination
gcloud logging sinks create org-audit-archive \
  storage.googleapis.com/org-audit-logs-$(date +%Y) \
  --organization=$ORG_ID \
  --include-children \
  --log-filter='logName:"cloudaudit.googleapis.com"'

# Prevent sink deletion via IAM condition
gcloud organizations add-iam-policy-binding $ORG_ID \
  --member="serviceAccount:security-automation@org-log-archive-prod.iam.gserviceaccount.com" \
  --role="roles/logging.admin" \
  --condition='expression=resource.name!="logging.sinks/org-audit-archive",title="no-sink-deletion"'

# Alert on sink deletion
gcloud alpha monitoring policies create \
  --display-name="Audit Log Sink Deleted" \
  --condition-display-name="Log sink deletion" \
  --notification-channels="projects/$PROJECT/notificationChannels/$CHANNEL_ID" \
  --documentation-content="A Cloud Logging export sink was deleted. Investigate immediately."

Step 6 — Monitor for logging gaps

Log delivery gaps (periods where no events appear in the log stream) can indicate tampering or delivery failure:

#!/usr/bin/env python3
"""Detect gaps in CloudTrail log delivery."""
import boto3
from datetime import datetime, timedelta, timezone

def check_log_delivery_gap(trail_name: str, gap_threshold_minutes: int = 15) -> None:
    ct = boto3.client('cloudtrail')
    s3 = boto3.client('s3')
    
    trail = ct.get_trail(Name=trail_name)['Trail']
    bucket = trail['S3BucketName']
    
    # Find the most recent log file in S3
    prefix = f"AWSLogs/{boto3.client('sts').get_caller_identity()['Account']}/CloudTrail/"
    response = s3.list_objects_v2(Bucket=bucket, Prefix=prefix)
    
    if not response.get('Contents'):
        print(f"ALERT: No log files found in {bucket}/{prefix}")
        return
    
    latest = max(response['Contents'], key=lambda o: o['LastModified'])
    age_minutes = (datetime.now(timezone.utc) - latest['LastModified']).total_seconds() / 60
    
    if age_minutes > gap_threshold_minutes:
        print(f"ALERT: Last CloudTrail log is {age_minutes:.0f} minutes old "
              f"(threshold: {gap_threshold_minutes}m) — possible tampering or delivery failure")
        print(f"Last file: {latest['Key']}")
    else:
        print(f"OK: CloudTrail log delivery current ({age_minutes:.0f}m ago)")

if __name__ == "__main__":
    check_log_delivery_gap("org-management-trail", gap_threshold_minutes=15)

Run this check every 5 minutes via Lambda or cron:

# Cron: run every 5 minutes and alert if gap detected
*/5 * * * * /usr/local/bin/check-cloudtrail-gap.py | \
  grep ALERT | \
  xargs -I{} aws sns publish \
    --topic-arn arn:aws:sns:us-east-1:${ACCOUNT_ID}:security-critical-alerts \
    --message "{}"

Expected Behaviour

Signal Before hardening After hardening
aws cloudtrail stop-logging by compromised role Succeeds; logging paused silently Blocked by SCP — AccessDenied
aws s3 rm s3://log-bucket/ --recursive Succeeds if no Object Lock Fails — COMPLIANCE mode Object Lock prevents deletion
CloudTrail disabled for > 15 minutes No alert Log delivery gap alert fires within 5 minutes of detection
cloudtrail:UpdateTrail to redirect logs Succeeds Alert fires within 60 seconds; cross-account archive preserves original stream
Log tampering in GCP (sink deletion) No alert IAM alert fires; organization-level sink continues to archive

Verification:

# Confirm Object Lock is in COMPLIANCE mode
aws s3api get-object-lock-configuration \
  --bucket org-cloudtrail-immutable-2026 \
  --query 'ObjectLockConfiguration.Rule.DefaultRetention'
# Expected: {"Mode": "COMPLIANCE", "Days": 365}

# Test: attempt to stop logging (should fail due to SCP)
aws cloudtrail stop-logging --name org-management-trail 2>&1
# Expected: An error occurred (AccessDeniedException): ...

# Confirm CloudWatch Events rule is active
aws events describe-rule --name detect-cloudtrail-tampering \
  --query 'State'
# Expected: "ENABLED"

Trade-offs

Aspect Benefit Cost Mitigation
COMPLIANCE Object Lock Cannot be bypassed by any IAM principal Cannot delete logs even for legitimate GDPR right-to-erasure requests Use GOVERNANCE mode for GDPR-subject logs; COMPLIANCE mode for non-personal data logs; document the separation
SCP denying StopLogging No IAM policy can override, including root Complicates legitimate maintenance and trail migration Create a SecurityBreakGlassRole exempted from the SCP; require two-person authorization to assume it
Cross-account archival Attacker in primary account cannot reach archive Requires managing a second AWS account and cross-account IAM Use AWS Organizations log archive account; cost is minimal (storage only)
5-minute gap detection Near-real-time detection of tampering May alert on transient CloudTrail delivery delays (usually < 15 minutes) Set gap threshold to 20 minutes to avoid false positives from delivery lag

Failure Modes

Failure Symptom Detection Recovery
SCP blocks legitimate log trail migration New trail creation in updated region fails; migration blocked CloudFormation or Terraform returns AccessDeniedException Temporarily exempt the migration role from the SCP; perform migration; re-apply SCP
Object Lock period too long conflicts with data retention policy Legal hold expires but logs cannot be deleted; storage costs accumulate Cost alert; legal team notification Set retention period to match your policy at bucket creation; cannot be shortened after COMPLIANCE lock is set
Log delivery gap alert floods on-call CloudTrail delivery delay (normal) triggers alert repeatedly High alert volume; on-call fatigue Increase gap threshold to 30 minutes; cross-check with CloudTrail metrics before alert fires
GCP org-level sink stopped by service account permission change No logs appear in archive bucket Monitor for sink write failures in Cloud Monitoring Restore sink service account permissions; verify sink status with gcloud logging sinks describe