SMTP DANE and MTA-STS: Preventing Opportunistic TLS Downgrade in Mail Delivery

The Problem

SMTP between mail servers uses opportunistic TLS by default: the sending MTA issues STARTTLS if the receiving MTA advertises it, but if the receiving server responds with “I don’t support TLS” — even falsely — the sending MTA falls back to plaintext and delivers the message anyway. This is by design: the SMTP specification prioritises delivery reliability over confidentiality, and the receiving server’s STARTTLS advertisement is unauthenticated.

A network attacker with a man-in-the-middle position between two MTAs can strip the STARTTLS capability advertisement, causing the sending MTA to deliver email in cleartext. This attack is trivially achievable on any network where the attacker controls a BGP route, a resolver, or a Wi-Fi access point. The attack leaves no trace in the sending server’s logs beyond the absence of a “TLS handshake” line.

The second variant is certificate substitution: the attacker allows STARTTLS to proceed but presents their own certificate for the receiving domain. Because SMTP MTAs traditionally accept any certificate without hostname validation (a behaviour inherited from the pre-SNI era), the sending MTA upgrades to TLS but delivers to the attacker’s server rather than the real recipient.

Two standards address this, complementarily:

DANE (DNS-Based Authentication of Named Entities, RFC 7671): publishes a TLSA record in DNS, signed with DNSSEC, that specifies exactly which certificate (or certificate authority, or public key hash) the MTA should expect when connecting to the receiving mail server. The sending MTA validates the certificate against the TLSA record; if they don’t match, delivery is aborted. DANE requires DNSSEC on the receiving domain.

MTA-STS (Mail Transfer Agent Strict Transport Security, RFC 8461): publishes a policy file over HTTPS at https://mta-sts.<domain>/.well-known/mta-sts.txt and a DNS TXT record announcing that the policy exists. The sending MTA fetches the policy and caches it; subsequent deliveries must use TLS with a valid WebPKI certificate or be rejected. MTA-STS does not require DNSSEC.

The two mechanisms have a complementary failure profile: DANE provides stronger guarantees (cryptographic proof via DNSSEC) but requires DNSSEC deployment. MTA-STS is deployable without DNSSEC but relies on WebPKI and has a first-fetch vulnerability window where the initial policy fetch occurs without TLS enforcement.

Target systems: Postfix 3.5+, Exim 4.91+, Sendmail with DANE patches; receiving domains controlled by the organisation; sending MTAs that support DANE (Postfix, Exim) or MTA-STS (Postfix with mta-sts-daemon, Google Workspace, Microsoft 365).

Threat Model

1. BGP hijack attacker (nation-state or well-resourced criminal). Objective: redirect SMTP traffic for a target domain to a controlled server; strip STARTTLS to receive plaintext email or perform certificate substitution. Impact: email confidentiality and integrity completely compromised for the hijacked routes.

2. Authoritative DNS resolver compromise (attacker with access to the receiving domain’s DNS zone). Objective: modify MX records to point to an attacker-controlled server; optionally strip TLSA records to prevent DANE validation. Impact: email redirection without TLS validation failure (if DNSSEC is absent).

3. ISP or transit provider interception (surveillance adversary). Objective: passive interception of SMTP traffic between MTAs in the same AS or peering point. Impact: bulk email content harvested without the sending organisation being aware.

4. Certificate mis-issuance (compromised CA or CT-logged misissuance). Objective: obtain a valid WebPKI certificate for the receiving domain; use it to intercept MTA-STS-protected delivery. Impact: MTA-STS provides no protection against this; DANE (using TLSA 3 1 1 pin-to-pubkey) prevents it.

Without DANE or MTA-STS, attackers 1 and 3 can silently downgrade any SMTP session to plaintext. With MTA-STS deployed (no DNSSEC), attacker 4 remains viable. Full protection against all four threat actors requires DANE with DNSSEC.

Hardening Configuration

Phase 1: Deploy MTA-STS (No DNSSEC Required)

MTA-STS is the lower-barrier starting point. It requires two things: an HTTPS endpoint serving the policy file, and a DNS TXT record.

Step 1: Create the MTA-STS policy file

# https://mta-sts.example.com/.well-known/mta-sts.txt
version: STSv1
mode: enforce
mx: mail.example.com
mx: mail2.example.com
max_age: 86400

Field explanations:

• mode: enforce — reject delivery if TLS cannot be established with a valid WebPKI cert. Use mode: testing initially to detect failures without blocking delivery.
• mx: — list every MX host for the domain. A sending MTA checks that the MX it’s connecting to is listed here.
• max_age — cache duration in seconds. Start with 86400 (1 day) during testing; raise to 604800 (1 week) once stable.

Host the policy file at https://mta-sts.example.com/.well-known/mta-sts.txt. The mta-sts subdomain must serve a valid WebPKI TLS certificate.

Step 2: Publish the DNS TXT record

# DNS zone for example.com
_mta-sts.example.com.  IN TXT "v=STSv1; id=20260608T000000Z"

The id field is a timestamp or opaque identifier that changes when the policy changes. Sending MTAs that have cached a policy re-fetch when the id changes.

Step 3: Publish TLS-RPT for failure reporting

# DNS zone for example.com
_smtp._tls.example.com.  IN TXT "v=TLSRPTv1; rua=mailto:tls-rpt@example.com"

TLS-RPT (RFC 8460) causes sending MTAs to email you aggregate reports of SMTP TLS failures and policy violations. Review these reports daily during the testing phase; failures indicate delivery issues that would become blocks under enforce mode.

Step 4: Configure Postfix as an MTA-STS-enforcing sender

# Install mta-sts-daemon (Python implementation)
pip3 install postfix-mta-sts-resolver

# /etc/mta-sts-daemon.yml
host: 127.0.0.1
port: 8461
cache:
  type: sqlite
  options:
    filename: /var/lib/mta-sts/cache.db
reuse_cached_on_failure: false   # fail closed if policy fetch fails

# Start daemon
systemctl enable --now mta-sts-daemon

# /etc/postfix/main.cf additions
smtp_tls_policy_maps = socketmap:inet:127.0.0.1:8461:postfix
smtp_tls_security_level = may    # daemon overrides to verify/secure per-domain
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt

Verify that the daemon is enforcing policy:

# Query the daemon for a specific domain
echo "example.com" | socat - TCP:127.0.0.1:8461
# Expected: secure match=nexthop

Phase 2: Deploy DANE (DNSSEC Required)

DANE provides stronger guarantees but requires the receiving domain to be signed with DNSSEC.

Step 1: Enable DNSSEC on the receiving domain

# With BIND, generate zone-signing keys
dnssec-keygen -a ECDSAP256SHA256 -b 256 -n ZONE example.com
dnssec-keygen -a ECDSAP256SHA256 -b 256 -n ZONE -f KSK example.com

# Sign the zone
dnssec-signzone -A -3 $(head -c 6 /dev/urandom | xxd -p) -N INCREMENT \
  -o example.com -t db.example.com

# Publish DS record to parent zone
# (provide the DS digest output from dnssec-signzone to your registrar)

Step 2: Generate TLSA records for MX hosts

# Extract the SHA-256 hash of the MX host's TLS certificate public key
# Type 3 1 1 = DANE-EE (end-entity), SubjectPublicKeyInfo, SHA-256
openssl s_client -connect mail.example.com:25 -starttls smtp 2>/dev/null | \
  openssl x509 -noout -pubkey | \
  openssl pkey -pubin -outform DER | \
  sha256sum | awk '{print $1}'

# The output becomes the TLSA record data
# _25._tcp.mail.example.com. IN TLSA 3 1 1 <hash>

Step 3: Publish TLSA records in DNS

# DNS zone for example.com (after DNSSEC signing)
_25._tcp.mail.example.com.   IN TLSA  3 1 1 \
  abc123def456...   ; SHA-256 of mail.example.com public key

_25._tcp.mail2.example.com.  IN TLSA  3 1 1 \
  789abc012def...   ; SHA-256 of mail2.example.com public key

TLSA record type field meanings:

• 3 1 1 (DANE-EE): pins the end-entity certificate’s public key hash. Most operationally convenient; doesn’t break on CA or intermediate changes.
• 2 1 1 (DANE-TA): pins the trust anchor (CA) public key. Useful when you rotate leaf certificates frequently but keep the same internal CA.

Step 4: Configure Postfix DANE on the sender side

# /etc/postfix/main.cf
smtp_dns_support_level = dnssec
smtp_tls_security_level = dane
# When a domain has TLSA records in DNSSEC-signed zones, Postfix
# automatically upgrades to DANE-verified TLS for that destination

Verify that Postfix is performing DANE validation:

# Send a test message and check Postfix logs
postqueue -f
grep "Verified TLS connection established" /var/log/mail.log
grep "DANE" /var/log/mail.log

# Expected log line:
# Verified TLS connection established to mail.example.com:25:
# TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange
# X25519 server-signature ECDSA (P-256) server-digest SHA256;
# DANE TLSA 3 1 1 verified

Monitoring and Certificate Rotation

DANE’s biggest operational risk is certificate rotation without updating TLSA records. If the TLSA record still references the old public key after a certificate rotation, DANE-validating senders will refuse delivery.

# Script to verify TLSA records match current certificate
# Run this before and after every certificate rotation

check_tlsa() {
  local host=$1
  local port=${2:-25}

  # Get current cert's pubkey hash
  current_hash=$(openssl s_client -connect "${host}:${port}" \
    -starttls smtp 2>/dev/null | \
    openssl x509 -noout -pubkey | \
    openssl pkey -pubin -outform DER | \
    sha256sum | awk '{print $1}')

  # Get published TLSA hash
  tlsa_hash=$(dig +short TLSA "_${port}._tcp.${host}" | \
    awk '/3 1 1/{print $4}')

  if [ "$current_hash" = "$tlsa_hash" ]; then
    echo "OK: TLSA matches for $host"
  else
    echo "MISMATCH: current=$current_hash published=$tlsa_hash for $host"
    return 1
  fi
}

check_tlsa mail.example.com 25
check_tlsa mail2.example.com 25

Pre-rotation procedure:

1. Generate the new certificate and extract its public key hash.
2. Add the new TLSA record to DNS before deploying the new certificate. Waiting for TTL to expire ensures the record propagates.
3. Deploy the new certificate.
4. After the old certificate TTL has passed, remove the old TLSA record.

Expected Behaviour After Hardening

Trade-offs and Operational Considerations

Failure Modes

Related Articles

• Email Security with SPF, DKIM, and DMARC
• DNSSEC Key Rollover Operational Security
• TLS 1.3 Migration and Cipher Hardening
• Encrypted DNS Infrastructure Deployment
• Certificate Transparency and TLS Certificate Monitoring