HAProxy 3.3 Native ACME with CertLocker, Part 2: Closing the Key Mismatch

The first article catalogued the pitfalls: reuse-key needing an argument, ssl-f-use not presenting a certificate, the admin socket race condition, key type and curve mismatches, and the Replay-Nonce cascade. All of those were solvable. The harder problem was left standing at the end of that article, and it only became clear after running the full stack against a live CertLocker deployment.

HAProxy generates a private key per ACME order and validates that the returned certificate's public key matches. CertLocker returns a pre-provisioned certificate from the vault. Those two things are in direct conflict unless someone ensures the keys agree before the order fires.

The key mismatch in detail

When HAProxy runs an ACME order, the sequence is:

1. Generate a fresh private key (or reuse the PEM key if reuse-key on).
2. Build a CSR embedding the corresponding public key.
3. Send new-order, then finalize with the CSR to the ACME gateway.
4. Download the issued certificate.
5. Check that the public key in the downloaded certificate matches the private key used for the CSR.
6. If they match: stage the certificate. If they do not: discard it. No log line. No error. Just silence.

CertLocker's ACME gateway does not issue a fresh certificate for every CSR. It stores a certificate in the vault and returns that stored bundle — including the private key — when the ACME order completes. The stored certificate was provisioned with a specific key pair that HAProxy has never seen.

Step 5 always failed. HAProxy generated its own key, sent a CSR, CertLocker returned a certificate with a completely different public key, and HAProxy discarded it.

reuse-key on was the right idea — reuse the existing PEM key so the CSR always carries the same public key — but there was no meaningful key to reuse until the container had already received a valid certificate. On a fresh deployment, the PEM file contained a throwaway self-signed key. Reusing that key meant the CSR carried the wrong public key. The cert was still discarded.

The fix: store the key in CertLocker first

The solution required a change on the CertLocker side. ACME workflows in CertLocker now store the private key associated with the certificate. When you create or edit an ACME workflow, you can supply the private key from the cert alongside the token and certificate ID.

# Create ACME workflow with a stored private key
# The key is uploaded once and referenced at deploy time
POST /rest/acme/workflow-create
{
  "name": "certlocker-haproxy",
  "certId": 42,
  "tokenId": 7,
  "keyPair": "-----BEGIN EC PRIVATE KEY-----\n...\n-----END EC PRIVATE KEY-----"
}

Open full-size screenshot

With the key stored in CertLocker, the deployment path becomes tractable. Before the HAProxy container starts, the private key is pulled from CertLocker and written to certlocker.acme.pem on the host. HAProxy reads that key, uses it for every ACME CSR via reuse-key on, CertLocker returns the matching certificate, and the validation in step 5 passes.

The key now flows in one direction: CertLocker vault → host volume → certlocker.acme.pem → ACME CSR → CertLocker matches the public key → cert issued and staged.

The three-PEM architecture

Once the key mismatch was understood, the PEM file layout needed to be redesigned. The original image baked a single certificate PEM and used it for both the crt-store ACME reference and the :443 bind. That conflated three distinct roles.

The current image maintains three separate PEM files:

• bootstrap.pem — a throwaway self-signed certificate generated on first run. Its only job is to satisfy the crt-store requirement that a certificate file exists on disk before HAProxy parses its config. :443 does not use it directly after the first start.
• certlocker.acme.pem — the ACME staging file. This must be pre-seeded with the real private key registered to the CertLocker ACME client. HAProxy loads this into the crt-store alias @certlocker-certs/certlocker, and with reuse-key on, uses its private key for every ACME CSR. After a successful ACME order, HAProxy writes the returned certificate into this file in memory.
• live.pem — the only file bound to :443. On first run it is seeded from bootstrap.pem. After a successful ACME order and key validation, the entrypoint promotes the staged bundle from certlocker.acme.pem into live.pem via the HAProxy Runtime API.

acme certlocker
    directory ${ACME_DIRECTORY}
    account-key /usr/local/etc/haproxy/certs/acme.account.key
    contact ${ACME_CONTACT}
    challenge http-01
    keytype ECDSA
    bits 256
    reuse-key on
    map virt@acme

crt-store certlocker-certs
    load crt "/usr/local/etc/haproxy/certs/certlocker.acme.pem" \
        acme certlocker \
        domains ${ACME_DOMAINS} \
        alias "certlocker"

# :443 binds live.pem — the entrypoint promotes validated ACME bundles into live.pem
frontend fe_public
    bind *:443 ssl crt "/usr/local/etc/haproxy/certs/live.pem" alpn h2,http/1.1

The important separation is that :443 never serves the ACME staging alias directly. The staging alias can contain in-progress or recently fetched material that has not yet been validated. Only after the entrypoint confirms that the private key matches the leaf certificate does it promote the bundle into live.pem and clients begin receiving the real certificate.

Startup validation

The image validates certlocker.acme.pem before HAProxy starts. The check is straightforward: extract the public key from the private key in the PEM, extract the public key from the certificate in the PEM, compare the SHA-256 hashes. If they do not match, startup fails with an actionable error message.

# Entrypoint validation on container start
# certlocker.acme.pem must contain a private key that matches its leaf certificate
ACME_PEM_KEY_HASH=$(openssl pkey -in "${CERT_DIR}/certlocker.acme.pem" -pubout 2>/dev/null \
  | openssl dgst -sha256 | awk '{print $2}')
ACME_PEM_CERT_HASH=$(openssl x509 -in "${CERT_DIR}/certlocker.acme.pem" -pubkey -noout 2>/dev/null \
  | openssl dgst -sha256 | awk '{print $2}')

if [[ "${ACME_PEM_KEY_HASH}" != "${ACME_PEM_CERT_HASH}" ]]; then
  echo "ERROR: certlocker.acme.pem private key does not match its certificate."
  echo "Seed the correct key from CertLocker before starting HAProxy."
  exit 1
fi
echo "ACME reusable key public sha256: ${ACME_PEM_KEY_HASH}"

This check exists because the key mismatch is silent at the HAProxy layer. A container that starts with the wrong key in certlocker.acme.pem will appear healthy — HAProxy loads, ACME fires, the cert is fetched — and then the cert is discarded and :443 serves the bootstrap self-signed certificate indefinitely. Failing loudly at startup is the right failure mode.

For CI and lab smoke tests where a real matched PEM is not available, set CERTLOCKER_ACME_REQUIRE_SEEDED_KEY=0 to skip the validation. Production deployments should never set this.

Ansible deployment

The key seeding step belongs in the deployment pipeline, not in the container image. The CertLocker Ansible collection pulls the PEM secret from CertLocker by stable name and writes it to the host before the HAProxy container starts.

# roles/certlocker_stack/tasks/haproxy_seed_pem.yml
# Pull the private key from CertLocker and write certlocker.acme.pem
- name: Retrieve seeded HAProxy ACME private key
  certlocker.gateway.secret:
    api_url: "{{ certlocker_api_url }}"
    token: "{{ certlocker_token }}"
    name: "ansible.{{ inventory_name }}.haproxy.acme.pem"
    asset_type: pem_key
  register: haproxy_acme_pem_result

- name: Write certlocker.acme.pem on target host
  copy:
    content: "{{ haproxy_acme_pem_result.value }}"
    dest: "{{ certlocker_haproxy_certs_dir }}/certlocker.acme.pem"
    mode: "0600"
  no_log: true

The PEM secret name follows the same naming convention used for other deployment secrets: ansible.<inventory>.haproxy.acme.pem. It is uploaded once when the certificate key pair is provisioned and rotated whenever the certificate is renewed with a new key.

The key is written with mode 0600 and no_log: true. It never appears in Ansible output and is not stored in the repo. After the deploy step runs and before docker compose up executes, the correct key is sitting on the host ready for HAProxy to find.

The full boot sequence

1. Entrypoint creates acme.account.key if absent (P-384 EC; signs ACME JWS, not the cert).
2. Entrypoint creates bootstrap.pem if absent — a throwaway self-signed cert so :443 can bind immediately.
3. live.pem is seeded from bootstrap.pem on first run; thereafter it holds the last promoted real certificate.
4. Entrypoint validates certlocker.acme.pem: the private key must match the leaf certificate. Startup fails if they do not match (CERTLOCKER_ACME_REQUIRE_SEEDED_KEY=0 skips this in CI only).
5. Config template is rendered with envsubst; HAProxy starts and :443 binds live.pem.
6. Entrypoint polls the admin socket, retrying on Connection refused until HAProxy accepts connections.
7. Once ready, acme renew @certlocker-certs/certlocker fires via socat.
8. HAProxy builds a CSR using the certlocker.acme.pem private key (reuse-key on). The public key in the CSR matches the key registered in CertLocker.
9. CertLocker receives the CSR, finds the matching certificate in the vault, returns a coherent PEM bundle.
10. HAProxy validates: private key ↔ returned certificate public key match. Cert is staged into @certlocker-certs/certlocker.
11. Entrypoint background loop detects the staged bundle, validates the match independently, then promotes it into live.pem via Runtime API (set ssl cert + commit ssl cert).
12. Clients served by :443 now receive the real Let's Encrypt wildcard certificate.

Open full-size screenshot

Lab proof

The end-to-end lab uses a minimal ACME mock server that captures the finalize request and logs the decoded CSR. The proof output is four lines:

ACME reusable key public sha256: a3f1c2...
POST /finalize/1; decoded JWS payload keys=['csr']
CSR DECODED: subject=CN=lab.local; Public-Key: (256 bit)
finalize_csr_public_sha256=a3f1c2...
acme: @certlocker-certs/certlocker: Successful update of the certificate.

The hash on the first line comes from the seeded certlocker.acme.pem private key. The hash on the fourth line comes from the CSR that HAProxy sent in the finalize request. They match. The mock server returns a certificate built from that public key. HAProxy validates the match and stages the cert. The entrypoint promotes it into live.pem.

That four-line match is the thing that was missing for months. Every other piece was in place: the ACME wire protocol was working, the config was correct, the admin socket retry was correct, the nonce handling was fixed. The one thing that needed to change was ensuring the key in the PEM file matched the key in the vault before HAProxy ever started.

Verification

# Confirm the staged ACME cert is in memory
echo "show ssl cert @certlocker-certs/certlocker" \
    | socat stdio /var/run/haproxy/admin.sock

# Confirm the promoted live cert is what :443 serves
echo "show ssl cert /usr/local/etc/haproxy/certs/live.pem" \
    | socat stdio /var/run/haproxy/admin.sock

# Confirm the cert is being presented over TLS with the right issuer
openssl s_client -connect 127.0.0.1:443 -servername your.domain.com </dev/null 2>&1 \
    | grep -E "subject|issuer|Verify|Protocol|Cipher"

Expected output after a successful boot:

subject=CN=certlocker.io
issuer=C=US, O=Let's Encrypt, CN=E8
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Verify return code: 0 (ok)

Summary

HAProxy native ACME against CertLocker now works end to end, including across container restarts. The change that made it work was not in HAProxy configuration — the config was already correct. The change was storing the certificate private key in CertLocker as part of the ACME workflow and seeding it into certlocker.acme.pem before the container starts.

With that key in place, reuse-key on does what it is supposed to do: every ACME CSR carries the same public key, CertLocker finds the matching certificate in the vault, and HAProxy accepts the result without discarding it.

The three-PEM split — bootstrap, staging, live — keeps the trust boundary explicit. The startup validation check keeps deployments honest. The Ansible seeding step keeps the key out of the image and under access control.

If you are running HAProxy 3.3 native ACME against a custom CA that returns pre-provisioned certificates rather than issuing fresh ones, the pattern generalises: ensure the ACME staging PEM contains the private key that matches the certificate your CA is going to return. The rest of the HAProxy configuration follows from there.