HAProxy 3.3 Native ACME with CertLocker: Everything That Can Go Wrong

CertLocker is a certificate lifecycle platform. It stores TLS certificates and delivers them via a custom ACME gateway. The idea was simple: instead of running certbot or acme.sh as a sidecar, HAProxy 3.3 has a native ACME client built in. Point HAProxy at the CertLocker gateway, give it a token, and it should fetch the right certificate and keep it renewed.

No sidecars. No cron jobs. No external certificate tooling.

HAProxy 3.3 shipped ACME support as experimental, behind an expose-experimental-directives flag. We found out the hard way what experimental means in practice.

Why it was hard

1. CertLocker's ACME model is not standard ACME

In the standard Let's Encrypt-style ACME flow, the client generates a private key, sends a CSR with the public key embedded, and the CA issues a brand new certificate for that public key.

CertLocker's model is different. The certificate is pre-provisioned and stored in the vault. The ACME gateway is a delivery mechanism, not a fresh issuance mechanism. When HAProxy calls new-order and finalize, CertLocker returns the stored certificate.

That difference broke a lot of assumptions. HAProxy was behaving like a standard ACME client. CertLocker was intentionally delivering a pre-existing certificate.

2. HAProxy generates a fresh private key per order and validates the result

This was the core problem and the one that took the longest to diagnose.

When HAProxy runs an ACME order, it generates a brand new private key, embeds the corresponding public key in the CSR, sends the CSR to the CA, downloads the certificate, and then checks that the public key in the downloaded certificate matches the private key it just generated.

If they do not match, HAProxy discards the certificate. In our testing there was no clear log line saying "key mismatch." The certificate was fetched successfully, then disappeared. HAProxy kept serving whatever was already in the PEM file.

CertLocker returned the stored certificate, which had originally been provisioned with a specific P-256 key. HAProxy generated a different key. The public keys did not match. The certificate was discarded.

3. reuse-key exists, but you have to know to look for it

Most ACME clients reuse the existing private key on renewal because that is usually the sensible default. HAProxy did not do that for this flow until the reuse-key directive was available.

reuse-key tells HAProxy to read the private key from the existing PEM file and reuse it when building the ACME CSR, instead of generating a fresh key pair. That is exactly what CertLocker needs: the CSR public key matches the stored certificate public key, so HAProxy accepts the downloaded certificate.

The directive was not obvious from the main path through the docs. We found it by reading changelogs.

4. reuse-key requires an argument

Once we found reuse-key, we added it to the config and the build broke:

[ALERT]: keyword 'reuse-key' in 'acme' section requires an argument

The correct syntax is reuse-key on, not bare reuse-key. HAProxy 3.3.7 is strict about this. One word added, half a day lost.

5. Key type and curve had to match exactly

The CertLocker-stored certificate used a P-256 ECDSA key. Our initial config was wrong:

keytype RSA
bits 2048

That caused a type mismatch. Changing to keytype EC was also wrong because HAProxy rejects it. The correct value is ECDSA. Then bits 384 was wrong because the certificate was P-256. It had to be bits 256.

Each mismatch ended the same way: HAProxy fetched the certificate and silently discarded it because the key material did not line up.

6. The bootstrap certificate is required, and its key must match

HAProxy's crt-store requires the certificate file to exist on disk before HAProxy starts. The bind on :443 cannot reference a certificate that does not exist yet. That means a bootstrap certificate is required.

Our first approach was to hardcode the real Let's Encrypt chain directly in the entrypoint. It worked, but it was brittle. The image would eventually contain stale certificate data.

The better development approach was to generate a throwaway self-signed certificate at startup using a fixed demo P-256 private key. HAProxy starts with the self-signed certificate, ACME fires immediately, and the real certificate replaces it within seconds.

The critical constraint is that the demo private key embedded in the entrypoint must have a public key that matches the public key in the CertLocker-stored certificate. That is the key reuse-key on sends in every CSR. If the keys do not match, the ACME exchange appears to work but HAProxy discards the result.

In production, the real private key should be seeded from a secrets manager such as CertLocker Secrets, Ansible Vault, or an equivalent system before the container starts. The demo key is only for development and CI.

7. ssl-f-use did not present the certificate

The documented-looking way to use a crt-store certificate in a frontend was this:

bind *:443 ssl alpn h2,http/1.1
ssl-f-use crt "@certlocker-certs/certlocker"

It compiled. HAProxy started cleanly. show ssl cert @certlocker-certs/certlocker showed the certificate was loaded. Everything looked fine.

But clients received no certificate during the TLS handshake:

openssl s_client -connect 127.0.0.1:443 -servername trust.certlocker.io
no peer certificate available

The fix was to reference the certificate directly on the bind line:

bind *:443 ssl crt "@certlocker-certs/certlocker" alpn h2,http/1.1

bind *:443 ssl crt-store certlocker-certs was also rejected by HAProxy 3.3.7 as an unknown keyword on bind. The @store/alias reference form is the one that worked.

8. The admin socket race condition

The entrypoint starts HAProxy in the background, polls for the admin socket to appear, and then fires acme renew via socat to trigger an immediate certificate fetch. This avoids waiting for HAProxy's built-in scheduler.

The problem is that HAProxy creates the socket file at bind() time, before it calls listen(). The entrypoint saw the socket file, immediately tried to connect, and got Connection refused because HAProxy was not accepting connections yet.

The original code tried once, failed, and gave up. Every restart left HAProxy serving the bootstrap self-signed certificate instead of the real certificate.

The fix was to retry on Connection refused, treating it as "socket exists but is not ready yet":

_out=$(echo "acme renew ${CERT_NAME}" | socat - UNIX-CONNECT:/var/run/haproxy/admin.sock 2>&1)
if grep -q "Connection refused" <<< "${_out}"; then
    sleep 0.05
    continue
fi

9. haproxy -c could not validate the config in CI

The standard CI check is haproxy -c -f haproxy.cfg. With expose-experimental-directives and crt-store @ references, that was not enough. The check failed because the experimental certificate loading path was not triggered in config-check mode.

We ended up validating by starting HAProxy inside the CI container, waiting a few seconds, and checking that the process was still running. It is slower and more fragile than haproxy -c, but it validates the real startup path.

10. Missing Replay-Nonce on error responses caused cascading failures

ACME requires a fresh nonce on every POST request. The server returns the next nonce in a Replay-Nonce response header. RFC 8555 requires this header on all responses, including errors.

CertLocker's ACMEExceptionMapper did not return Replay-Nonce on 4xx responses. When any request failed, HAProxy retried with an already-consumed nonce and produced a cascade of invalid nonce errors.

The short-term workaround was to create a new ACME token, which reset the server-side nonce state. The permanent fix was to return Replay-Nonce on every ACME response, including errors. That fix is now in place.

Open full-size screenshot

The working configuration

global
    expose-experimental-directives
    httpclient.ssl.ca-file /etc/ssl/certs/ca-certificates.crt

acme certlocker
    directory ${ACME_DIRECTORY}
    account-key /usr/local/etc/haproxy/certs/acme.account.key
    contact ${ACME_CONTACT}
    challenge http-01
    keytype ECDSA
    bits 256
    reuse-key on
    map virt@acme

crt-store certlocker-certs
    load crt "/usr/local/etc/haproxy/certs/certlocker.io.pem" \
        acme certlocker \
        domains ${ACME_DOMAINS} \
        alias "certlocker"

frontend fe_public
    bind *:443 ssl crt "@certlocker-certs/certlocker" alpn h2,http/1.1
    mode http

The important details are:

• expose-experimental-directives is required in global
• Use keytype ECDSA, not EC
• Use bits 256, not 384, when the stored certificate is P-256
• Use reuse-key on, not bare reuse-key
• Reference the certificate on the bind line, not through ssl-f-use
• Make sure the bootstrap PEM exists before HAProxy starts

The boot sequence

1. The entrypoint checks for acme.account.key and generates an account key if absent. The account key is separate from the certificate key and only signs ACME JWS messages.
2. The entrypoint checks for certlocker.io.pem and writes a fixed demo P-256 private key plus a one-day self-signed certificate if absent.
3. The config template is rendered with envsubst.
4. HAProxy starts and binds :443 with the self-signed bootstrap certificate.
5. The entrypoint polls the admin socket and retries on Connection refused.
6. Once HAProxy is ready, acme renew @certlocker-certs/certlocker is sent through socat.
7. HAProxy's ACME client fetches the real certificate from CertLocker.
8. The in-memory certificate store is updated and clients receive the real certificate.
9. HAProxy's scheduler handles subsequent renewal checks.

Verification

# Confirm HAProxy has the real certificate in memory
echo "show ssl cert @certlocker-certs/certlocker" \
    | socat stdio /var/run/haproxy/admin.sock

# Confirm the certificate is being presented over TLS
openssl s_client -connect 127.0.0.1:443 -servername your.domain.com </dev/null 2>&1 \
    | grep -E "subject|issuer|Verify|Protocol|Cipher"

Expected output after a successful boot should show the real certificate subject and issuer, TLS negotiation details, and Verify return code: 0 (ok).

What remains

• Certificate persistence after ACME update: HAProxy updates the in-memory crt-store, but the PEM file on disk keeps the bootstrap certificate. Every restart triggers a fresh ACME exchange.
• Production key seeding: the demo private key is for development only. Production deployments should seed the real private key from a secrets manager before container start.

Summary

HAProxy 3.3 native ACME works, but the combination of a pre-provisioned certificate model, HAProxy's default key-per-order behaviour, and several rough edges in the experimental implementation made this harder than it should have been.

The two changes that unlocked everything were reuse-key on and putting the certificate reference directly on the bind line instead of using ssl-f-use.

If you have had similar experiences getting native ACME, HAProxy, custom CAs, or certificate delivery working in production, reach out. We are interested in hearing what broke, what worked, and which patterns other infrastructure teams are relying on.