pingora-core: unsound setenv during upstream TLS connector initialization can free environ under concurrent readers #1022

Describe the bug

pingora-core writes to the process environment from Connector::new, the constructor of its upstream TLS connector, by calling openssl_probe::init_openssl_env_vars() inside a one-shot Once closure. Because the closure runs at most once, and because the call site carries a comment stating that its safety cannot be proven, the code appears to treat a single execution as sufficient for soundness. Soundness, however, is violated by concurrency rather than by repetition: Once::call_once synchronizes writers with one another, whereas std::env::set_var is unsound whenever another thread may concurrently read the environment.

On glibc 2.40 and earlier, setenv extends the environ pointer array with realloc, releasing the array that readers may currently be traversing. A thread that scans the environment through getenv, or through a direct walk of environ, can therefore continue reading the released block and later dereference a stale pointer. This is the mechanism documented in openssl-probe issue #30, where an unrelated getenv call issued by a C library brought down an entire process. The condition is far from exotic: any long-lived process that embeds C code, whether OpenSSL, gettext, a DNS resolver, or an interpreter, can host such a reader.

The defect is reachable from the public safe API under the default configuration, and it depends on no input value. TransportConnector::new(None) and the construction of an HttpProxy both suffice, since ServerConf::ca_file defaults to None and the connector consequently selects the environment-assisted trust path. Applications that construct their proxies at runtime, as the documentation encourages for custom accept loops, may therefore trigger the mutation after many threads have already started.

Root Cause

The trigger site is pingora-core/src/connectors/tls/boringssl_openssl/mod.rs:79-85:

static INIT_CA_ENV: Once = Once::new();
fn init_ssl_cert_env_vars() {
    // this sets env vars to pick up the root certs
    // it is universal across openssl and boringssl
    // safety: although impossible to prove safe we assume it's safe since the call is
    // wrapped in a call_once and it's unlikely other threads are reading these vars
    INIT_CA_ENV.call_once(|| unsafe { openssl_probe::init_openssl_env_vars() });
}

Connector::new invokes the helper whenever no explicit CA file is configured (pingora-core/src/connectors/tls/boringssl_openssl/mod.rs:94-140, with the two call sites at lines 113 and 138):

pub fn new(options: Option<ConnectorOptions>) -> Self {
    let mut builder = SslConnector::builder(SslMethod::tls()).unwrap();
    ...
    if let Some(conf) = options.as_ref() {
        if let Some(ca_file_path) = conf.ca_file.as_ref() {
            builder.set_ca_file(ca_file_path).unwrap();
        } else {
            init_ssl_cert_env_vars();
            builder.set_default_verify_paths().unwrap();
        }
        ...
    } else {
        init_ssl_cert_env_vars();
        builder.set_default_verify_paths().unwrap();
    }
    ...
}

The path to those lines is entirely public. TransportConnector::new (pingora-core/src/connectors/mod.rs:188) constructs the TLS connector directly (pingora-core/src/connectors/mod.rs:205):

tls_ctx: tls::Connector::new(options),

HttpProxy::new (pingora-proxy/src/lib.rs:228) reaches the same constructor with default options, since ServerConf::ca_file is None by default (pingora-core/src/server/configuration/mod.rs:86,225) and ConnectorOptions::from_server_conf copies the field verbatim (pingora-core/src/connectors/mod.rs:138). One call to the public constructor is therefore sufficient.

The dependency has documented the hazard and marked its entry point accordingly (openssl-probe-0.1.6/src/lib.rs:71-83):

/// # Safety
///
/// This function is not safe because it mutates the process's environment
/// variables which is generally not safe. See the [documentation in libstd][doc]
/// for information about why setting environment variables is not safe.
///
/// If possible use the [`probe`] function and directly configure OpenSSL
/// methods instead of relying on environment variables.
///
/// [doc]: https://doc.rust-lang.org/stable/std/env/fn.set_var.html#safety
pub unsafe fn init_openssl_env_vars() {
    try_init_openssl_env_vars();
}

The mutation itself is performed through std::env::set_var, separately for SSL_CERT_FILE and SSL_CERT_DIR (openssl-probe-0.1.6/src/lib.rs:112-142):

unsafe fn put(var: &str, path: &Path) {
    // Avoid calling `setenv` if the variable already has the same contents. ...
    if env::var_os(var).as_deref() != Some(path.as_os_str()) {
        unsafe {
            env::set_var(var, path);
        }
    }
}

The memory hazard that follows is a property of the C library. On glibc 2.40 and earlier, __add_to_environ extends the array with realloc (glibc stdlib/setenv.c:151-172, branch release/2.40/master):

char **new_environ;

/* We allocated this space; we can extend it.  Avoid using the raw
   reallocated pointer to avoid GCC -Wuse-after-free.  */
uintptr_t ip_last_environ = (uintptr_t)last_environ;
new_environ = (char **) realloc (last_environ,
                                 (size + 2) * sizeof (char *));
...
last_environ = __environ = new_environ;

Whenever realloc relocates the block, the previous array is returned to the allocator while concurrent readers may still be traversing it. On Unix, Rust's own environment accessors are serialized by an internal lock, so a race constructed only from std::env::var_os and std::env::set_var cannot be produced; the standard library nevertheless does not restrict its contract to its own accessors. Foreign readers that call getenv, or that walk environ, fall outside that lock, which is why std::env::set_var is documented as unsound in multithreaded programs and became an unsafe fn in Rust 2024. The comment above call_once reasons about probability, that is, how likely a reader is to be present, where the language reasons about possibility.

Suggested fix

A read-only probe removes the mutation altogether, which is the direction that openssl-probe itself recommends:

let probe = openssl_probe::probe();
if let Some(cert_file) = &probe.cert_file {
    // configure the certificate file on the SslConnector
}
if let Some(cert_dir) = &probe.cert_dir {
    // configure the certificate directory on the SslConnector
}

Should the environment-assisted path be retained for compatibility, the mutation would have to be confined to a single-threaded initialization phase, executed before any runtime, service, or background thread is started, and documented as such on Connector::new, TransportConnector::new, and HttpProxy::new. Performing the initialization in Server before run_forever, and removing the lazy trigger from Connector::new, would satisfy that constraint. Dropping the openssl-probe dependency in favor of internal CA discovery would eliminate the unsafe surface entirely.

For validation, a regression test that constructs the connector while environment readers are running would protect the property directly; the reproducer below is such a test, and it is most effective under AddressSanitizer.

Pingora info

Pingora version: 0.9.0, commit 4487f7b2ab50f159e4a2cf4f6a6b813f61bb6e19 (branch main)

Rust version: nightly-2026-03-21, the toolchain used for the AddressSanitizer build

Operating system version: Ubuntu 22.04, glibc 2.35, x86_64

Steps to reproduce

1. Save the file below as pingora-core/tests/env_setvar_environ_uaf.rs.
2. Run the sanitizer build from the repository root:

CARGO_TARGET_DIR=target/asan \
RUSTFLAGS="-Zsanitizer=address -Cunsafe-allow-abi-mismatch=sanitizer" \
ASAN_OPTIONS=detect_leaks=0 \
  cargo +nightly-2026-03-21 test --target x86_64-unknown-linux-gnu \
  -p pingora-core --features openssl --test env_setvar_environ_uaf \
  -- --nocapture --test-threads=1

The reproduction requires the openssl feature, or the boringssl feature, because the rustls and s2n backends do not call openssl-probe. --test-threads=1 prevents the control test from overlapping the trigger test. Since the closure is guarded by Once, the trigger fires once per process; rerunning the process is sufficient when an execution misses the race.

// Reproducer: pingora-core performs setenv during upstream TLS connector
// initialization, racing concurrent environment readers.
//
// Integration location: pingora-core/tests/env_setvar_environ_uaf.rs
// Run under AddressSanitizer on glibc 2.40 or earlier:
//   CARGO_TARGET_DIR=target/asan \
//   RUSTFLAGS="-Zsanitizer=address -Cunsafe-allow-abi-mismatch=sanitizer" \
//   ASAN_OPTIONS=detect_leaks=0 \
//     cargo +nightly-2026-03-21 test --target x86_64-unknown-linux-gnu \
//     -p pingora-core --features openssl --test env_setvar_environ_uaf \
//     -- --nocapture --test-threads=1
//
// Expected AddressSanitizer signature: heap-use-after-free, READ of size 8,
// in walk_environ (release stack: realloc and setenv, from openssl_probe,
// reached through Once::call_once and TransportConnector::new).
//
// The control test below must pass without a sanitizer report.
#![cfg(all(
    any(feature = "openssl", feature = "boringssl"),
    target_os = "linux",
    target_env = "gnu"
))]

use std::os::raw::c_char;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant};

// libc symbols declared directly so this file needs no extra dependencies.
unsafe extern "C" {
    static mut environ: *mut *mut c_char;
    fn getenv(name: *const c_char) -> *mut c_char;
    fn gnu_get_libc_version() -> *const c_char;
}

/// glibc 2.41 rewrote `setenv`/`getenv` so that environ arrays are kept on a
/// list and never freed by `setenv`; the ASan-visible UAF only exists on
/// glibc <= 2.40 (or on other libc implementations).
fn glibc_version() -> Option<(u32, u32)> {
    unsafe {
        let p = gnu_get_libc_version();
        if p.is_null() {
            return None;
        }
        let s = std::ffi::CStr::from_ptr(p).to_str().ok()?;
        let mut it = s.split('.');
        Some((it.next()?.parse().ok()?, it.next()?.parse().ok()?))
    }
}

fn warn_if_libc_mitigates() {
    if let Some((major, minor)) = glibc_version() {
        println!("[env-uaf] glibc {major}.{minor}");
        if (major, minor) >= (2, 41) {
            eprintln!(
                "[env-uaf] WARNING: glibc >= 2.41 keeps old environ arrays and no longer \
                 reallocs/frees them, so this ASan variant is expected to be \
                 inconclusive on this host. Run under glibc <= 2.40 (Ubuntu 22.04/24.04, \
                 Debian 12, RHEL 9) or a musl host; the Rust-level `set_var` contract \
                 violation remains regardless."
            );
        }
    } else {
        println!("[env-uaf] could not read glibc version");
    }
}

const PAD_PREFIX: &str = "PINGORA_ENV_PAD_";

struct Shared {
    stop: AtomicBool,
    scans: AtomicUsize,
}

fn knob_usize(name: &str, default: usize) -> usize {
    std::env::var(name)
        .ok()
        .and_then(|v| v.parse().ok())
        .unwrap_or(default)
}

fn env_len_at(mut p: *mut *mut c_char) -> usize {
    let mut n = 0usize;
    unsafe {
        while !p.is_null() && !(*p).is_null() {
            n += 1;
            p = p.add(1);
        }
    }
    n
}

fn env_len() -> usize {
    env_len_at(unsafe { environ })
}

fn environ_addr() -> usize {
    unsafe { environ as usize }
}

/// An instrumented scan of the `environ` pointer array: the same access pattern
/// every C library `getenv` performs, but visible to AddressSanitizer.
#[inline(never)]
unsafe fn walk_environ(mut p: *mut *mut c_char) -> usize {
    let mut n = 0usize;
    while !(*p).is_null() {
        std::hint::black_box(*p);
        n += 1;
        p = p.add(1);
    }
    n
}

/// A realistic foreign reader: the real libc `getenv` (no Rust ENV_LOCK).
fn spawn_getenv_scanner(shared: &Arc<Shared>) -> std::thread::JoinHandle<()> {
    let shared = Arc::clone(shared);
    std::thread::spawn(move || {
        let key = b"SSL_CERT_FILE\0";
        let key = key.as_ptr() as *const c_char;
        while !shared.stop.load(Ordering::Relaxed) {
            unsafe {
                std::hint::black_box(getenv(key));
            }
        }
    })
}

fn spawn_scanner(shared: &Arc<Shared>, base: usize) -> std::thread::JoinHandle<()> {
    let shared = Arc::clone(shared);
    std::thread::spawn(move || {
        let base = base as *mut *mut c_char;
        while !shared.stop.load(Ordering::Relaxed) {
            let n = unsafe { walk_environ(base) };
            std::hint::black_box(n);
            shared.scans.fetch_add(1, Ordering::Relaxed);
        }
    })
}

fn spawn_all(
    shared: &Arc<Shared>,
    base: usize,
    readers: usize,
) -> Vec<std::thread::JoinHandle<()>> {
    let mut handles = Vec::with_capacity(readers + 2);
    for _ in 0..readers {
        handles.push(spawn_scanner(shared, base));
    }
    for _ in 0..2 {
        handles.push(spawn_getenv_scanner(shared));
    }
    handles
}

fn stop_and_join(shared: &Arc<Shared>, handles: Vec<std::thread::JoinHandle<()>>) {
    shared.stop.store(true, Ordering::Relaxed);
    for h in handles {
        let _ = h.join();
    }
}

/// Size the `environ` pointer array onto an exact multiple of 4096 bytes (an
/// ASan size class boundary): the next entry then makes `setenv`'s realloc move
/// it (allocate + copy + free) instead of extending it in place, which is what
/// frees the block the readers hold. `PINGORA_PAD` overrides the entry count.
fn pad_environment() {
    let override_entries = std::env::var("PINGORA_PAD")
        .ok()
        .and_then(|v| v.parse::<usize>().ok());
    let mut i = 0usize;
    loop {
        let len = env_len();
        let done = match override_entries {
            Some(target) => len >= target,
            // allocation = (entries + 1) * 8 bytes; land on a 4096-byte multiple
            None => (len + 1) % 512 == 0,
        };
        if done || i >= 100_000 {
            break;
        }
        std::env::set_var(format!("{PAD_PREFIX}{i}"), "1");
        i += 1;
    }
}

/// Control: identical readers, no environment mutation while they run.
#[test]
fn control_environ_scan_without_mutation() {
    pad_environment();
    let base = environ_addr();
    let shared = Arc::new(Shared {
        stop: AtomicBool::new(false),
        scans: AtomicUsize::new(0),
    });
    let handles = spawn_all(&shared, base, knob_usize("PINGORA_READERS", 8));
    std::thread::sleep(Duration::from_millis(
        knob_usize("PINGORA_CONTROL_MS", 1500) as u64
    ));
    stop_and_join(&shared, handles);
    let scans = shared.scans.load(Ordering::Relaxed);
    assert!(scans > 0, "readers did not run");
    println!("[env-uaf][control] {scans} environ scans over a stable array; no report expected");
}

/// Trigger: scan `environ` while pingora-core's public safe API mutates the
/// process environment (`TransportConnector::new(None)` -> `init_ssl_cert_env_vars`
/// -> `openssl_probe::init_openssl_env_vars` -> `std::env::set_var`).
#[test]
fn trigger_pingora_env_setenv_environ_uaf() {
    warn_if_libc_mitigates();

    // Make openssl-probe actually call setenv for the system CA bundle.
    std::env::remove_var("SSL_CERT_FILE");
    std::env::remove_var("SSL_CERT_DIR");
    pad_environment();
    let base = environ_addr();

    // Live allocations behind the environ block make an in-place grow impossible.
    let fragments: Vec<Vec<u8>> = (0..256).map(|i| vec![i as u8; 96]).collect();
    std::hint::black_box(&fragments);

    let shared = Arc::new(Shared {
        stop: AtomicBool::new(false),
        scans: AtomicUsize::new(0),
    });
    let readers = knob_usize("PINGORA_READERS", 8);
    let handles = spawn_all(&shared, base, readers);
    std::thread::sleep(Duration::from_millis(
        knob_usize("PINGORA_WARMUP_MS", 300) as u64
    ));

    println!(
        "[env-uaf] environ base={base:#x} entries={} readers={} scans={}",
        env_len(),
        readers,
        shared.scans.load(Ordering::Relaxed)
    );

    // === trigger: pingora-core public safe API ===
    let _connector = pingora_core::connectors::TransportConnector::new(None);
    println!(
        "[env-uaf] trigger done: SSL_CERT_FILE={:?} SSL_CERT_DIR={:?}",
        std::env::var_os("SSL_CERT_FILE"),
        std::env::var_os("SSL_CERT_DIR")
    );

    // Under ASan the process is expected to abort inside a scanner here, on the
    // environ block captured at `base` that setenv's realloc has freed.
    let deadline = Instant::now() + Duration::from_secs(knob_usize("PINGORA_SECONDS", 5) as u64);
    while Instant::now() < deadline {
        std::thread::sleep(Duration::from_millis(20));
    }

    stop_and_join(&shared, handles);

    let mutated = std::env::var_os("SSL_CERT_FILE").is_some()
        || std::env::var_os("SSL_CERT_DIR").is_some();
    if mutated {
        eprintln!(
            "[env-uaf] the trigger mutated the environment but no ASan report was seen \
             (captured array was not freed); rerun under ASan"
        );
    } else {
        eprintln!(
            "[env-uaf] openssl-probe found no system CA path on this host; \
             the trigger did not mutate the environment"
        );
    }
}

A note on the reader threads is in order. AddressSanitizer cannot observe the scan performed by libc's getenv, because the C library is not built with sanitizer instrumentation. The reproducer therefore repeats the same scan of the environ array from instrumented Rust code, while two additional threads call the real getenv, so that the production shape is covered as well. The array inflation performed by pad_environment merely positions the allocation on an AddressSanitizer size class boundary so that the next setenv must relocate the block; it fabricates no memory.

Expected results

Constructing an upstream TLS connector should invoke no undefined behavior, irrespective of how many threads are running and of which libraries those threads call. The connector should either configure its trust store through read-only discovery, or confine any environment mutation to an initialization phase that is documented and enforced as single-threaded. Under AddressSanitizer, both the control test and the trigger test should complete without a report.

Observed results

The control test passes, which shows that scanning a stable environ array is harmless in itself. The trigger test aborts the process with a heap-use-after-free report whose release stack begins at the setenv performed by pingora and whose use stack belongs to the concurrent reader. The report therefore ties the released memory to the public API call rather than to the harness.

running 2 tests
test control_environ_scan_without_mutation ... [env-uaf][control] 2641934 environ scans over a stable array; no report expected
ok
test trigger_pingora_env_setenv_environ_uaf ... [env-uaf] glibc 2.35
[env-uaf] environ base=0x... entries=511 readers=8 scans=530854
AddressSanitizer:DEADLYSIGNAL
AddressSanitizer:DEADLYSIGNAL
==...==ERROR: AddressSanitizer: heap-use-after-free on address 0x... at pc 0x...
READ of size 8 at 0x... thread T13
[env-uaf] trigger done: SSL_CERT_FILE=Some("/usr/lib/ssl/certs/ca-certificates.crt") SSL_CERT_DIR=Some("/usr/lib/ssl/certs")
    #0 env_setvar_environ_uaf::walk_environ    pingora-core/tests/env_setvar_environ_uaf.rs:107:30
    #1 env_setvar_environ_uaf::spawn_scanner::{closure#0}    pingora-core/tests/env_setvar_environ_uaf.rs:133:30
    #2 std::sys::backtrace::__rust_begin_short_backtrace::<...>
    ...
    #13 asan_thread_start(void*)    compiler-rt/lib/asan/asan_interceptors.cpp:246:28
    #14 start_thread    nptl/pthread_create.c:442:8
    #15 clone3    sysdeps/unix/sysv/linux/x86_64/clone3.S:81

0x... is located 1584 bytes inside of 4096-byte region [0x...,0x...)
freed by thread T12 here:
    #0 realloc    compiler-rt/lib/asan/asan_malloc_linux.cpp:81:3
    #1 __add_to_environ    glibc stdlib/setenv.c:156:31
    #2 std::sys::env::unix::setenv
    #3 std::sys::env::unix::setenv::{closure#0}
    #4 std::env::set_var
    #5 openssl_probe::try_init_openssl_env_vars::put    openssl-probe-0.1.6/src/lib.rs:136:17
    #6 openssl_probe::try_init_openssl_env_vars    openssl-probe-0.1.6/src/lib.rs:121:13
    #7 openssl_probe::init_openssl_env_vars    openssl-probe-0.1.6/src/lib.rs:82:5
    #8 <std::sync::once::Once>::call_once::<pingora_core::connectors::tls::boringssl_openssl::init_ssl_cert_env_vars::{closure#0}>
    #9 pingora_core::connectors::tls::boringssl_openssl::init_ssl_cert_env_vars    pingora-core/src/connectors/tls/boringssl_openssl/mod.rs:85:17
    #10 <pingora_core::connectors::TransportConnector>::new    pingora-core/src/connectors/mod.rs:205:22
    #11 env_setvar_environ_uaf::trigger_pingora_env_setenv_environ_uaf    pingora-core/tests/env_setvar_environ_uaf.rs:240:22
    ...

previously allocated by thread T12 here:
    #0 realloc    compiler-rt/lib/asan/asan_malloc_linux.cpp:81:3
    #1 __add_to_environ    glibc stdlib/setenv.c:156:31
    #2 std::sys::env::unix::setenv
    #3 std::env::set_var
    #4 env_setvar_environ_uaf::pad_environment    pingora-core/tests/env_setvar_environ_uaf.rs:181:9
    #5 env_setvar_environ_uaf::trigger_pingora_env_setenv_environ_uaf    pingora-core/tests/env_setvar_environ_uaf.rs:215:5
    ...

SUMMARY: AddressSanitizer: heap-use-after-free pingora-core/tests/env_setvar_environ_uaf.rs:107:30 in env_setvar_environ_uaf::walk_environ
Shadow bytes around the buggy address:
  0x...: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  ...
=>0x...: fd fd fd fd fd fd[fd]fd fd fd fd fd fd fd fd fd
==...==ABORTING
error: test failed, to rerun pass `-p pingora-core --test env_setvar_environ_uaf`

Several details in the report deserve emphasis. First, the release stack passes through __add_to_environ at stdlib/setenv.c:156, which is the realloc call quoted above, and it reaches that call from init_ssl_cert_env_vars at pingora-core/src/connectors/tls/boringssl_openssl/mod.rs:85, that is, from the Once closure that pingora itself supplies. Second, the use stack belongs to walk_environ, a reader that was started before the connector was constructed. Third, the released region is exactly 4096 bytes, which matches the size class boundary prepared by pad_environment, and the allocation stack confirms that the block was allocated by the environment inflation rather than by the harness in some unrelated way. Fourth, the two DEADLYSIGNAL lines record a signal that was intercepted by the sanitizer runtime shortly before the report was printed, most plausibly from one of the uninstrumented getenv readers; the definitive diagnosis remains the heap-use-after-free report.

Additional context

The same failure mode has an upstream precedent. openssl-probe issue #30, titled "Environment setters are unsound", records a production crash in which one thread called init_ssl_cert_env_vars indirectly through a first reqwest access, while another thread ran Python asyncio and reached getenv("LANGUAGE") from gettext; the setenv performed by the first thread triggered a realloc that moved environ, and the second thread read an invalid pointer and terminated the process. The maintainer of openssl-probe responded by deprecating the safe entry points and introducing unsafe equivalents in 0.1.6, with documentation that explicitly recommends probe() followed by direct OpenSSL configuration. A RustSec advisory was drafted but later abandoned; its checklist names pingora-core among the projects that remain unfixed.

For completeness, glibc 2.41 and later contain a rewritten environment implementation that retains previous environ arrays on a list and protects getenv with a counter, so the sanitizer-visible failure narrows to glibc 2.40 and earlier, musl, and other platforms. That mitigation does not remove the defect at the level of the API contract, and it does not remove the crash on the distributions that remain in wide use; the library still performs an operation that Rust documents as unsound, and it still does so at an arbitrary point in the process lifetime. Correcting the call site therefore remains worthwhile regardless of the C library version.

No existing issue in this repository refers to openssl-probe, init_openssl_env_vars, SSL_CERT_FILE, or set_var, and the code at pingora-core/src/connectors/tls/boringssl_openssl/mod.rs:79-85 has not changed since the dependency was upgraded to 0.1.6 in January 2025. The reproducer and the sanitizer output above should be sufficient to confirm the behaviour on any glibc 2.40 or earlier host; I can provide the complete unedited report and can test a candidate fix.