Commit graph

12 commits

Author SHA1 Message Date
Owen Lin
014e19510d
feat(app-server, core): add more spans (#14479)
## Description

This PR expands tracing coverage across app-server thread startup, core
session initialization, and the Responses transport layer. It also gives
core dispatch spans stable operation-specific names so traces are easier
to follow than the old generic `submission_dispatch` spans.

Also use `fmt::Display` for types that we serialize in traces so we send
strings instead of rust types
2026-03-13 13:16:33 -07:00
Josh McKinney
6912da84a8
client: extend custom CA handling across HTTPS and websocket clients (#14239)
## Stacked PRs

This work is now effectively split across two steps:

- #14178: add custom CA support for browser and device-code login flows,
docs, and hermetic subprocess tests
- #14239: extend that shared custom CA handling across Codex HTTPS
clients and secure websocket TLS

Note: #14240 was merged into this branch while it was stacked on top of
this PR. This PR now subsumes that websocket follow-up and should be
treated as the combined change.

Builds on top of #14178.

## Problem

Custom CA support landed first in the login path, but the real
requirement is broader. Codex constructs outbound TLS clients in
multiple places, and both HTTPS and secure websocket paths can fail
behind enterprise TLS interception if they do not honor
`CODEX_CA_CERTIFICATE` or `SSL_CERT_FILE` consistently.

This PR broadens the shared custom-CA logic beyond login and applies the
same policy to websocket TLS, so the enterprise-proxy story is no longer
split between “HTTPS works” and “websockets still fail”.

## What This Delivers

Custom CA support is no longer limited to login. Codex outbound HTTPS
clients and secure websocket connections can now honor the same
`CODEX_CA_CERTIFICATE` / `SSL_CERT_FILE` configuration, so enterprise
proxy/intercept setups work more consistently end-to-end.

For users and operators, nothing new needs to be configured beyond the
same CA env vars introduced in #14178. The change is that more of Codex
now respects them, including websocket-backed flows that were previously
still using default trust roots.

I also manually validated the proxy path locally with mitmproxy using:
`CODEX_CA_CERTIFICATE=~/.mitmproxy/mitmproxy-ca-cert.pem
HTTPS_PROXY=http://127.0.0.1:8080 just codex`
with mitmproxy installed via `brew install mitmproxy` and configured as
the macOS system proxy.

## Mental model

`codex-client` is now the owner of shared custom-CA policy for outbound
TLS client construction. Reqwest callers start from the builder
configuration they already need, then pass that builder through
`build_reqwest_client_with_custom_ca(...)`. Websocket callers ask the
same module for a rustls client config when a custom CA bundle is
configured.

The env precedence is the same everywhere:
- `CODEX_CA_CERTIFICATE` wins
- otherwise fall back to `SSL_CERT_FILE`
- otherwise use system roots

The helper is intentionally narrow. It loads every usable certificate
from the configured PEM bundle into the appropriate root store and
returns either a configured transport or a typed error that explains
what went wrong.

## Non-goals

This does not add handshake-level integration tests against a live TLS
endpoint. It does not validate that the configured bundle forms a
meaningful certificate chain. It also does not try to force every
transport in the repo through one abstraction; it extends the shared CA
policy across the reqwest and websocket paths that actually needed it.

## Tradeoffs

The main tradeoff is centralizing CA behavior in `codex-client` while
still leaving adoption up to call sites. That keeps the implementation
additive and reviewable, but it means the rule "outbound Codex TLS that
should honor enterprise roots must use the shared helper" is still
partly enforced socially rather than by types.

For websockets, the shared helper only builds an explicit rustls config
when a custom CA bundle is configured. When no override env var is set,
websocket callers still use their ordinary default connector path.

## Architecture

`codex-client::custom_ca` now owns CA bundle selection, PEM
normalization, mixed-section parsing, certificate extraction, typed
CA-loading errors, and optional rustls client-config construction for
websocket TLS.

The affected consumers now call into that shared helper directly rather
than carrying login-local CA behavior:
- backend-client
- cloud-tasks
- RMCP client paths that use `reqwest`
- TUI voice HTTP paths
- `codex-core` default reqwest client construction
- `codex-api` websocket clients for both responses and realtime
websocket connections

The subprocess CA probe, env-sensitive integration tests, and shared PEM
fixtures also live in `codex-client`, which is now the actual owner of
the behavior they exercise.

## Observability

The shared CA path logs:
- which environment variable selected the bundle
- which path was loaded
- how many certificates were accepted
- when `TRUSTED CERTIFICATE` labels were normalized
- when CRLs were ignored
- where client construction failed

Returned errors remain user-facing and include the relevant env var,
path, and remediation hint. That same error model now applies whether
the failure surfaced while building a reqwest client or websocket TLS
configuration.

## Tests

Pure unit tests in `codex-client` cover env precedence and PEM
normalization behavior. Real client construction remains in subprocess
tests so the suite can control process env and avoid the macOS seatbelt
panic path that motivated the hermetic test split.

The subprocess coverage verifies:
- `CODEX_CA_CERTIFICATE` precedence over `SSL_CERT_FILE`
- fallback to `SSL_CERT_FILE`
- single-cert and multi-cert bundles
- malformed and empty-file errors
- OpenSSL `TRUSTED CERTIFICATE` handling
- CRL tolerance for well-formed CRL sections

The websocket side is covered by the existing `codex-api` / `codex-core`
websocket test suites plus the manual mitmproxy validation above.

---------

Co-authored-by: Ivan Zakharchanka <3axap4eHko@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
2026-03-13 00:59:26 +00:00
pakrym-oai
e958d0337e
Log headers in trace mode (#9214)
To enable:

```
export RUST_LOG="warn,codex_=trace"
```

Sample: 
```
Request completed method=POST url=https://chatgpt.com/backend-api/codex/responses status=200 OK headers={"date": "Wed, 14 Jan 2026 18:21:21 GMT", "transfer-encoding": "chunked", "connection": "keep-alive", "x-codex-plan-type": "business", "x-codex-primary-used-percent": "3", "x-codex-secondary-used-percent": "6", "x-codex-primary-window-minutes": "300", "x-codex-primary-over-secondary-limit-percent": "0", "x-codex-secondary-window-minutes": "10080", "x-codex-primary-reset-after-seconds": "9944", "x-codex-secondary-reset-after-seconds": "171121", "x-codex-primary-reset-at": "1768424824", "x-codex-secondary-reset-at": "1768586001", "x-codex-credits-has-credits": "False", "x-codex-credits-balance": "", "x-codex-credits-unlimited": "False", "x-models-etag": "W/\"7a7ffbc83c159dbd7a2a73aaa9c91b7a\"", "x-oai-request-id": "ffedcd30-6d8a-4c4d-be10-8ebb23c142c8", "x-envoy-upstream-service-time": "417", "x-openai-proxy-wasm": "v0.1", "cf-cache-status": "DYNAMIC", "set-cookie": "__cf_bm=xFKeaMbWNbKO5ZX.K5cJBhj34OA1QvnF_3nkdMThjlA-1768414881-1.0.1.1-uLpsE_BDkUfcmOMaeKVQmv_6_2ytnh_R3lO_il5N5K3YPQEkBo0cOMTdma6bK0Gz.hQYcIesFwKIJht1kZ9JKqAYYnjgB96hF4.sii2U3cE; path=/; expires=Wed, 14-Jan-26 18:51:21 GMT; domain=.chatgpt.com; HttpOnly; Secure; SameSite=None", "report-to": "{\"endpoints\":[{\"url\":\"https:\/\/a.nel.cloudflare.com\/report\/v4?s=4Kc7g4zUhKkIm3xHuB6ba4jyIUqqZ07ETwIPAYQASikRjA8JesbtUKDP9tSrZ5PnzWldaiSz5dZVQFI579LEsCMlMUSelTvmyQ8j4FbFDawi%2FprWZ5iRePiaSalr\"}],\"group\":\"cf-nel\",\"max_age\":604800}", "nel": "{\"success_fraction\":0.01,\"report_to\":\"cf-nel\",\"max_age\":604800}", "strict-transport-security": "max-age=31536000; includeSubDomains; preload", "x-content-type-options": "nosniff", "cross-origin-opener-policy": "same-origin-allow-popups", "referrer-policy": "strict-origin-when-cross-origin", "server": "cloudflare", "cf-ray": "9bdf270adc7aba3a-SEA"} version=HTTP/1.1
```
2026-01-14 18:38:12 +00:00
gt-oai
cc0b5e8504
Add URL to responses error messages (#8984)
Put the URL in error messages, to aid debugging Codex pointing at wrong
endpoints.

<img width="759" height="164" alt="Screenshot 2026-01-09 at 16 32 49"
src="https://github.com/user-attachments/assets/77a0622c-955d-426d-86bb-c035210a4ecc"
/>
2026-01-10 00:53:47 +00:00
zbarsky-openai
2a06d64bc9
feat: add support for building with Bazel (#8875)
This PR configures Codex CLI so it can be built with
[Bazel](https://bazel.build) in addition to Cargo. The `.bazelrc`
includes configuration so that remote builds can be done using
[BuildBuddy](https://www.buildbuddy.io).

If you are familiar with Bazel, things should work as you expect, e.g.,
run `bazel test //... --keep-going` to run all the tests in the repo,
but we have also added some new aliases in the `justfile` for
convenience:

- `just bazel-test` to run tests locally
- `just bazel-remote-test` to run tests remotely (currently, the remote
build is for x86_64 Linux regardless of your host platform). Note we are
currently seeing the following test failures in the remote build, so we
still need to figure out what is happening here:

```
failures:
    suite::compact::manual_compact_twice_preserves_latest_user_messages
    suite::compact_resume_fork::compact_resume_after_second_compaction_preserves_history
    suite::compact_resume_fork::compact_resume_and_fork_preserve_model_history_view
```

- `just build-for-release` to build release binaries for all
platforms/architectures remotely

To setup remote execution:
- [Create a buildbuddy account](https://app.buildbuddy.io/) (OpenAI
employees should also request org access at
https://openai.buildbuddy.io/join/ with their `@openai.com` email
address.)
- [Copy your API key](https://app.buildbuddy.io/docs/setup/) to
`~/.bazelrc` (add the line `build
--remote_header=x-buildbuddy-api-key=YOUR_KEY`)
- Use `--config=remote` in your `bazel` invocations (or add `common
--config=remote` to your `~/.bazelrc`, or use the `just` commands)

## CI

In terms of CI, this PR introduces `.github/workflows/bazel.yml`, which
uses Bazel to run the tests _locally_ on Mac and Linux GitHub runners
(we are working on supporting Windows, but that is not ready yet). Note
that the failures we are seeing in `just bazel-remote-test` do not occur
on these GitHub CI jobs, so everything in `.github/workflows/bazel.yml`
is green right now.

The `bazel.yml` uses extra config in `.github/workflows/ci.bazelrc` so
that macOS CI jobs build _remotely_ on Linux hosts (using the
`docker://docker.io/mbolin491/codex-bazel` Docker image declared in the
root `BUILD.bazel`) using cross-compilation to build the macOS
artifacts. Then these artifacts are downloaded locally to GitHub's macOS
runner so the tests can be executed natively. This is the relevant
config that enables this:

```
common:macos --config=remote
common:macos --strategy=remote
common:macos --strategy=TestRunner=darwin-sandbox,local
```

Because of the remote caching benefits we get from BuildBuddy, these new
CI jobs can be extremely fast! For example, consider these two jobs that
ran all the tests on Linux x86_64:

- Bazel 1m37s
https://github.com/openai/codex/actions/runs/20861063212/job/59940545209?pr=8875
- Cargo 9m20s
https://github.com/openai/codex/actions/runs/20861063192/job/59940559592?pr=8875

For now, we will continue to run both the Bazel and Cargo jobs for PRs,
but once we add support for Windows and running Clippy, we should be
able to cutover to using Bazel exclusively for PRs, which should still
speed things up considerably. We will probably continue to run the Cargo
jobs post-merge for commits that land on `main` as a sanity check.

Release builds will also continue to be done by Cargo for now.

Earlier attempt at this PR: https://github.com/openai/codex/pull/8832
Earlier attempt to add support for Buck2, now abandoned:
https://github.com/openai/codex/pull/8504

---------

Co-authored-by: David Zbarsky <dzbarsky@gmail.com>
Co-authored-by: Michael Bolin <mbolin@openai.com>
2026-01-09 11:09:43 -08:00
Channing Conger
21c6d40a44
Add feature for optional request compression (#8767)
Adds a new feature
`enable_request_compression` that will compress using zstd requests to
the codex-backend. Currently only enabled for codex-backend so only enabled for openai providers when using chatgpt::auth even when the feature is enabled

Added a new info log line too for evaluating the compression ratio and
overhead off compressing before requesting. You can enable with
`RUST_LOG=$RUST_LOG,codex_client::transport=info`

```
2026-01-06T00:09:48.272113Z  INFO codex_client::transport: Compressed request body with zstd pre_compression_bytes=28914 post_compression_bytes=11485 compression_duration_ms=0
```
2026-01-07 13:21:40 -08:00
pakrym-oai
7078a0b676
Log compaction request bodies (#8676)
We already log request bodies for normal requests, logging for
compaction helps with debugging.
2026-01-02 11:27:37 -08:00
jif-oai
d7482510b1
nit: trace span for regular task (#8053)
Logs are too spammy

---------

Co-authored-by: Anton Panasenko <apanasenko@openai.com>
2025-12-16 16:53:15 +00:00
Anton Panasenko
ad7b9d63c3
[codex] add otel tracing (#7844) 2025-12-12 17:07:17 -08:00
pakrym-oai
4b684c53ae
Remove conversation_id and bring back request ID logging (#7830) 2025-12-10 10:44:12 -08:00
pakrym-oai
8b1e397211
Add request logging back (#7471)
Having full requests helps debugging
2025-12-02 07:57:55 -08:00
jif-oai
4502b1b263
chore: proper client extraction (#6996) 2025-11-25 18:06:12 +00:00