397279d46e
## 🐛 Problem Users running commands with non-ASCII characters (like Russian text "пример") in Windows/WSL environments experience garbled text in VSCode's shell preview window, with Unicode replacement characters (�) appearing instead of the actual text. **Issue**: https://github.com/openai/codex/issues/6178 ## 🔧 Root Cause The issue was in `StreamOutput<Vec<u8>>::from_utf8_lossy()` method in `codex-rs/core/src/exec.rs`, which used `String::from_utf8_lossy()` to convert shell output bytes to strings. This function immediately replaces any invalid UTF-8 byte sequences with replacement characters, without attempting to decode using other common encodings. In Windows/WSL environments, shell output often uses encodings like: - Windows-1252 (common Windows encoding) - Latin-1/ISO-8859-1 (extended ASCII) ## 🛠️ Solution Replaced the simple `String::from_utf8_lossy()` call with intelligent encoding detection via a new `bytes_to_string_smart()` function that tries multiple encoding strategies: 1. **UTF-8** (fast path for valid UTF-8) 2. **Windows-1252** (handles Windows-specific characters in 0x80-0x9F range) 3. **Latin-1** (fallback for extended ASCII) 4. **Lossy UTF-8** (final fallback, same as before) ## 📁 Changes ### New Files - `codex-rs/core/src/text_encoding.rs` - Smart encoding detection module - `codex-rs/core/tests/suite/text_encoding_fix.rs` - Integration tests ### Modified Files - `codex-rs/core/src/lib.rs` - Added text_encoding module - `codex-rs/core/src/exec.rs` - Updated StreamOutput::from_utf8_lossy() - `codex-rs/core/tests/suite/mod.rs` - Registered new test module ## ✅ Testing - **5 unit tests** covering UTF-8, Windows-1252, Latin-1, and fallback scenarios - **2 integration tests** simulating the exact Issue #6178 scenario - **Demonstrates improvement** over the previous `String::from_utf8_lossy()` approach All tests pass: ```bash cargo test -p codex-core text_encoding cargo test -p codex-core test_shell_output_encoding_issue_6178 ``` ## 🎯 Impact - ✅ **Eliminates garbled text** in VSCode shell preview for non-ASCII content - ✅ **Supports Windows/WSL environments** with proper encoding detection - ✅ **Zero performance impact** for UTF-8 text (fast path) - ✅ **Backward compatible** - UTF-8 content works exactly as before - ✅ **Handles edge cases** with robust fallback mechanism ## 🧪 Test Scenarios The fix has been tested with: - Russian text ("пример") - Windows-1252 quotation marks (""test") - Latin-1 accented characters ("café") - Mixed encoding content - Invalid byte sequences (graceful fallback) ## 📋 Checklist - [X] Addresses the reported issue - [X] Includes comprehensive tests - [X] Maintains backward compatibility - [X] Follows project coding conventions - [X] No breaking changes --------- Co-authored-by: Josh McKinney <joshka@openai.com> |
||
|---|---|---|
| .. | ||
| .cargo | ||
| .config | ||
| ansi-escape | ||
| app-server | ||
| app-server-protocol | ||
| app-server-test-client | ||
| apply-patch | ||
| arg0 | ||
| async-utils | ||
| backend-client | ||
| chatgpt | ||
| cli | ||
| cloud-tasks | ||
| cloud-tasks-client | ||
| codex-backend-openapi-models | ||
| common | ||
| core | ||
| docs | ||
| exec | ||
| exec-server | ||
| execpolicy | ||
| execpolicy-legacy | ||
| feedback | ||
| file-search | ||
| keyring-store | ||
| linux-sandbox | ||
| lmstudio | ||
| login | ||
| mcp-server | ||
| mcp-types | ||
| ollama | ||
| otel | ||
| process-hardening | ||
| protocol | ||
| responses-api-proxy | ||
| rmcp-client | ||
| scripts | ||
| stdio-to-uds | ||
| tui | ||
| utils | ||
| windows-sandbox-rs | ||
| .gitignore | ||
| Cargo.lock | ||
| Cargo.toml | ||
| clippy.toml | ||
| code | ||
| config.md | ||
| default.nix | ||
| justfile | ||
| README.md | ||
| rust-toolchain.toml | ||
| rustfmt.toml | ||
Codex CLI (Rust Implementation)
We provide Codex CLI as a standalone, native executable to ensure a zero-dependency install.
Installing Codex
Today, the easiest way to install Codex is via npm:
npm i -g @openai/codex
codex
You can also install via Homebrew (brew install --cask codex) or download a platform-specific release directly from our GitHub Releases.
Documentation quickstart
- First run with Codex? Follow the walkthrough in
docs/getting-started.mdfor prompts, keyboard shortcuts, and session management. - Already shipping with Codex and want deeper control? Jump to
docs/advanced.mdand the configuration reference atdocs/config.md.
What's new in the Rust CLI
The Rust implementation is now the maintained Codex CLI and serves as the default experience. It includes a number of features that the legacy TypeScript CLI never supported.
Config
Codex supports a rich set of configuration options. Note that the Rust CLI uses config.toml instead of config.json. See docs/config.md for details.
Model Context Protocol Support
MCP client
Codex CLI functions as an MCP client that allows the Codex CLI and IDE extension to connect to MCP servers on startup. See the configuration documentation for details.
MCP server (experimental)
Codex can be launched as an MCP server by running codex mcp-server. This allows other MCP clients to use Codex as a tool for another agent.
Use the @modelcontextprotocol/inspector to try it out:
npx @modelcontextprotocol/inspector codex mcp-server
Use codex mcp to add/list/get/remove MCP server launchers defined in config.toml, and codex mcp-server to run the MCP server directly.
Notifications
You can enable notifications by configuring a script that is run whenever the agent finishes a turn. The notify documentation includes a detailed example that explains how to get desktop notifications via terminal-notifier on macOS.
codex exec to run Codex programmatically/non-interactively
To run Codex non-interactively, run codex exec PROMPT (you can also pass the prompt via stdin) and Codex will work on your task until it decides that it is done and exits. Output is printed to the terminal directly. You can set the RUST_LOG environment variable to see more about what's going on.
Experimenting with the Codex Sandbox
To test to see what happens when a command is run under the sandbox provided by Codex, we provide the following subcommands in Codex CLI:
# macOS
codex sandbox macos [--full-auto] [--log-denials] [COMMAND]...
# Linux
codex sandbox linux [--full-auto] [COMMAND]...
# Windows
codex sandbox windows [--full-auto] [COMMAND]...
# Legacy aliases
codex debug seatbelt [--full-auto] [--log-denials] [COMMAND]...
codex debug landlock [--full-auto] [COMMAND]...
Selecting a sandbox policy via --sandbox
The Rust CLI exposes a dedicated --sandbox (-s) flag that lets you pick the sandbox policy without having to reach for the generic -c/--config option:
# Run Codex with the default, read-only sandbox
codex --sandbox read-only
# Allow the agent to write within the current workspace while still blocking network access
codex --sandbox workspace-write
# Danger! Disable sandboxing entirely (only do this if you are already running in a container or other isolated env)
codex --sandbox danger-full-access
The same setting can be persisted in ~/.codex/config.toml via the top-level sandbox_mode = "MODE" key, e.g. sandbox_mode = "workspace-write".
Code Organization
This folder is the root of a Cargo workspace. It contains quite a bit of experimental code, but here are the key crates:
core/contains the business logic for Codex. Ultimately, we hope this to be a library crate that is generally useful for building other Rust/native applications that use Codex.exec/"headless" CLI for use in automation.tui/CLI that launches a fullscreen TUI built with Ratatui.cli/CLI multitool that provides the aforementioned CLIs via subcommands.