184 points • 3 days ago • Article Link

尝试用本地 large language model 替代编码工具框架中对远程 API 的调用，往往会令人很沮丧。虽然像 llama-bench 这样的基准测试可能显示很高的 tokens-per-second，但真实的使用体验常常被明显的延迟、卡顿和漫长的等待时间破坏。这种差距主要因为大多数编码工具框架在设计时假定会运行在高速的数据中心 GPU 上，而非受限的消费级笔记本上。

本地硬件的物理特性会把系统逼到三种典型的设计困境上。第一，巨大的系统提示（system prompt）和复杂的工具 schema 带来沉重的预填充负担。当笔记本处理 token 的速度只是数据中心的一小部分时，一个 18,000 token 的提示可能导致模型在开始生成代码前就要等待几分钟。第二，这类大提示会占用大量上下文窗口，几乎没剩多少空间用于实际任务。第三，许多框架会不断发起会话标题或摘要的侧请求，在本地运行时这些请求会把模型推入频繁且重叠的队列，导致重复的长时间预填充，使得性能降到不可用的程度。

为了解这些瓶颈，在一台高端 M4 MacBook Pro 上，使用量化的 Qwen 3.8 27B 模型对九个流行的编码框架进行了多项 Python 练习测试。结果把这些工具分为三类：精简且稳定的框架保持短小的系统提示并高度重用缓存，是本地使用的最佳候选；尽管提示较长或工具 schema 较多但有纪律的重型工具，在完成初次预填充后仍能正常工作；而启动沉重型的工具可能需要三到四分钟的静默等待，代理才能迈出第一步，因而不适合本地工作流。

问题的核心不是工程水平低下，而是这些工具最初的设计环境不同。当延迟以毫秒计时，几乎免费的特性在本地硅片上以秒或分钟计时就成了沉重的负担。因此，像 chad 这样的项目应运而生，采取稀缺性思维，限制工具表面并针对持久化的前缀缓存（prefix caching）进行优化。这类设计把效率放在首位，确保开发循环保持响应，而不是被模型的开销阻塞。

真正的性能突破发生在框架与推理引擎运行在同一进程时。通过打破传统的客户端 - 服务器模型，将引擎内嵌到进程中（例如通过 MLX），可以将与请求模式相关的延迟降到最低。那些拥有自身缓存并采用专门草稿生成技术的框架，能够实现显著更高的 tokens-per-second，把曾经缓慢、令人沮丧的体验变成流畅高效的编码环境。 Attempting to use a local large language model as a replacement for remote API calls in coding harnesses often leads to frustration. While benchmarking tools like llama-bench might suggest high tokens-per-second, the actual user experience is frequently marred by significant latency, stalling, and long wait times. This discrepancy exists primarily because most coding harnesses were built with the assumption of high-speed data center GPUs, rather than the constraints of a consumer laptop.

The physics of local hardware often pits the system against three specific design choices. First, large system prompts and extensive tool schemas create a massive prefill burden. When a laptop processes tokens at a fraction of the speed of a data center, a 18,000-token prompt can result in several minutes of waiting before the model even begins to generate code. Second, these large prompts consume a significant portion of the available context window, leaving little room for actual task execution. Third, many harnesses are designed to make constant side requests for session titles or summaries, which, when run locally, force the model into frequent, overlapping queues and repeated long prefills that degrade performance to the point of being unusable.

To better understand these bottlenecks, nine popular coding harnesses were tested against a series of Python exercises on a high-end M4 MacBook Pro using a quantized Qwen 3.8 27B model. The results categorized these tools into three distinct groups. Lean and stable harnesses maintain trim system prompts and high cache reuse, making them the most viable candidates for local use. Heavy but disciplined tools, while burdened by long prompts or excessive tool schemas, remain functional once the initial prefill is complete. Finally, heavy-to-start tools can require three to four minutes of inactivity before the agent takes its first step, rendering them impractical for a local workflow.

The core issue is not poor engineering, but rather the environment for which these tools were originally designed. Features that are virtually free when latency is measured in milliseconds become heavy liabilities when measured in seconds or minutes on local silicon. Consequently, projects like chad have emerged to embrace a scarcity mindset, limiting tool surfaces and optimizing for persistent prefix caching. These designs prioritize efficiency to ensure that the development loop remains responsive rather than blocked by the model's overhead.

The true breakthrough in performance, however, arises when the harness and the inference engine share the same process. By breaking away from the standard client-server model and moving the engine in-process, such as through MLX, the latency associated with request patterns is minimized. Harnesses that own their cache and utilize specialized drafting techniques can achieve significantly higher token-per-second rates, turning what was once a sluggish, frustrating experience into a fluid and efficient coding environment.