一句话总结当大模型 API 从「技术尝鲜」变成「生产基础设施」Java 后端必须有自己的 LLM 网关层——不是简单转发而是统一接入、Token 感知限流、智能降级、全链路审计与成本治理。一、为什么 Java 后端需要 LLM 网关2026 年的 AI 工程已经过了一张 API Key 调用到底的阶段。企业里真实的 LLM 调用链路往往是这样的客服系统调用 Azure OpenAI代码助手接 Claude内部 RAG 用通义千问营销文案又接了 DeepSeek每个业务团队各自维护 API Key、限流、重试、日志。结果就变成了「Key 泛滥、账单爆炸、故障互相影响、审计无从下手」。LLM 网关LLM Gateway / AI Gateway解决的就是这个问题。它不是 OpenAI 的 SDK 包装而是位于所有业务服务和模型提供商之间的一层统一治理平面负责能力说明Provider 抽象业务代码只认网关换模型不用改业务API Key 隔离真实 Key 只在网关存储业务侧用虚拟 KeyToken 感知限流按输入/输出 Token 计费限速而不是粗糙的 QPS智能降级OpenAI 挂了自动切 Anthropic长上下文模型降级到短上下文审计与合规记录谁、在什么时间、调用了什么模型、花了多少 Token成本控制租户级预算、模型路由到性价比最高的可用模型对 Java 团队来说Spring Cloud Gateway 是最自然的落地选择它本来就是微服务体系里的入口和 Spring AI、Spring Security、Micrometer、Resilience4j 都是一家人。本文就围绕一个完整的Multi-Tenant LLM Gateway项目从源码原理讲到生产代码。二、LLM 网关与传统 API 网关的差异传统 API 网关假设请求开销相近、后端无状态、响应一次性返回。但 LLM 调用完全不是这个模型请求成本差异巨大一个 128K 上下文的请求和一个 1K 上下文的请求GPU 消耗可能差两个数量级流式响应SSE/Chunked 响应需要网关具备长连接透传能力不能等响应体完整再转发Token 计费按请求数限流会误导容量规划必须按 Token 计费模型降级当 GPT-4 不可用时能否自动降到 GPT-3.5 或国产模型并保持响应格式一致上下文亲和性多轮对话最好打到同一推理实例以复用 KV Cache否则首 Token 延迟会飙升。这些差异决定了传统网关的「按 QPS 限流 轮询负载均衡」在 LLM 场景下会失效。LLM 网关需要引入Token 感知、模型路由、流式透传、语义缓存、成本审计等新的抽象。三、核心原理与源码解析3.1 Spring Cloud Gateway 请求生命周期Spring Cloud Gateway 基于 Project Reactor 和 Netty一个请求进入后依次经过RoutePredicateHandlerMapping - FilteringWebHandler - GlobalFilter - GatewayFilter - 下游服务关键源码在org.springframework.cloud.gateway.handler.FilteringWebHandlerpublic MonoVoid handle(ServerWebExchange exchange) { Route route exchange.getRequiredAttribute(GATEWAY_ROUTE_ATTR); ListGatewayFilter gatewayFilters route.getFilters(); ListGatewayFilter combined new ArrayList(this.globalFilters); combined.addAll(gatewayFilters); // 按 order 排序 AnnotationAwareOrderComparator.sort(combined); return new DefaultGatewayFilterChain(combined).filter(exchange); }这意味着我们可以在GlobalFilter层统一做认证、限流、审计在GatewayFilter层做路由转发、Provider 选择、Header 注入。3.2 RouteDefinition 与 PredicateRouteDefinition是路由的配置抽象核心字段public class RouteDefinition { private String id; private ListPredicateDefinition predicates new ArrayList(); private ListFilterDefinition filters new ArrayList(); private URI uri; private MapString, Object metadata new HashMap(); // ... }LLM 网关里我们会让Predicate按路径匹配例如/ai/openai/**转发到 OpenAI/ai/anthropic/**转发到 Claude。真正生产环境中更推荐用统一入口 Header 模型选择而不是硬编码路由。3.3 RedisRateLimiter 令牌桶实现Spring Cloud Gateway 内置的RequestRateLimiter基于 Redis Lua 脚本实现分布式令牌桶。核心 Lua 在org.springframework.cloud.gateway.filter.ratelimit.RedisRateLimiterlocal tokens_key KEYS[1] local timestamp_key KEYS[2] local rate tonumber(ARGV[1]) local capacity tonumber(ARGV[2]) local now tonumber(ARGV[3]) local requested tonumber(ARGV[4]) local fill_time capacity / rate local ttl math.floor(fill_time * 2) local last_tokens tonumber(redis.call(get, tokens_key)) if last_tokens nil then last_tokens capacity end local last_updated tonumber(redis.call(get, timestamp_key)) if last_updated nil then last_updated 0 end local delta math.max(0, now - last_updated) local filled_tokens math.min(capacity, last_tokens (delta * rate)) local allowed filled_tokens requested local new_tokens filled_tokens if allowed then new_tokens filled_tokens - requested end redis.call(setex, tokens_key, ttl, new_tokens) redis.call(setex, timestamp_key, ttl, now) return { allowed, new_tokens }关键认知默认requestedTokens 1也就是每个请求只消耗 1 个令牌。对于 LLM 网关这远远不够。我们需要自定义RateLimiter或GatewayFilter让请求消耗与预估 Token 成正比的令牌数。3.4 Resilience4j 熔断降级Spring Cloud Gateway 通过CircuitBreakerGatewayFilter 集成 Resilience4j。CircuitBreakerConfig核心参数failureRateThreshold失败率阈值默认 50%waitDurationInOpenState开启状态持续时间slidingWindowSize统计窗口大小permittedNumberOfCallsInHalfOpenState半开状态允许请求数。在 LLM 网关里熔断需要区分错误类型429/503provider 侧限流适合立即 fallback400/401请求或鉴权错误fallback 无意义应快速失败5xxprovider 内部错误应触发熔断并切换模型。四、项目实战构建 Multi-Tenant LLM Gateway4.1 项目结构llm-gateway/ ├── src/main/java/com/example/llmgateway/ │ ├── LlmGatewayApplication.java │ ├── config/ │ │ └── GatewayConfig.java │ ├── filter/ │ │ ├── ProviderRoutingFilter.java │ │ ├── TokenAwareRateLimitFilter.java │ │ ├── ApiKeyInjectionFilter.java │ │ └── AuditLoggingFilter.java │ ├── limiter/ │ │ └── TokenBucketRateLimiter.java │ ├── model/ │ │ ├── LlmProvider.java │ │ ├── LlmRequest.java │ │ └── LlmResponse.java │ └── util/ │ └── TokenEstimator.java ├── src/main/resources/ │ └── application.yml └── pom.xml4.2 Maven 依赖dependencies dependency groupIdorg.springframework.cloud/groupId artifactIdspring-cloud-starter-gateway/artifactId /dependency dependency groupIdorg.springframework.boot/groupId artifactIdspring-boot-starter-data-redis-reactive/artifactId /dependency dependency groupIdorg.springframework.cloud/groupId artifactIdspring-cloud-starter-circuitbreaker-reactor-resilience4j/artifactId /dependency dependency groupIdcom.knuddels/groupId artifactIdjtokkit/artifactId version1.1.0/version /dependency dependency groupIdio.micrometer/groupId artifactIdmicrometer-registry-prometheus/artifactId /dependency /dependencies4.3 配置文件spring: application: name: llm-gateway redis: host: localhost port: 6379 timeout: 800ms cloud: gateway: routes: - id: openai-proxy uri: https://api.openai.com predicates: - Path/ai/openai/** filters: - StripPrefix2 - name: CircuitBreaker args: name: openai-cb fallbackUri: forward:/fallback/openai - id: anthropic-proxy uri: https://api.anthropic.com predicates: - Path/ai/anthropic/** filters: - StripPrefix2 - name: CircuitBreaker args: name: anthropic-cb fallbackUri: forward:/fallback/anthropic default-filters: - DedupeResponseHeaderAccess-Control-Allow-Credentials Access-Control-Allow-Origin resilience4j: circuitbreaker: instances: openai-cb: slidingWindowSize: 100 failureRateThreshold: 50 waitDurationInOpenState: 30s permittedNumberOfCallsInHalfOpenState: 10 recordExceptions: - org.springframework.web.reactive.function.client.WebClientResponseException4.4 路由与 Provider 抽象统一入口让业务只调用/ai/chat/completions由 Header 里的X-AI-Provider决定具体走哪家Component Slf4j public class ProviderRoutingFilter implements GlobalFilter, Ordered { private static final String PROVIDER_HEADER X-AI-Provider; private static final String MODEL_HEADER X-AI-Model; private static final int ORDER -100; // 最早执行 private final MapString, ProviderConfig providers Map.of( openai, new ProviderConfig(https://api.openai.com, /v1/chat/completions, OPENAI_API_KEY), anthropic, new ProviderConfig(https://api.anthropic.com, /v1/messages, ANTHROPIC_API_KEY), deepseek, new ProviderConfig(https://api.deepseek.com, /v1/chat/completions, DEEPSEEK_API_KEY) ); Override public MonoVoid filter(ServerWebExchange exchange, GatewayFilterChain chain) { ServerHttpRequest request exchange.getRequest(); String provider request.getHeaders().getFirst(PROVIDER_HEADER); if (provider null || !providers.containsKey(provider)) { return errorResponse(exchange, HttpStatus.BAD_REQUEST, Missing or invalid X-AI-Provider); } ProviderConfig config providers.get(provider); String targetPath config.baseUrl() config.path(); ServerHttpRequest mutated request.mutate() .uri(URI.create(targetPath)) .header(Authorization, Bearer resolveApiKey(config.keyEnv())) .header(X-Resolved-Provider, provider) .build(); exchange.getAttributes().put(llm.provider, provider); exchange.getAttributes().put(llm.model, request.getHeaders().getFirst(MODEL_HEADER)); return chain.filter(exchange.mutate().request(mutated).build()); } private MonoVoid errorResponse(ServerWebExchange exchange, HttpStatus status, String msg) { exchange.getResponse().setStatusCode(status); byte[] bytes ({\error\:\ msg \}).getBytes(StandardCharsets.UTF_8); exchange.getResponse().getHeaders().setContentType(MediaType.APPLICATION_JSON); return exchange.getResponse().writeWith(Mono.just(bytes) .map(b - exchange.getResponse().bufferFactory().wrap(b))); } private String resolveApiKey(String envName) { String key System.getenv(envName); if (key null || key.isBlank()) { throw new IllegalStateException(Missing API key env: envName); } return key; } Override public int getOrder() { return ORDER; } record ProviderConfig(String baseUrl, String path, String keyEnv) {} }4.5 Token 感知限流器这是 LLM 网关与普通网关最大的区别。我们用jtokkit估算请求体里的 Token 数然后让 Redis 令牌桶按 Token 数扣减。Component public class TokenAwareRateLimitFilter implements GlobalFilter, Ordered { private static final int ORDER -50; private final ReactiveRedisTemplateString, String redisTemplate; private final Encoding encoding; public TokenAwareRateLimitFilter(ReactiveRedisTemplateString, String redisTemplate) { this.redisTemplate redisTemplate; this.encoding Encodings.newDefaultEncodingRegistry().getEncoding(EncodingType.CL100K_BASE); } Override public MonoVoid filter(ServerWebExchange exchange, GatewayFilterChain chain) { String tenantId exchange.getRequest().getHeaders().getFirst(X-Tenant-Id); if (tenantId null) tenantId anonymous; String key llm:tenant: tenantId :tokens; return exchange.getRequest().getBody() .collectList() .flatMap(buffers - { String body buffers.stream() .map(b - b.toString(StandardCharsets.UTF_8)) .collect(Collectors.joining()); int estimatedTokens estimateTokens(body); exchange.getAttributes().put(llm.estimatedTokens, estimatedTokens); return acquireTokens(key, estimatedTokens) .flatMap(allowed - { if (allowed) { // 重新把 body 写回流避免下游读不到 DataBuffer newBuffer exchange.getResponse() .bufferFactory().wrap(body.getBytes(StandardCharsets.UTF_8)); ServerHttpRequest mutated new ServerHttpRequestDecorator(exchange.getRequest()) { Override public FluxDataBuffer getBody() { return Flux.just(newBuffer); } }; return chain.filter(exchange.mutate().request(mutated).build()); } else { exchange.getResponse().setStatusCode(HttpStatus.TOO_MANY_REQUESTS); exchange.getResponse().getHeaders().set(Retry-After, 10); return exchange.getResponse().setComplete(); } }); }); } private int estimateTokens(String json) { // 生产级建议解析 messages 数组这里简化按文本内容估算 try { int charCount json.length(); // 中文场景1 token ≈ 0.75 中文字符保守按 0.6 return Math.max(1, (int) (charCount * 0.6)); } catch (Exception e) { return 1000; // 失败时给一个保守默认值避免误判通过 } } private MonoBoolean acquireTokens(String key, int tokens) { // 实际生产建议用 Lua 脚本保证原子性这里用 RedisTemplate 演示 String lua local key KEYS[1] local capacity tonumber(ARGV[1]) local rate tonumber(ARGV[2]) local requested tonumber(ARGV[3]) local now tonumber(ARGV[4]) local ttl math.floor(capacity / rate * 2) local current redis.call(get, key) local last current and tonumber(current) or capacity local allowed last requested local new allowed and (last - requested) or last redis.call(setex, key, ttl, new) return allowed and 1 or 0 ; RedisScriptLong script new DefaultRedisScript(lua, Long.class); return redisTemplate.execute(script, List.of(key), 100000, 1000, String.valueOf(tokens), String.valueOf(System.currentTimeMillis() / 1000)) .next() .map(r - r 1) .onErrorReturn(true); // Redis 故障时 fail-open避免网关雪崩 } Override public int getOrder() { return ORDER; } }说明上面用字符数估算 Token 是为了演示。生产环境应集成jtokkit按模型对应编码CL100K_BASE / O200K_BASE精确计算 messages 数组。4.6 审计与成本监控审计需要记录请求模型、租户、Token 数、耗时、是否命中缓存、是否降级。这里用GlobalFilter在响应完成后写日志 Micrometer 指标。Component Slf4j public class AuditLoggingFilter implements GlobalFilter, Ordered { private static final int ORDER Ordered.LOWEST_PRECEDENCE - 100; private final MeterRegistry meterRegistry; public AuditLoggingFilter(MeterRegistry meterRegistry) { this.meterRegistry meterRegistry; } Override public MonoVoid filter(ServerWebExchange exchange, GatewayFilterChain chain) { long start System.currentTimeMillis(); String tenantId exchange.getRequest().getHeaders().getFirst(X-Tenant-Id); String provider exchange.getAttributeOrDefault(llm.provider, unknown); String model exchange.getAttributeOrDefault(llm.model, unknown); Integer estimatedTokens exchange.getAttributeOrDefault(llm.estimatedTokens, 0); return chain.filter(exchange) .doFinally(sig - { long duration System.currentTimeMillis() - start; int status exchange.getResponse().getStatusCode() ! null ? exchange.getResponse().getStatusCode().value() : 0; log.info(LLM_AUDIT tenant{} provider{} model{} status{} estimatedTokens{} durationMs{}, tenantId, provider, model, status, estimatedTokens, duration); meterRegistry.counter(llm.requests.total, tenant, tenantId, provider, provider, model, model, status, String.valueOf(status)).increment(); meterRegistry.counter(llm.tokens.estimated, tenant, tenantId, provider, provider).increment(estimatedTokens); meterRegistry.timer(llm.request.duration, provider, provider, model, model).record(duration, TimeUnit.MILLISECONDS); }); } Override public int getOrder() { return ORDER; } }4.7 降级 Controller当 OpenAI 熔断打开时网关把请求forward:/fallback/openai由下面 Controller 返回统一降级响应RestController RequestMapping(/fallback) public class FallbackController { PostMapping(/{provider}) public MonoResponseEntityMapString, Object fallback( PathVariable String provider, RequestBody(required false) MapString, Object body) { MapString, Object resp new HashMap(); resp.put(provider, provider); resp.put(fallback, true); resp.put(message, Provider provider is currently unavailable. Please retry or switch to another model.); resp.put(suggested_models, List.of(deepseek-chat, qwen-turbo)); return Mono.just(ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE).body(resp)); } }五、生产踩坑清单1. 按请求数限流等于「没限流」很多团队直接把 Spring Cloud Gateway 的RequestRateLimiter配成 100 QPS结果一个长上下文请求就把后端打爆。LLM 场景必须做Token 感知限流建议按(输入 Token 预估输出 Token) / 分钟配置租户配额。2. Token 估算不准导致限流失效用字符数估算中文 Token 误差很大。生产环境务必用jtokkit或对应厂商的 tokenizer并按模型选择编码GPT-4 / GPT-3.5 →CL100K_BASEGPT-4o →O200K_BASEClaude → 需要调用 Anthropic 官方 tokenizer 或近似估算3. 流式响应被网关「憋大招」默认ModifyResponseBodyGatewayFilter会等完整响应再转发LLM 流式场景下会彻底破坏 SSE 体验。Gateway 转发流式响应时不要做任何需要消费完整 body 的 filter只读 Header 和元数据。4. Redis 限流器故障导致全站不可用Redis 挂掉时如果 fail-closed所有请求都会被 429。生产级限流必须fail-openRedis 异常时允许流量通过并告警而不是阻断。代码里onErrorReturn(true)就是这个意图。5. API Key 在日志里裸奔网关会打印下游请求 Header一定要把Authorization加入日志脱敏列表。同时建议用虚拟 Key 网关真实 Key 映射业务侧只持有虚拟 Key即使泄露也能随时吊销。6. X-Forwarded-For 被伪造导致限流绕过按 IP 限流时恶意客户端可以伪造X-Forwarded-For。必须只信任经过负载均衡追加的最后一跳 IP或者按X-API-Key限流。7. 模型降级时响应格式不一致OpenAI 和 Anthropic 的响应格式不同直接 fallback 会让上游解析失败。建议在网关层做响应格式归一化把 Anthropic 的content数组转成 OpenAI 的choices结构或者让业务统一按 OpenAI 兼容格式调用。8. 审计日志写磁盘导致网关 IO 被打满高频 LLM 调用下同步写文件会把网关拖垮。审计日志应走异步队列Kafka / MQ或采样记录例如只记录 10% 完整 prompt并用结构化日志JSON方便后续分析。9. 多租户配额互相影响如果没有按租户隔离 Redis Key一个租户刷流量会把全局桶打空。Key 设计必须是llm:tenant:{tenantId}:tokens并且为每个租户单独配置capacity和rate。10. 忽略 provider 侧 429 的 retry-afterOpenAI 返回 429 时通常带retry-after头网关应把这个头原样透传或主动根据它做指数退避而不是无脑重试。六、总结LLM 网关是 Java 后端把大模型能力「工程化」的关键一层。它不是可有可无的代理而是企业生产 AI 的入口Provider 抽象让业务与模型解耦Token 感知限流让容量规划贴合真实成本智能降级让单点故障不拖垮业务统一审计让成本、合规、安全可控。Spring Cloud Gateway Spring AI Redis Resilience4j 这套栈对已经熟悉 Spring 生态的 Java 团队来说几乎是零额外学习成本的落地路径。再一次印证了那个核心判断Java 工程师做 AI不需要转 Python工程化能力才是真正的护城河。完整示例代码已按模块化结构给出生产部署时建议再补上Nacos 动态路由、Sentinel 集群流控、Micrometer Prometheus Grafana 监控、Kafka 异步审计、以及 Provider 响应格式归一化中间件。网关这一层做厚了后面的 Agent、RAG、代码助手才能真正跑得稳。