Build yourself an AI gateway in nginx

nginz-token ships as prebuilt Docker images to Pro and Enterprise subscribers. That is the product. But the source is available under the Business Source License 1.1 — anyone can pull the repo, build it, and try it out. If you like what you see, Pro is self-serve: subscribe and get the prebuilt images, updates, and support.

This post is the build tutorial. One repo. Three commands. A working AI gateway.

What you are building

Eight native nginx modules, compiled from Zig into position-independent object files, linked into nginx at build time:

If you have run an nginx reverse proxy before, the config surface will feel familiar. No sidecars, no control planes, no YAML DSLs. Each module registers a handful of directives that compose inside location blocks. The gateway is nginx.

Prerequisites

A Linux machine (amd64 or aarch64) with:

• Zig 0.16.0 — the exact version. The build system checks at comptime and refuses anything else. Download from ziglang.org and put zig on your $PATH.
• C build tools and libraries: make, gcc, openssl, pcre2, zlib, libpq, libxml2, libxslt.
• git — needed to clone the repo in Step 1.
• nginx source is vendored as a git submodule — you do not need to download it separately.
• curl — used in Step 4 to send the test request.
• jq (optional) — used in Step 4 to pretty-print the JSON response.

Debian/Ubuntu:

apt install -y make build-essential libssl-dev libpcre2-dev \
  zlib1g-dev libpq-dev libxml2-dev libxslt-dev git curl xz-utils jq

Alpine:

apk add make build-base openssl-dev pcre2-dev zlib-dev \
  libpq-dev libxml2-dev libxslt-dev git curl xz jq

Arch:

pacman -S make gcc openssl pcre2 zlib postgresql-libs \
  libxml2 libxslt git curl xz jq

Step 1: Clone and build the modules

git clone https://github.com/darkanchor/nginz-token.git
cd nginz-token
git submodule update --init
zig build -Doptimize=ReleaseSmall package

That is it. git submodule update --init pulls the vendored nginx source (plus njs and QuickJS, which you can ignore — they are not needed for the AI gateway). zig build -Doptimize=ReleaseSmall package compiles each module into a position-independent .o file and generates an nginx config stub for ./configure --add-module. The output lands in zig-out/modules/:

zig-out/modules/
├── llm-proxy/
│   ├── config
│   ├── ngx_http_llm_proxy_module.o
│   └── libcjson.a
├── llm-auth/
│   ├── config
│   └── ngx_http_llm_auth_module.o
├── llm-metrics/
│   ├── config
│   └── ngx_http_llm_metrics_module.o
├── llm-fallback/
│   ├── config
│   └── ngx_http_llm_fallback_module.o
├── llm-ratelimit/
│   ├── config
│   └── ngx_http_llm_ratelimit_module.o
├── llm-cost/
│   ├── config
│   └── ngx_http_llm_cost_module.o
├── llm-cache/
│   ├── config
│   └── ngx_http_llm_cache_module.o
└── llm-security/
    ├── config
    └── ngx_http_llm_security_module.o

Each config file is a short shell fragment that tells nginx’s build system which module name to register and where the precompiled object lives. The entire zig-out/modules/ tree is a few megabytes.

Step 2: Configure and build nginx

Point nginx’s ./configure at the eight module directories. The nginx source lives in submodules/nginx — that is what the submodule init pulled:

cd submodules/nginx

./auto/configure \
    --with-compat \
    --with-http_ssl_module \
    --with-http_v2_module \
    --with-http_realip_module \
    --with-http_auth_request_module \
    --with-http_stub_status_module \
    --add-module=../../../nginz-token/zig-out/modules/llm-proxy \
    --add-module=../../../nginz-token/zig-out/modules/llm-auth \
    --add-module=../../../nginz-token/zig-out/modules/llm-metrics \
    --add-module=../../../nginz-token/zig-out/modules/llm-fallback \
    --add-module=../../../nginz-token/zig-out/modules/llm-ratelimit \
    --add-module=../../../nginz-token/zig-out/modules/llm-cost \
    --add-module=../../../nginz-token/zig-out/modules/llm-cache \
    --add-module=../../../nginz-token/zig-out/modules/llm-security

make -j$(nproc)

The --add-module paths jump from submodules/nginx back to the repo root’s zig-out/modules/. The --with-compat flag is required — it ensures nginx’s struct layout matches what the Zig modules expect at link time. SSL, HTTP/2, realip, auth_request, and stub_status cover the basics. Add whatever else your deployment needs.

Verify:

objs/nginx -V

You should see all eight --add-module=.../llm-* entries. The modules are compiled in.

Step 3: Write a config and start the gateway

daemon off;
error_log /dev/stderr info;

env OPENAI_API_KEY;

events {
    worker_connections 64;
}

http {
    variables_hash_max_size 2048;
    variables_hash_bucket_size 128;

    upstream openai_api {
        server api.openai.com:443;
    }

    server {
        listen 8080;

        location /v1/chat/completions {
            llm_proxy;
            llm_proxy_route openai openai_api;
            llm_proxy_default_provider openai;

            llm_auth;
            llm_auth_credential openai env:OPENAI_API_KEY;

            proxy_ssl_server_name on;
            proxy_ssl_name api.openai.com;
            proxy_set_header Host api.openai.com;
            proxy_pass https://openai_api;
        }
    }
}

Nine directives. llm_proxy parses the request body, identifies it as OpenAI dialect, selects the openai route. llm_auth reads $OPENAI_API_KEY from the environment and injects Authorization: Bearer <key> into the proxied request. proxy_ssl_name and proxy_set_header Host matter because proxy_pass targets a named upstream (openai_api): without them, nginx sends SNI/Host as the upstream name, and the TLS handshake to OpenAI fails. proxy_pass sends it upstream. On the response path, llm_proxy extracts usage tokens and populates $llm_prompt_tokens, $llm_completion_tokens, and $llm_total_tokens — available for access logging.

Start it (save the config as nginx.conf in your current directory, then point nginx at it with the full path):

mkdir -p /tmp/nginx-test
cp nginx.conf /tmp/nginx-test/
OPENAI_API_KEY=sk-... objs/nginx -p /tmp/nginx-test -c /tmp/nginx-test/nginx.conf

Step 4: Send a request

curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4.1-mini",
    "messages": [{"role": "user", "content": "hello"}]
  }' | jq .

If the key is valid, you get a normal OpenAI chat completion response — same JSON shape as hitting api.openai.com directly. Even with a dummy key the gateway works: you will get a 401 Unauthorized from OpenAI, which proves the request was forwarded through the gateway modules.

Point any OpenAI-compatible SDK at http://localhost:8080/v1 as its API base URL, or configure your agent to hit http://localhost:8080/v1/chat/completions directly — the gateway handles credential injection, so your tools never need the real key.

What the other six modules give you

That short config works. The other six modules are additive — each a few more directives in a location block:

• llm-auth — client → project → org credential cascade with fail-closed guards. Not stuck with one API key.
• llm-ratelimit — token-per-minute and request-per-minute budgets in shared memory, enforced before the upstream call leaves the building.
• llm-fallback — retry on 5xx or connect error, with automatic dialect translation when the fallback provider speaks a different protocol.
• llm-metrics — Prometheus export: provider counters, latency histogram, usage accounting.
• llm-cost — per-request cost attribution to client/project/org, with optional PostgreSQL persistence.
• llm-cache — cache eligibility and isolation rules. Explicit policy, not semantic replay magic.
• llm-security — PII, secrets, and prompt injection scanning at the edge, before prompts leave your infrastructure.

Full directive reference for every module lives in /docs/reference/token-modules/ — each module has its own page with design rationale and every directive documented.

What you do not get yet

This is a source build. It is not the product — the license allows you to evaluate, learn, and tinker, but commercial production use requires a subscription. The Docker images ship with more: the full nginz infrastructure layer (24 additional modules — JWT, OIDC, WAF, circuit breaking, dynamic upstreams, and more), njs scripting, entrypoint scripts that tune worker processes, envsubst templating, and pre-tested module compatibility across pinned revisions. Pro and Enterprise subscribers get private-registry images and support.

The build process above is intentionally minimal. It compiles the AI gateway modules into nginx and nothing else. If you later want the full stack, the Docker images use the same zig build package → ./configure --add-module → make pipeline — your nginx.conf does not change.

Why inside nginx

Some AI gateways run as separate services. Some run as SaaS proxies. Some embed a LuaJIT runtime.

This one is compiled to native code. It runs in the same process as nginx — same event loop, same shared memory, same connection pool. The added latency is microseconds of JSON parsing and in-process bookkeeping, not a network hop to an external service. nginx itself has been battle-tested for two decades; it is performant and stable as a rock. And because the gateway is nginx, everything you already know about operating nginx still applies: same reload signal, same log format, same proxy_* directives.