Cloud and local models. One API for all of them.

TokenHub speaks the OpenAI API, so it connects to the major cloud providers and to local inference servers like Ollama and vLLM. Switching models is a one-line change to model; routing, failover, metering and quotas all happen in the gateway.

Free Download Read the docs

15+cloud model providers
7local inference frameworks
1unified API

Runs on your own servers: single-node Docker, a high-availability cluster, or an air-gapped network. Cloud models use your own provider API keys.

localhost:8080 / Model channels

ChannelTypeStatus

OAOpenAIPrimary chat · weight 30CloudHealthy
QwQwenMultilingual chat · weight 40CloudHealthy
DSDeepSeekCode & reasoning · weight 30CloudHealthy
AnAnthropic ClaudeFallbackCloudStandby
OlOllama · GPU-01http://10.0.8.21:11434LocalHealthy
vLvLLM · Inference clusterhttp://10.0.8.30:8000LocalHealthy

23 models discovered automaticallySample data

Supported models

Major cloud providers + local inference

Add your own provider API key or a local server address and you're connected. You pay providers directly for cloud usage; local models cost nothing per call. When a provider ships a new model, you usually don't need to upgrade TokenHub.

Deployment

Capability

22 model sources in total

Cloud models Provider APIs

OpenAI
GPT chat and embedding models
ChatEmbeddingVision
Anthropic Claude
Long documents and code
ChatVision
Google Gemini
Natively multimodal models
ChatEmbeddingVision
DeepSeek
General chat and deep reasoning
Chat
Qwen
Alibaba Cloud Model Studio
ChatEmbeddingRerankVision
ERNIE
Baidu AI Cloud Qianfan
ChatEmbeddingVision
GLM
Zhipu AI open platform
ChatEmbeddingVision
Kimi
Moonshot AI long-context models
ChatVision
Doubao
ByteDance Volcano Ark
ChatEmbeddingVision
Hunyuan
Tencent Cloud models
ChatEmbeddingVision
MiniMax
Chat and text embeddings
ChatEmbedding
Spark
iFLYTEK cognitive models
ChatEmbedding
Mistral
Open-weight and commercial models
ChatEmbeddingVision
Azure OpenAI
Regional enterprise deployments
ChatEmbeddingVision
AWS Bedrock
Many providers, one AWS account
ChatEmbeddingRerankVision

Local models On your hardware · data stays on your network

Ollama
Run open-weight models on one machine
ChatEmbeddingVision
vLLM
High-throughput GPU inference
ChatEmbeddingRerankVision
Xinference
Distributed model serving
ChatEmbeddingRerankVision
LM Studio
Local model server for desktops
ChatEmbeddingVision
LocalAI
CPU-friendly local inference
ChatEmbeddingRerankVision
SGLang
Fast structured-generation serving
ChatEmbeddingVision
Any OpenAI-compatible API
In-house or third-party servers
ChatEmbedding

Capability tags reflect typical support at the integration layer. Actual support depends on the models each provider offers; the admin console's auto-discovery is the source of truth. All brand names belong to their respective owners.

Unified API

Cloud or local, the call looks the same

Keep the official OpenAI SDKs and your existing code. Just point base_url at TokenHub. The only difference between a cloud model and a local one is the name in the model field.

auto: Let the routing policy pick the model
deepseek-chat: Use the provider's model name directly
ollama/qwen2.5:7b: Local channel prefix + local model name
team-default: An admin-defined alias you can repoint anytime

Supports Chat Completions, Embeddings, streaming, and function calling
Provider keys stay encrypted in your TokenHub install; apps only get TokenHub keys
The gateway normalizes provider-specific parameters and error codes

from openai import OpenAI

client = OpenAI(
    # replace with your TokenHub address
    base_url="http://localhost:8080/v1",
    api_key="th-your-api-key",
)
msgs = [{"role": "user", "content": "Turn this spec into tasks"}]

# Cloud model: use the provider's model name
cloud = client.chat.completions.create(
    model="deepseek-chat", messages=msgs)

# Local model: change only `model`, nothing else
local = client.chat.completions.create(
    model="ollama/qwen2.5:7b", messages=msgs)

# Embeddings go through the same endpoint
vec = client.embeddings.create(
    model="ollama/bge-m3", input=["travel expense policy"])

import OpenAI from "openai";

const client = new OpenAI({
  // replace with your TokenHub address
  baseURL: "http://localhost:8080/v1",
  apiKey: process.env.TOKENHUB_API_KEY,
});
const messages = [{ role: "user", content: "Turn this spec into tasks" }];

// Cloud and local models are called the same way
for (const model of ["qwen-plus", "ollama/qwen2.5:7b"]) {
  const stream = await client.chat.completions.create({
    model, messages, stream: true,
  });
  for await (const chunk of stream) {
    process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
  }
}

# Replace the host with your TokenHub address
# Cloud model
curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $TOKENHUB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "deepseek-chat",
       "messages": [{"role": "user", "content": "Hello"}]}'

# Local model: same URL, same key
curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $TOKENHUB_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "ollama/qwen2.5:7b",
       "messages": [{"role": "user", "content": "Hello"}]}'

Smart routing

Send every request to the right model

Take model selection out of your application code and let the gateway decide by policy. Policies live in the admin console and take effect immediately, with no code changes or redeploys.

Route by cost: send simple requests to local models (no per-call fees) or cheaper models, and save premium models for hard tasks
Route by latency: shift traffic based on each channel's recent response times to keep things fast
Route by capability: when a request needs function calling, vision, or long context, only models that support it are considered

Try the interactive demo: switch policies to see how traffic shifts, or simulate a primary-model outage to watch failover kick in.

Sample

POST/v1/chat/completionsmodel: "auto"

TokenHub routing engine

Rule: cheapest healthy channel first; local models handle routine requests

QwenCloud · primary

0%Healthy
DeepSeekCloud · backup

0%Healthy
Local QwenvLLM · on-prem

0%Healthy

All channels healthy. Traffic is split by the "Lowest cost" policy.

Failover

Give each model an ordered chain of backups. If the primary times out, gets rate-limited, or errors, traffic switches over and your apps never notice.

Load balancing

Attach multiple keys or local nodes to one model and spread traffic by weight, round robin, or least connections to get past per-key rate limits.

Automatic retries

Retry recoverable errors with exponential backoff, capped by attempt count and total timeout, so retries never snowball into an outage.

Circuit breaking

When a channel's error rate crosses a threshold, it's taken out of rotation and health-checked; once it recovers, traffic ramps back up gradually.

Metering & quotas

Account for every token

Every call through the gateway is metered. Token usage and spend are attributed to people, teams, and projects in real time, and you get a heads-up before a budget runs out.

Track tokens and spend by user, team, project, and API key
Convert usage to cost with each model's rates; set internal chargeback rates for local models
Set daily or monthly caps on tokens, spend, and request count
Get email or webhook alerts at thresholds; when a cap is hit, alert, downgrade, or block
Export detailed usage for chargebacks and department billing

Employee quotas & cost estimates See the Live Dashboard

Cost center / Team usage (provider spend)

Tokens this month246M

Spend this month$2,540

Budget used73%

Customer Support98.2M tokens · $860

82% · Warning
Engineering74.6M tokens · $710

64% · OK
Marketing39.8M tokens · $530

96% · Near cap
Legal & Compliance11.2M tokens · $145

35% · OK

≥ 80%: email alert ≥ 100%: fall back to local model ≥ 120%: block calls

Sample data · cost is calculated from provider rates and paid directly to providers

Capabilities

More than chat

Every model type shares the same auth, routing, and metering, and falls under the same permissions and audit logs.

Available

Chat

Multi-turn chat, streaming, function calling, and structured JSON output.

/v1/chat/completions

Available

Embedding

Turn text into vectors for semantic search, clustering, and RAG knowledge bases.

/v1/embeddings

Available

Rerank

Re-score search results by relevance for noticeably more accurate answers.

/v1/rerank

Partial

Images

Image understanding (image + text input) is available; image generation is on the roadmap.

/v1/images/*

Planned

Audio

Speech-to-text and text-to-speech for meeting notes, voice support, and more.

/v1/audio/*

Comparison

Model sources at a glance

Pick the right model for each job. Once connected, they're all available through the same API.

Capability comparison of common model sources
Model source	Deployment	Context length	Typical use
Cloud models
OpenAI / Azure OpenAI	Cloud	Long	General assistants, agents
Anthropic Claude	Cloud	Long	Long-document analysis, code
Google Gemini	Cloud	Very long	Multimodal understanding
DeepSeek	Cloud	Long	Reasoning, code, Chinese
Qwen / ERNIE / GLM	Cloud	Long	Chinese-language workloads
Kimi / Doubao / Hunyuan	Cloud	Long	Long text, chat apps
Local inference frameworks
Ollama / LM Studio	Local	Depends on model	Dev/test, lightweight on-prem
vLLM / SGLang	Local	Depends on model	High-concurrency production
Xinference / LocalAI	Local	Depends on model	Embedding / rerank services

Supported Some models Capabilities vary by model. Check each provider's documentation and the admin console's discovery results.

Local models

Connect on-prem models in three steps

Already have GPU servers? With no extra development, your local models can be called and managed just like cloud models in minutes, and your data never leaves your network.

Start a local inference server

Serve a model with Ollama, vLLM, Xinference, or any supported framework, and make sure TokenHub can reach it over the network.

ollama serve
ollama pull qwen2.5:7b

Add it in TokenHub

Under Model channels, pick the framework, enter the server address and an optional access key. TokenHub tests the connection right away.

Type:    Ollama
Address: http://10.0.8.21:11434

Models are discovered for you

TokenHub pulls the server's model list and detects each model's type. Select the ones you want, then set aliases, permissions, routing, and internal rates.

✓ qwen2.5:7b      Chat
✓ bge-m3          Embedding

Model channels / Add local channel

FrameworkOllama

Server addresshttp://10.0.8.21:11434

ConnectionConnected · 38 ms (sample)

4 models foundSample

qwen2.5:7bChat
llama3.1:8bChat
bge-m3Embedding
llava:7bVision

FAQ

Model integration questions

Do I need to change code that uses the OpenAI SDK?

Usually you only change two settings: base_url and api_key. TokenHub uses the OpenAI request and response formats and translates other providers' parameters for you.

Where are provider API keys stored? Are they safe?

Provider keys are stored encrypted in the TokenHub instance you deploy, and they never pass through a third-party server. Only admins with the right permissions can configure them. Developers and apps only get TokenHub-issued keys, which you can revoke, rate-limit, and audit at any time. See Access & Security.

What hardware do local models need?

The TokenHub gateway doesn't run inference itself, so its own requirements are modest. Inference runs on your Ollama, vLLM, or other servers, and their hardware needs depend on the model size. As long as the gateway can reach the inference server over the network, you can connect it.

When a provider releases a new model, do I have to wait for a TokenHub update?

If the new model uses the existing API protocol, just refresh the model list in the admin console or add the model name manually. Capabilities that need a new protocol are noted in the Changelog.

How are cloud model costs billed?

TokenHub doesn't mark up or resell model usage. Cloud models use your own provider API keys and you pay providers directly; local models have no per-call fees. TokenHub just meters usage and enforces quotas so every call is attributed to a person and project. TokenHub itself is free for personal and non-commercial use; for business use, ask about a commercial license.

How much latency does the gateway add?

The gateway only handles lightweight work like auth, routing, and metering, so the added latency is typically far smaller than the model's own generation time. Streaming responses are forwarded chunk by chunk, so time to first token is barely affected. Results depend on your environment, so we recommend measuring after you deploy.

Can I run fully offline with only local models?

Yes. TokenHub supports air-gapped deployment: it runs without internet access, connects only to local models and a local vector store, and your data never leaves your network. Get the installer on the Download page, and see the air-gapped deployment solution for details.

Put every model behind one gateway

Download TokenHub, deploy it on your own servers, and connect your first cloud or local model in minutes. Free for personal and non-commercial use; businesses need a commercial license.

Free Download Commercial licensing inquiry