Answers from your docs, with sources you can check
Upload policies, product manuals, and technical docs, or connect websites and databases. TokenHub parses, chunks, and embeds them automatically. Hybrid search and reranking find the right passages, and every answer cites them, one click away from the original.
The knowledge base runs inside your TokenHub deployment. Pair local embeddings, the built-in vector store, and a local LLM to run fully air-gapped, with documents that never leave your network.
New York is a high-cost city, so hotels are capped at $300 per night[1]. You cover any overage unless an exception is approved in advance. Submit your expense report within 15 calendar days after the trip[2].
From documents to answers you can trust
Ingestion turns documents into searchable knowledge. At question time, TokenHub finds the right passages first and only then generates an answer. Every step is configurable.
Connect knowledge wherever it lives
Common office formats work out of the box, and websites and business systems sync on a schedule, so your knowledge base stays current.
Handles multi-column layouts, headers, footers, and tables. Turn on OCR for scans. Citations point to the exact page.
Word
Keeps heading levels and lists intact. Great for policies, contracts, and proposals.
Excel / CSV
Chunks by row or table region and carries the headers along, so fields keep their meaning.
Markdown / TXT
Splits into sections by heading. Ideal for technical docs, FAQs, and wiki exports.
Web crawling
Enter a site URL or sitemap, crawl body content by depth and path rules, and sync changes on a schedule.
Databases
Connect MySQL, PostgreSQL, and more. Pick fields with SQL and turn them into searchable text.
API sync
Push documents or add, update, and delete chunks through the API for tight integration with your systems.
Lark · DingTalk · Confluence
Sync content from collaboration and wiki platforms, along with their existing access permissions.
Better chunks, better retrieval
Chunking sets how granular search can be. TokenHub offers three strategies you can set per knowledge base, or even per document, and check instantly with a retrieval test.
Splits text into chunks of a set token length, with overlap between neighbors so sentences cut mid-way don't lose their context.
- Simple and fast; good for long, loosely structured text
- Configure chunk size, overlap, and preferred break characters
Detects heading levels and paragraph boundaries, splits by section, and stores each chunk's heading path as metadata.
- Best for well-structured content like policies, manuals, and technical docs
- Citations can point to the exact chapter and article
Measures how similar neighboring sentences are and splits where the topic changes, so each chunk covers one idea.
- Best for conversational content like meeting notes, interviews, and support chats
- More precise retrieval, at the cost of slower ingestion
\n\n.;##
Highlighted text is the overlap between neighboring chunks
Built in and ready, or bring your own
Use the built-in vector store for small and mid-sized knowledge bases with zero ops. For large volumes or existing infrastructure, connect a mainstream vector database.
- Built-in vector storeOn by default, deployed with TokenHubDefault
- MilvusDistributed vector search at scaleExternal
- QdrantHigh-performance vector searchExternal
- pgvectorReuse your existing PostgreSQLExternal
- ElasticsearchVector + full-text in oneExternal
Embedding models can be cloud or local, and you manage them all under Model Integration. All brand names belong to their respective owners.
Beyond vector similarity
Layered retrieval techniques cut down on "nothing found" and "found the wrong thing," so the model sees the passages that actually matter.
Hybrid search
Semantic vector search and keyword full-text search run in parallel, so part numbers, IDs, and proper nouns match exactly.
Reranking
A rerank model re-scores the results so the most relevant chunks go to the LLM first.
Multiple knowledge bases
Attach several knowledge bases to one app, like policies plus product docs. Results are ranked together and merged into one answer.
Query rewriting
Uses the conversation to resolve references and add synonyms, turning "how do I expense that?" into a complete, searchable question.
Retrieval test: see what the system finds
Before going live, test retrieval with real questions. Check each chunk's relevance and source, then decide how to tune your settings.
Click "Run search" to see what comes back.
All documents, chunks, scores, and answers above are built-in demo data, shown only to illustrate how the retrieval test works.
The model only cites what you're allowed to see
Permissions apply at retrieval time. Documents a user can't access are never retrieved, so they can't show up in answers or citations. Leaks are stopped at the source.
- Grant access to knowledge bases and documents by department, role, or user, with inheritance and exceptions
- The same assistant gives each employee answers that match their permissions
- Connect your SSO and org chart so permissions follow people when they change roles
- Every search and citation is written to the audit logs, so you can trace who saw which passage
- Sales Incentive Plan.pdfSales · Everyone can readCited
- Commission Guide.xlsxSales · Everyone can readCited
- Salary Adjustment Plan.docxHR onlyFiltered
- Leadership Meeting Notes.pdfDirectors and aboveFiltered
2 restricted documents were excluded at retrieval. The answer will say: "For salary adjustment rules, please contact HR."
Documents change. Your knowledge base keeps up.
When a document changes, only the changed parts are reprocessed. Progress and failure reasons are easy to see, and the knowledge base stays in sync with the source.
- Incremental updates: only new or edited chunks are re-embedded, saving time and tokens
- Versioning: keep history, compare versions, and roll back in one click
- Scheduled sync: websites and data sources sync automatically, and deleted content is removed
- Clear status: see parsing, done, and failed states live, with reasons you can act on and retry
| Document | Version | Chunks | Updated | Status |
|---|---|---|---|---|
| PDFTravel Policy.pdf | v3 | 128 | Oct 14 | Done |
| DOCEmployee Handbook.docx | v5 | 342 | Oct 14 | Parsing 62% |
| WEBHelp Center (web crawl) | v12 | 906 | Daily | Done |
| XLSVendor List.xlsx | v1 | — | Oct 13 | Failed |
| MDIT Service Guide.md | v8 | 57 | Oct 12 | Done |
Why it failed: Vendor List.xlsx is password-protected and can't be parsed. Remove the password, then click Retry.
Sample data
RAG Knowledge Base questions
Are documents in the knowledge base used to train models?
No. RAG only retrieves relevant chunks at question time and passes them to the model as context; it never changes model weights. The knowledge base is stored in your own TokenHub deployment, and if you also use local models and a local vector store, your documents never leave your network.
Will the model make things up when the answer isn't in the knowledge base?
You can set a relevance threshold and an answer policy. When no chunk meets the threshold, the assistant says plainly that the knowledge base has nothing on the topic, and can point users to the right contact, instead of guessing.
Can it read scanned PDFs and text in images?
Yes. Turn on OCR for a knowledge base and scans and embedded images are converted to text before chunking. Accuracy depends on the quality of the original, so spot-check with a retrieval test.
How many documents can one knowledge base hold?
The built-in vector store is a good fit for small and mid-sized knowledge bases. For large volumes or demanding search performance, connect an external store like Milvus or Elasticsearch and scale out. Actual capacity depends on your server resources; see the deployment options and hardware guidance on the Download page.
How do my applications query the knowledge base?
Pass a knowledge base ID when you call the unified API and answers come back with citations. You can also call the search endpoint on its own to get just the retrieved chunks for your own pipeline. See the Quickstart.
Do I need to rebuild the index if I switch embedding models?
Yes. Vectors from different embedding models can't be mixed. When you switch, TokenHub rebuilds the index in the background while the old index keeps serving, then cuts over automatically when it's done.
Run a retrieval test on your own documents
Download TokenHub, load a few real documents on your own server, and use the retrieval test to check search accuracy and answer quality. Free for personal and non-commercial use; businesses need a commercial license.