Files & retrieval controls the file pipeline for every user: upload limits, content extraction, OCR, full-text injection, vector retrieval, semantic enhancement, and RAG. It decides whether an uploaded file can be previewed, indexed, and used in conversation context.
Where to find it#
Open Admin console → Files & retrieval. The page has six groups, each with its own Save button:
- Upload limits
- Content extraction
- Full-text injection
- Vector retrieval
- Semantic enhancement
- RAG
These are platform settings. They apply to files processed after the change; already extracted or indexed files keep their current state until you rebuild the index.
The pipeline is: upload → text extraction → OCR when needed → full-text injection or chunking and vectorization → vector index → RAG retrieval into context. Semantic enhancement adds message vectors on top of the same Embedding service. Configure upload limits and extraction first, then Embedding, then RAG.
Upload limits#
Open Upload limits.
| Field | Default | Notes |
|---|---|---|
| Attachment limit | — | Maximum number of attachments in one chat message. |
| Attachment MIME allowlist | — | Allowed attachment MIME types, separated by commas. |
| Default size limit | — | Default size limit for a single attachment. The UI uses MB. |
| User storage quota | — | Maximum total file storage per user, in MB. Set 0 for unlimited. |
| Image size limit | — | Overrides the default limit for image attachments, in MB. Leave empty to use the default attachment limit. |
| Document size limit | — | Overrides the default limit for document attachments, in MB. Leave empty to use the default attachment limit. |
The MIME allowlist decides which file types users can upload. Keep it aligned with the extraction engines you enable. If you use MinerU, the platform warns when the allowlist is missing MIME types for the selected MinerU file types and offers Add and save missing MIME types.
Content extraction#
Open Content extraction. This group selects the primary extraction engine and the shared OCR engine.
Extraction engine#
| Engine | When to use it |
|---|---|
| Built-in | Local text extraction for common document, code, and text files. |
| Tika | General document text extraction through an Apache Tika service. |
| Docling | Structured document parsing through a Docling service. |
| MinerU | PDF, Word, PowerPoint, and Excel parsing through MinerU cloud or a self-hosted service. |
The engine you pick shows its own settings. Each external service has an address, an optional auth key, and a timeout. Use Test connection to check the address and credentials; it is disabled while the group has unsaved changes.
| Engine | Fields |
|---|---|
| Tika | Tika service URL (required), Tika auth key, Tika timeout (default 60). |
| Docling | Docling service URL (required), Docling auth key, Docling timeout (default 60). |
| MinerU | MinerU service type (Cloud or Self-hosted), MinerU service URL (required), MinerU file types (PDF, Word, PowerPoint, Excel), MinerU auth key, MinerU timeout (default 180). |
MinerU defaults to PDF, Word, and PowerPoint. Self-hosted MinerU accepts only DOCX, PPTX, and XLSX for those categories; cloud accepts the older DOC, PPT, and XLS formats too. The default URL is https://mineru.net/api/v4 for cloud and http://127.0.0.1:8000 for self-hosted.
Saving fails when the selected engine has no service URL:
- Tika service URL is required.
- Docling service URL is required.
- MinerU service URL is required.
OCR scope#
| Switch | Behavior |
|---|---|
| Image OCR | Send image attachments through OCR and generate text for retrieval. |
| PDF OCR fallback | Use OCR fallback when native PDF text extraction fails or has poor quality. |
The OCR engine field appears when either switch is on.
OCR engine#
| Engine | Required fields |
|---|---|
| Rapid OCR | Rapid OCR service URL, optional auth key, timeout (default 60). |
| Tesseract OCR | Tesseract OCR service URL, optional auth key, timeout (default 60). |
| Paddle OCR | Paddle OCR service URL, optional auth key, timeout (default 60). |
| Tencent Cloud OCR | Tencent Cloud SecretId, SecretKey, and region (default ap-guangzhou); endpoint defaults to ocr.tencentcloudapi.com; timeout default 60. |
| Alibaba Cloud OCR | Alibaba Cloud AccessKey ID, AccessKey Secret, and region (default cn-hangzhou); endpoint defaults to ocr-api.cn-hangzhou.aliyuncs.com; timeout default 60. |
| Mistral OCR | Mistral OCR service URL (default https://api.mistral.ai/v1/ocr), API key, model (default mistral-ocr-latest), timeout default 60. |
| LLM OCR | LLM OCR service URL, optional auth key, model, timeout default 60, optional prompt. |
OCR validation errors match the selected engine, for example Tesseract OCR service URL is required., Tencent Cloud OCR SecretId, SecretKey, and region are required., Mistral OCR API key is required., or LLM OCR model is required.
Full-text injection#
Open Full-text injection. Full-text injection puts the extracted text directly into the conversation context instead of retrieving chunks.
| Field | Notes |
|---|---|
| Full-text injection limits | When on, full-text injection is limited by size, token budget, and PDF pages. When off, these limits are ignored. Default on. |
| Full-text size limit | Maximum extracted text size for injection, in MB. Empty or 0 means no size limit. Shown when limits are on. |
| Full-text token limit | Maximum token budget for injection. Empty or 0 means no token limit. Shown when limits are on. |
| Full-text page limit | Maximum PDF pages for injection. Empty or 0 means no page limit. Shown when limits are on. |
Use full-text injection for small files where exact wording matters. For large files or many files, use vector retrieval and RAG instead.
Vector retrieval#
Open Vector retrieval. This group configures the external Embedding service that converts file content into vectors.
| Field | Default | Notes |
|---|---|---|
| Enable Embedding | Off | Enables file vectorization and vector retrieval. Related pipelines are disabled when this is off. |
| Embedding service URL | — | Required when Embedding is on. The service must be compatible with the OpenAI /embeddings API. |
| Embedding auth key | — | Auth key used when calling the Embedding service. |
| Embedding model | — | Required when Embedding is on. Model name used for vectorization. |
| Embedding timeout | 60 | Maximum wait time for one Embedding API request, in seconds. |
| Vector dimensions | 1536 | Dimensions used consistently for storage and retrieval. |
| Dimensions request policy | Include | Whether to include the dimensions parameter in requests. Choose Omit when the service rejects it. |
| Normalize vectors | On | Normalize vectors before storage and retrieval. |
| Auto-vectorize | On | Trigger Embedding asynchronously after file upload completes. |
| Batch size | 20 | Number of text chunks sent in one Embedding API request. |
| Chunk size | 1024 | Document vectorization chunk size, estimated by tokens. |
| Chunk overlap | 64 | Token overlap between adjacent chunks. |
Vector dimensions must match the model output. If the provider rejects the dimensions parameter, switch the policy to Omit; the returned vector is still validated against the configured dimensions. Changing the Embedding model or Vector dimensions marks existing vectors stale.
Vector index status#
When Embedding is on, the group shows Vector index status with the current model signature and counts for Ready, Stale, Pending, No text, and Failed. The panel shows a warning when stale vectors are detected: Stale vectors detected. Rebuild the index to restore retrieval quality.
- Rebuild index submits a background reindex of the files that need it.
- Reindex including no-text files appears when files without extractable text exist, and includes them in the rebuild.
- The signature is computed automatically after the first save. Before that it shows No signature yet. It is computed automatically after the first save.
A reindex runs in the background. Large libraries take time, and retrieval for some files can be incomplete until the rebuild finishes.
Semantic enhancement#
Open Semantic enhancement. This group recalls relevant historical messages using the same Embedding service.
| Field | Behavior |
|---|---|
| Message embedding | Generate message vectors asynchronously after each conversation turn. Requires an available Embedding service. |
| Semantic context recall | Recall historically relevant snippets and inject them into context before sending a message. Requires message embedding. |
Both switches are rejected until Embedding is enabled and both the service URL and model are set: Enable vector retrieval and configure both the Embedding service URL and model first. Turning on Semantic context recall without Message embedding shows Enable message embedding first.
RAG#
Open RAG. This group controls retrieval and how chunks are injected.
| Field | Default | Notes |
|---|---|---|
| RAG | Off | Allow files to enter the retrieval-augmented pipeline. Requires an available Embedding service. |
| Retrieved chunks | 5 | Number of RAG chunks returned and injected. |
| Similarity threshold | 0.45 | Minimum similarity for injecting retrieved chunks. |
| Injected token budget | 2000 | Token budget available for injected chunks. |
| Fetch multiplier | 3 | The system fetches extra candidates before reranking and filtering. |
| Ready wait | 3000 | Maximum time to wait for vectorization before sending a request, in milliseconds. |
| Query history turns | 0 | Recent user turns included when generating the retrieval query. |
| Retrieval cache TTL | 120 | RAG retrieval result cache duration, in seconds. |
| Hybrid retrieval (BM25 + vector) | Off | Run vector retrieval and full-text retrieval in parallel, then merge results with RRF. |
RAG cannot be turned on until Embedding is ready: Enable vector retrieval and configure both the Embedding service URL and model first. The fields after the RAG switch appear only when RAG is on and Embedding is ready.
File evidence retention is part of context compression, not this page. Set Evidence retention days on the Conversation page under Context management; see Conversation.
Setting key reference#
| Group | Setting keys |
|---|---|
| Upload limits | storage.max_message_files, file.allowed_mime_types, storage.max_upload_file_bytes, storage.user_storage_quota_bytes, file.image_max_bytes, file.doc_max_bytes |
| Content extraction | extract.engine, extract.tika_base_url, extract.tika_auth_token, extract.tika_timeout_seconds, extract.docling_base_url, extract.docling_auth_token, extract.docling_timeout_seconds, extract.mineru_source, extract.mineru_base_url, extract.mineru_file_types, extract.mineru_auth_token, extract.mineru_timeout_seconds |
| OCR scope | extract.image_ocr_enabled, extract.pdf_ocr_fallback_enabled, extract.ocr_engine |
| OCR engines | extract.rapidocr_base_url, extract.rapidocr_auth_token, extract.rapidocr_timeout_seconds, extract.tesseract_ocr_base_url, extract.tesseract_ocr_auth_token, extract.tesseract_ocr_timeout_seconds, extract.paddle_ocr_base_url, extract.paddle_ocr_auth_token, extract.paddle_ocr_timeout_seconds, extract.tencent_ocr_secret_id, extract.tencent_ocr_secret_key, extract.tencent_ocr_region, extract.tencent_ocr_endpoint, extract.tencent_ocr_timeout_seconds, extract.aliyun_ocr_access_key_id, extract.aliyun_ocr_access_key_secret, extract.aliyun_ocr_region, extract.aliyun_ocr_endpoint, extract.aliyun_ocr_timeout_seconds, extract.mistral_ocr_base_url, extract.mistral_ocr_auth_token, extract.mistral_ocr_model, extract.mistral_ocr_timeout_seconds, extract.llm_ocr_base_url, extract.llm_ocr_auth_token, extract.llm_ocr_model, extract.llm_ocr_timeout_seconds, extract.llm_ocr_prompt |
| Full-text injection | file.full_context_limit_enabled, file.file_full_context_max_bytes, file.full_context_max_tokens, file.full_context_pdf_max_pages |
| Vector retrieval | file.embedding_enabled, file.embedding_host, file.embedding_key, file.rag_model, file.embedding_timeout_seconds, file.embedding_output_dimensions, file.embedding_dimensions_policy, file.embedding_normalize, file.embed_trigger_on_upload, file.embed_batch_size, file.embed_chunk_size_tokens, file.embed_chunk_overlap_tokens |
| Semantic enhancement | chat.message_embedding_enabled, chat.semantic_context_enabled |
| RAG | chat.rag_enabled, file.rag_top_k, chat.rag_min_similarity, chat.rag_token_budget, chat.rag_fetch_multiplier, chat.rag_wait_ready_ms, chat.rag_query_history_turns, chat.rag_retrieval_cache_ttl_seconds, chat.rag_hybrid_enabled |
Troubleshooting#
- RAG or semantic enhancement will not turn on. Embedding must be enabled with both a service URL and a model. Semantic context recall also needs Message embedding.
- Retrieval quality dropped after a model change. Existing vectors are stale. Open Vector index status and click Rebuild index.
- A file has no searchable text. Its status is No text. The file can still be used through full-text injection, but it is not indexed. Use Reindex including no-text files only if extraction should produce text.
- A service test fails. Check the service URL, credentials, and that the group has no unsaved changes. Test connection is disabled while fields are dirty.
- MinerU does not receive a file type. The MIME allowlist is missing that format. Add the missing MIME types from the warning, or add them manually.
- Uploads are rejected. Check the MIME allowlist, the default size limit, the per-type image or document limit, and the user storage quota.