Skip to main content

Indexing settings

Indexing the data loaded into the knowledge base consists of several stages:

  1. Data processing: converting text into MD format.
  2. Chunking: dividing text into fragments (chunks).
  3. Summarisation (optional): creating summaries to improve responses to general questions.
  4. Vectorisation: converting the chunks into vector representations (embeddings).

In the Project settings → Indexing tab, you can change the parameters for chunking, vectorisation, and summarisation.

  • To view project settings, you need at least the KHUB_EDITOR role.
  • To edit project settings, you need the KHUB_OWNER or KHUB_ADMIN role.
caution

After changing the indexing settings, you’ll need to re-index the knowledge base.

Vectorisation​

The Vectoriser model parameter determines the language model for text vectorisation. This model will vectorise both your data and user queries:

  • text-embedding-3-large: a model by OpenAI. If you use this model, your data is sent to their servers.
  • intfloat/multilingual-e5-large: a model hosted on Tovie AI’s servers.

Chunking​

Chunking method​

The Chunking method parameter determines how the text will be split into chunks:

  • By length: The text will be chunked by length, considering word boundaries.
  • Using LLM: The text will be chunked using a language model. In this case, chunking is based on the text hierarchy, such as headings, paragraphs, and section or document titles.

The list of settings depends on the selected chunking method.

  • Max chunk size in characters.

    How the text will be chunked

    Suppose the setting value is 70. You have a text consisting of 2 sentences, 100 characters each.

    The text will be divided into 3 chunks:

    1. 70 characters from the first sentence.
    2. The remaining 30 characters from the first sentence and 40 from the second one.
    3. The remaining 60 characters from the second sentence.

  • Language: the language of the source documents. This setting helps chunk the text correctly. If your sources are in several languages, select the one most used in queries to the knowledge base.

Data preparation using LLM​

caution

Enabling these options may significantly increase your costs.

  • Enrich chunks: Add additional information to chunks to improve search quality: title, summary, keywords, and questions answered by the chunk.
  • Generate image descriptions: Add image description chunks for image search. For more details, see Images in response.

Document summarisation​

This function builds a hierarchy of summaries based on document contents. It improves the quality of answers to general and comparative questions, such as: “What is this document about?”, “What is the main strategy?”, or “Compare approaches A and B”.

Summarisation is most useful for long documents (more than 10 pages). For short documents, FAQs, or reference guides, the effect will be minimal.

caution

Enabling summarisation increases indexing time and cost. In the Per-document mode with 100 chunks, expect: +20% LLM calls, +30% tokens, +50–100% indexing time, and +20% index size. In the Cross-document mode, the load is even higher.

Mode​

Determines which summaries will be built:

  • Per-document: Suitable for most cases. Summaries are built within each document. When a document is updated or deleted, its summaries are automatically recreated or removed.

  • Cross-document: Enables answering questions that span multiple documents, such as “What are the common themes in the project?” or “How are documents X and Y related?”

    caution

    When any document is changed, cross-document summaries are completely rebuilt. This mode is not recommended for knowledge bases with frequent updates or a large number of documents.

Summarisation model​

For summarisation, it is recommended to select the same model used for chunk enrichment, or a more powerful one. More powerful models produce higher-quality summaries but increase indexing costs. The quality of upper-level summaries directly affects the quality of responses to general questions.

Access to summaries​

This setting is available only when the Cross-document mode is selected.

Cross-document summaries combine information from several documents that may belong to different access groups. This setting determines who can see such summaries:

  • Extended: A summary is available if the user has access to at least one of the source documents.
  • Strict: A summary is available only if the user has access to all source documents. Public documents are not included in the access check.

Summarisation prompt​

In the prompt, you can specify additional requirements for the content of summaries.

info

To view and edit the prompt, you need the KHUB_ADMIN role.

Advanced options​

  • Maximum tree depth: How many summary levels to build. A value of 3 is suitable for most cases. Increase it for very long documents (more than 100 pages) where high-level summaries are needed.
  • Minimum number of chunks to start: Documents with fewer chunks will not be summarised. For short documents, summarisation has no significant effect.
  • Maximum number of tokens in a single summary: Limits the length of each summary generated by the model. Increase it for long documents to get more comprehensive summaries.

LLM settings​

The LLM settings are used:

  • To form chunks if LLM-based chunking is selected.
  • To enrich chunks, regardless of the chunking method.
caution

These settings do not apply:

  • To generating image descriptions. The cloud version of Tovie Data Agent uses GPT-4o mini. If Tovie Data Agent is installed in your company’s infrastructure, it uses the model specified in its configuration.
  • To document summarisation. A separate model is used, which you select in the Document summarisation section.

Available settings:

  • Model: Select one of the available language models.
  • Max tokens in request: Limits the number of tokens that can be sent to the LLM.
  • Max tokens in response: Limits the number of tokens that the LLM can generate in one iteration.
  • Temperature: Adjusts the creativity of responses. Higher temperature values produce more creative and less predictable results.
tip

To see how your source is chunked, download the archive with chunks:

  1. Go to the Sources section and hover over the desired source.
  2. Click → Chunk archive.

When testing the knowledge base, you also can see which chunks are selected to generate the response.