Contact Us 1-800-596-4880
  1. Homepage
  2. Omni Gateway (1.13)
  3. LLM Proxy Overview

  4. Configuring a Semantic Service

Configuring a Semantic Service

A semantic service compares incoming requests to the defined prompt topic utterances and sends the request to the route that best matches it. The semantic service also compares the request to the deny list topic utterances to block certain requests.

LLM Proxy supports two types of semantic services:

  • Advanced Scale: For complex semantic routing. Advanced scale semantic services use a vector database to store and compare prompt topic utterances. Advanced scale semantic services support unlimited prompt topics and 2000 utterances per prompt topic.

  • Basic Scale: For simple semantic routing and blocking. Basic scale semantic services support up to 6 prompt topics and 10 utterances per prompt topic.

Configure an Advanced Scale Semantic Service

  1. From API Manager, click Semantic Service Configuration.

  2. Click + Create a Semantic Service Configuration.

  3. Select Advanced Scale.

  4. Configure the semantic service parameters:

    • Service label: Label to identify the new service.

    • Embedding Service Provider: The provider of the embedding model. OpenAI or Hugging Face.

    • URL: The URL of the embedding service.

    • Model: The embedding model to use.

    • Auth key: The API authentication key for the embedding service.

  5. Click Vector connection.

  6. Select a Vector Database Provider from these options:

    • Quadrant

    • Pinecone

    • Azure AI Search

  7. Configure the parameters to connect your database.

  8. Create prompt topics:

    1. Click Create prompt topics.

    2. Define a Prompt topic name.

    3. Define prompt utterances or click Upload utterances to upload a plain text file containing your prompt utterances.

    4. Create as many prompt topics as neccesary. You can also create new prompt topics later by editing the semantic service.

      To deny users from asking about certain subjects, create prompt topics for the subjects and apply them as deny list topics when configuring your LLM Proxy.

  9. Click Save & download script.

  10. Open the downloaded .sh script file in you database to populate it with your scaled vectors.

Configure a Basic Scale Semantic Service

  1. From API Manager, click Semantic Service Configuration.

  2. Click + Create a Semantic Service Configuration.

  3. Select Basic Scale.

  4. Configure the semantic service parameters:

    • Service label: Label to identify the new service.

    • Embedding Service Provider: The provider of the embedding model. OpenAI or Hugging Face.

    • URL: The URL of the embedding service.

    • Model: The embedding model to use.

    • Auth key: The API authentication key for the embedding service.

  5. Click Deploy.

Basic Scale Semantic Service Routing Limits

Limit Value

Prompt topics (across all routes of an LLM Proxy)

6

Utterances per prompt topic

10

Deny list topics

6

Utterances per deny list topic

10

Maximum characters per utterance

500

Edit a Semantic Service

To edit a semantic service:

  1. From Semantic Service Setup, click the three-dots menu (3%) of the semantic service you want to edit.

  2. Make the necessary edits.

  3. Click Redeploy.

Redownload Vector Script

If creating new prompt topics, it is necessary to redownload and run the vector script in your database again:

  1. Download the script for the advanced scale semantic service by clicking the three-dots menu (3%) from Semantic Service Configuration.

  2. Click Download script.