Contact Us 1-800-596-4880
  1. Homepage
  2. Enhanced MuleSoft Experience
  3. Creating and Managing Model Proxies

  4. Configuring Semantic Services

Configuring Semantic Services

Semantic services compare incoming requests to defined prompt topics to route traffic to the best-matching model path or block requests that match denylist topics. Model Proxy supports Basic Scale semantic services for simple routing and Advanced Scale semantic services for complex routing backed by an external vector database. Configure a semantic service before creating a model proxy that uses semantic routing.

Model Proxy supports two types of semantic services:

  • Advanced Scale:

    For complex semantic routing. Advanced scale semantic services use a vector database to store and compare prompt topic utterances. Advanced scale semantic services support unlimited prompt topics and 2000 utterances per prompt topic.

  • Basic Scale:

    For simple semantic routing and blocking. Basic scale semantic services support up to 6 prompt topics and 10 utterances per prompt topic.

Before You Begin

Before getting started, make sure you have:

  • An Anypoint Platform account.

  • One of these permissions:

    • Exchange: Exchange Contributor

    • Exchange: Exchange Administrator

    • Exchange: Exchange Creator

    For more information, see Enhanced Experience Permissions.

Configure an Advanced Scale Semantic Service

  1. In the navigation pane, select Model Proxies > Semantic Services.

  2. Click Add Semantic Service > Semantic Routing Service.

  3. Select Advanced Scale.

    Advanced scale supports large utterance sets per topic and requires a dedicated external vector database connection.

  4. Configure the embedding connection parameters:

    • Environment: The environment to use for the semantic service.

    • Embedding service provider: The provider of the embedding model. Select from these options:

      • OpenAI

      • Hugging Face

      • Azure OpenAI.

    • Service label: Label to identify the new service. Shown as the label in the Semantic Services catalog.

    • URL: The URL of the embedding service.

    • Model: The embedding model to use.

    • Auth key: The API authentication key for the embedding service.

  5. Configure the vector connection parameters:

    • Environment: The environment to use for the vector database.

    • Vector Database provider: The provider of the vector database. Select from these options:

      • Qdrant

      • Pinecone

      • Azure AI Search

    • Host: The host URL of your vector database.

    • API Key: The API authentication key for the vector database.

    • Collection: The collection name in your vector database.

  6. Define prompt topics:

    Supports up to 2000 utterances per topic.

    1. Click Create Prompt Topic.

    2. Define a Prompt topic name.

    3. Define prompt utterances or click Upload utterances to upload a plain text file containing your prompt utterances.

    4. Create as many prompt topics as necessary. You can also create new prompt topics later by editing the semantic service.

      To deny users from asking about certain subjects, create prompt topics for the subjects and apply them as deny list topics when configuring your Model Proxy.

  7. Click Save & Download Script.

  8. Open the downloaded .sh script file in your database to populate it with your scaled vectors.

Configure a Basic Scale Semantic Service

  1. In the navigation pane, click the Model Proxies dropdown arrow.

  2. Select Semantic Services.

  3. Click Add Semantic Service > Semantic Routing Service.

  4. Select Basic Scale.

    Basic scale supports up to six topics and 10 utterances per topic. Managed via internal Model Proxy configuration.

  5. Configure the embedding connection parameters:

    • Environment: The environment to use for the semantic service.

    • Embedding service provider: The provider of the embedding model: OpenAI, Hugging Face, or Azure OpenAI.

    • Service label: Label to identify the new service. Shown as the label in the Semantic Services catalog.

    • URL: The URL of the embedding service.

    • Model: The embedding model to use.

    • Auth key: The API authentication key for the embedding service.

  6. Click Save.

Edit a Semantic Service

To edit a semantic service:

  1. From Semantic Service Setup, click the three-dots menu (3%) of the semantic service you want to edit.

  2. Make the necessary edits.

  3. Click either Save or Save & Download Script depending on your sematic service.

    If creating new prompt topics for an advanced scale semantic service, you must download and run the vector script in your database again.