Skip to main content
The Prompt Library is a central repository for reusable prompts. It enables organized and repeatable testing of your AI applications.

​
Building a Prompt Library

  1. Go to the Prompt Library tab in the navigation bar.
  2. Click New Prompt Library.
  3. Provide a unique name that reflects the library’s purpose.
    • Example: Customer Service Chatbot Prompts
    • Example: Medical Diagnosis Prompts
1

Create New Prompt

Within your newly created prompt library, you can add individual prompts for evaluation. Trusys currently supports three types of prompts.

Standard text inputs
  • Example: "What is the capital of France?"
  • Example with variables: "Translate the following sentence into French: {{sentence_to_translate}}"
You can use variables within your prompts using the syntax {{var}}. This allows you to create dynamic prompts where different values can be substituted during evaluation, enabling a single prompt to generate multiple test cases. You can use multiple variables per prompt.
The Conversation Builder helps simulate realistic user–AI interactions, generate scalable test cases, and evaluate how well your application handles context, timing, and intent over multiple turns.Supported conversation types are Text Conversations – Chat-based interactions without timing controls and Audio Conversations – Voice-based interactions with optional timing and flow controls

Conversation Structure

Each interaction between the user and the AI application is called a TURN.
  • Conversations are built turn by turn
  • Turns alternate between User and AI Application
  • Full conversation context is preserved and passed to the application at every turn

​
Building Conversation (Defining User Turns)

For every user turn, you can choose how the user response is generated.

​
1. Dataset-Based Responses

Use datasets to create structured, repeatable conversations.Options:
  • Select specific rows from a dataset
  • Select all items from a selected column
Each row represents a unique user input for that turn.

​
2. AI-Generated User Responses

Generate dynamic user messages using AI by specifying:
  • User Intention – What the user wants to achieve in that turn
    (e.g., “Request a refund”, “Ask a follow-up question”)
  • User Profile – Persona or context of the user
    (e.g., impatient customer, first-time user, expert user)
This approach is ideal for testing open-ended and realistic conversations.

​
Audio Conversation Controls

Audio conversations support additional timing controls to simulate real-world voice interactions:
  • Delay After Response
    Adds a pause (in seconds) before the next step begins
    Timer starts after the application finishes responding
  • Start After Response Begins
    Triggers the next step after a specified time
    Timer starts when the application begins responding
These options are not available for text conversations.

​
Test Case Generation

Trusys automatically expands conversations into multiple test cases based on input combinations.
  • Each dataset row creates one variation
  • Each User Intention × User Profile pair creates one variation
  • Variations multiply across turns
Example:
  • Turn 1: Dataset-based input with 4 rows
  • Turn 2: AI-generated input with 2 intention–profile combinations
➡️ Total test cases generated:
4 × 2 = 8 test casesThis logic applies equally to text and audio conversations.

​
Rules & Constraints

  • All dataset-based user turns must use the same number of selected rows
  • Once AI-generated user responses are used:
    • No additional turns can be added
    • The conversation ends after that turn
  • Dataset-based and AI-generated responses cannot be mixed beyond the AI-generated turn
These constraints ensure predictable execution and consistent evaluations.
2

Define Variables & Metrics

  1. Variables – Define multiple values for variables. Trusys generates test cases for each value.
Example: “Summarize: {{document_text}}” with document_text = [Article A, Article B, Article C] → 3 test cases.
Optionally, link a dataset to populate values (see Mapping Prompt Variables with Dataset).
  1. Metrics – Select evaluation metrics and expected values.
Example: Translation prompt → Fluency: High
Example: Factual question → Accuracy: Correct
Once all the details are filled, you can save the prompt. You can define multiple prompts within a single prompt set, allowing for organized testing of related functionalities.
3

Define Global Metrics

Instead of defining metrics for each individual prompt, you have the option to define global metrics for the entire prompt library. When global metrics are defined, all prompts within that library will automatically use these metrics for measurement, streamlining the configuration process, especially for large prompt libraries with consistent evaluation criteria.

​
Prompt Library List

The Prompt Library List provides an overview of all prompt libraries created within your project. For each library, you can view:
  • The number of prompts it contains.
  • The number of test runs that were executed using that specific prompt library.

​
Prompt Library Details

Clicking on any prompt library from the list will open its Prompt Library Details page. Here, you can:
  • View existing prompts and their metrics: Review all the prompts within the library and the specific metrics configured for each.
  • Edit the library: You have the flexibility to add new prompts, modify existing ones, or delete prompts that are no longer needed.Add, edit, or delete prompts
  • Review all functional evaluations done using this library: A list of all past functional evaluation test runs conducted using the current prompt library is displayed. Clicking on any item in this list will lead you to the detailed results of that specific test run, providing in-depth insights into the AI’s performance during that evaluation.