Skip to main content
The Prompt Library is a central repository for reusable prompts. It enables organized and repeatable testing of your AI applications.

​
Building a Prompt Library

  1. Go to the Prompt Library tab in the navigation bar.
  2. Click New Prompt Library.
  3. Provide a unique name that reflects the library’s purpose.
    • Example: Customer Service Chatbot Prompts
    • Example: Medical Diagnosis Prompts
1

Create New Prompt

Within your newly created prompt library, you can add individual prompts for evaluation. Trusys currently supports three types of prompts.
Standard text inputs
  • Example: "What is the capital of France?"
  • Example with variables: "Translate the following sentence into French: {{sentence_to_translate}}"
You can use variables within your prompts using the syntax {{var}}. This allows you to create dynamic prompts where different values can be substituted during evaluation, enabling a single prompt to generate multiple test cases. You can use multiple variables per prompt.

​
Attachments (Optional)

You can optionally attach files (images, audio, video, documents) to your prompts. Attachments are passed to your AI application using the syntax {{attachment1}}, {{attachment2}}, and so on.How Attachments Work:
  • The prompt text is passed as {{prompt}}
  • The first attachment is passed as {{attachment1}}
  • The second attachment is passed as {{attachment2}}
  • Continue this pattern for multiple attachments
Adding Attachments:
  1. Click + Add Attachment
  2. Choose your source:
    • Upload Files – Upload image, audio, video, or document files directly
    • Use Dataset – Link a dataset to populate attachments (only audio, video, image, and file column types are supported)
Example:If your prompt is: "Analyze this image and describe what you see: {{prompt}}" and you attach an image file, the image will be passed as {{attachment1}} to your AI application.You can attach the same prompt library to attachments, variables, and metrics for comprehensive testing.
The Conversation Builder helps simulate realistic user–AI interactions, generate scalable test cases, and evaluate how well your application handles context, timing, and intent over multiple turns.Supported conversation types are Text Conversations – Chat-based interactions without timing controls and Audio Conversations – Voice-based interactions with optional timing and flow controls

Conversation Structure

Each interaction between the user and the AI application is called a TURN.
  • Conversations are built turn by turn
  • Turns alternate between User and AI Application
  • Full conversation context is preserved and passed to the application at every turn

​
Building Conversation (Defining User Turns)

For every user turn, you can choose how the user response is generated.

​
1. Prompt-Based Responses

Use prompts to create structured, repeatable conversations (by attaching datasets). Prompt-based responses can now include attachments from your dataset columns. This allows you to build conversations that include images, audio, video, or documents along with text prompts.How It Works:
  • Enter a text prompt (optionally include dataset in the prompt).
  • Optionally select media columns (audio, video, image, file) for attachments.
  • Attachments from the dataset are passed as {{attachment1}} , {{attachment2}} etc.
  • Each dataset row creates a unique user turn with both text prompt and attachments.

​
2. AI-Generated User Responses

Generate dynamic user messages using AI by specifying:
  • User Intention – What the user wants to achieve in that turn
    (e.g., “Request a refund”, “Ask a follow-up question”)
  • User Profile – Persona or context of the user
    (e.g., impatient customer, first-time user, expert user)
This approach is ideal for testing open-ended and realistic conversations.

​
Audio Conversation Controls

Audio conversations support additional timing controls to simulate real-world voice interactions:
  • Delay After Response
    Adds a pause (in seconds) before the next step begins
    Timer starts after the application finishes responding
  • Start After Response Begins
    Triggers the next step after a specified time
    Timer starts when the application begins responding
These options are not available for text conversations.

​
Test Case Generation

Trusys automatically expands conversations into multiple test cases based on input combinations.
  • Each dataset row creates one variation
  • Each User Intention × User Profile pair creates one variation
  • Variations multiply across turns
Example:
  • Turn 1: Dataset-based input with 4 rows
  • Turn 2: AI-generated input with 2 intention–profile combinations
➡️ Total test cases generated:
4 × 2 = 8 test casesThis logic applies equally to text and audio conversations.

​
Rules & Constraints

  • All dataset-based user turns must use the same number of selected rows
  • Once AI-generated user responses are used:
    • No additional turns can be added
    • The conversation ends after that turn
  • Dataset-based and AI-generated responses cannot be mixed beyond the AI-generated turn
These constraints ensure predictable execution and consistent evaluations.
2

Define Variables & Metrics

  1. Variables – Define multiple values for variables. Trusys generates test cases for each value.
Example: “Summarize: {{document_text}}” with document_text = [Article A, Article B, Article C] → 3 test cases.
Optionally, link a dataset to populate values - see Mapping Prompt Variables with Dataset.
  1. Metrics – Select evaluation metrics and expected values.
Example: Translation prompt → Fluency: High
Example: Factual question → Accuracy: Correct
Once all the details are filled, you can save the prompt. You can define multiple prompts within a single prompt set, allowing for organized testing of related functionalities.
3

Define Global Metrics

Instead of defining metrics for each individual prompt, you have the option to define global metrics for the entire prompt library. When global metrics are defined, all prompts within that library will automatically use these metrics for measurement, streamlining the configuration process, especially for large prompt libraries with consistent evaluation criteria.

​
Prompt Library List

The Prompt Library List provides an overview of all prompt libraries created within your project. For each library, you can view:
  • The number of prompts it contains.
  • The number of test runs that were executed using that specific prompt library.

​
Prompt Library Details

Clicking on any prompt library from the list will open its Prompt Library Details page. Here, you can:
  • View existing prompts and their metrics: Review all the prompts within the library and the specific metrics configured for each.
  • Edit the library: You have the flexibility to add new prompts, modify existing ones, or delete prompts that are no longer needed.Add, edit, or delete prompts
  • Review all functional evaluations done using this library: A list of all past functional evaluation test runs conducted using the current prompt library is displayed. Clicking on any item in this list will lead you to the detailed results of that specific test run, providing in-depth insights into the AI’s performance during that evaluation.