Skip to main content
This document provides a comprehensive overview of TRU SCOUT, a core component of the Trusys AI Assurance Platform. It focuses on identifying and mitigating potential vulnerabilities in your AI applications through Security Configs and Security Evaluation processes.
Key Steps
  • Select your AI application or LLM model.
  • Define vulnerable categories and attack strategies.
  • Run security evaluations to identify and remediate risks.

​
New Security Evaluation

To initiate a new security evaluation, navigate to the ‘Security Eval’ tab on the left menu and click on ‘New Security Run’. This process allows you to perform a targeted security assessment of your AI application.
1

Choose Configuration

You have two options for setting up your security evaluation
  1. Using an existing security config: Select a previously created security configuration from the dropdown list. All the details defined in that config (e.g., application, AI application context, vulnerable categories, attack strategies) will be pre-populated. You can make temporary edits to this configuration for the current run; these edits will be saved when the run is initiated.
  2. Create a new config: If you need a fresh configuration, select this option.
    You will then be guided through the steps to create a new security config in section 1.1
2

Initiate Security Run

Once your configuration is set, click on ‘Run Security Eval’ to initiate the security run. Trusys will then execute the defined security tests against your selected AI application.

​
Security Evaluation List

The Security Runs of the Project section provides a comprehensive list of all security evaluations that have been performed within your project. This allows you to track the history and status of your security testing efforts. For each security run, you can view key information such as:
  • Run ID: A unique identifier for the security run.
  • Security Config Used: The name of the security configuration that was used for this run.
  • Application Tested: The AI application or LLM model that was subjected to the security evaluation.
  • Vulnerability & Strategy Count: A count of the number of vulnerabilities tested and the number of attack strategies used in that particular run.
  • Start Time: The timestamps indicating the start time of the security run.
  • Status: The current status of the run (e.g., Pending, Running, Completed, Failed).

​
Security Evaluation Run Details

Clicking on any security run from the security evaluation list will take you to the Security Run Details page. This page offers an in-depth analysis of the security evaluation results, providing both raw probe-level data and a summarized report.
  1. Summary Report: This section provides a high-level overview and aggregated analysis of the entire test run. It includes:
    • Attack-wise Analysis: A breakdown of vulnerabilities and their impact based on the different attack strategies employed (e.g., how effective were prompt injection attempts, or data leakage attempts).
    • Category-wise Analysis: An overview of the security probes across various vulnerable categories.
    • Overall Test Run Summary: A consolidated summary of the entire security evaluation, including the total number of probes, the overall pass/fail rate, and a risk categories etc. This provides a clear picture of the AI application’s security resilience and highlights areas requiring immediate attention.
  2. Raw Report of each probe: This section provides granular details of each security probe that was generated and executed for a specific combination of attack strategy and vulnerable category. For each probe, you can view:
    • Status: The outcome of the probe (e.g., Passed, Failed, Error).
    • Detailed Explanation: A comprehensive breakdown of the result, including the input used, the AI application’s response, and why the probe was flagged as a success or failure against the security objective. This allows for deep-dive analysis and understanding of specific vulnerabilities.