• Select Model – Pick your LLM from available options
• Authentication – Supply credentials to securely connect

• Provide the Phone Number associated with the AI-powered calling system
• Trusys automatically captures and evaluates conversational interactions

• Flowise
• Langflow
• Dify
• Microsoft Foundry Agents
• CrewAI
• n8n
• AWS Bedrock
• Google Vertex
• Salesforce Agentforce
• Microsoft Copilot Studio

• Select the agentic framework
• Provide framework-specific authentication details
• Configure connection parameters required by the selected framework

• Monitor multi-agent workflows
• Evaluate agent decisions and outputs
• Track security and functional risks across agent interactions

1. Select the Custom Provider Type
2. Choose the Request Type (e.g., GET, POST)
3. Enter the Base URL of the application’s API.
4. Choose how to structure the request body sent to your API.
   • Raw JSON — Write the request body as a JSON string directly.
   • Key-Value Pairs — Add individual fields using a structured editor.
   • Raw HTTP request The request body can contain the following variables that Trusys will automatically populate: Text Prompt: {{prompt}} – The text prompt/input for the test case Attachments (Optional): {{attachment1}} – First attachment (image, audio, video, document, etc.) {{attachment2}} – Second attachment {{attachment3}} – Third attachment Continue this pattern for additional attachments
     Mandatory: The API request must contain at least the {{prompt}} variable, as Trusys passes all test cases through this placeholder. Without {{prompt}}, evaluations and monitoring will not function correctly.Optional: If your prompts include attachments, include the corresponding {{attachment1}}, {{attachment2}}, etc. placeholders in your request body. If attachments are not used in a test case, these placeholders will be empty or omitted depending on your API’s requirements.
     Example Request Body (with attachments):
     Raw JSON

     {
       "message": "{{prompt}}",
       "image": "{{attachment1}}",
       "audio": "{{attachment2}}"
     }
     

     Key-Value Pairs

     Key	Value
     message	{{prompt}}
     image	{{attachment1}}
     audio	{{attachment2}}

5. Provide API authentication details for secure communication.
6. Define how to transform response data for compatibility with Trusys’s systems. Extract or reshape the API response before it reaches Trusys. If unset, Trusys parses the response as JSON, falling back to raw text.

{
  "choices": [
    {
      "message": {
        "role": "assistant",
        "content": "This is a test!"
      },
      "finish_reason": "stop",
      "index": 0
    }
  ]
}

providers:
  - id: https
    config:
      url: 'https://example.com/chat/completions'
      transformResponse: 'json.choices[0].message.content'

data: {"event":"token","data":{"content":"Hello"}}
data: {"event":"complete","data":{"answer":"This is the final answer.","confidence":0.92}}

(json, text) => {
  const line = text
    .split('\n')
    .find((l) => l.includes('"event":"complete"'));
  if (!line) {
    return '';
  }
  const payload = JSON.parse(line.replace(/^data:\s*/, ''));
  return payload.data.answer || '';
}

​

Conversation-Based Applications

{
  "chat_history": [
    {% for completion in _conversation %}
    {
      "role": "user",
      "content": "{{ completion.input }}"
    },
    {
      "role": "assistant",
      "content": "{{ completion.output }}"
    },
    {% endfor %}
    {
      "role": "user",
      "content": "{{prompt}}"
    }
  ]
}

{
  "chat_history": [
    {% for completion in _conversation %}
    {
      "role": "user",
      "content": "{{ completion.input }}"
    },
    {
      "role": "assistant",
      "content": "{{ completion.output }}"
    },
    {% endfor %}
    {
      "role": "user",
      "content": "{{prompt}}",
      "attachments": [
        "{{attachment1}}",
        "{{attachment2}}"
      ]
    }
  ]
}

1. Develop your custom provider script: Write a JavaScript or Python file that implements the callApi function (or call_api for Python) to interact with your AI model or API. This function will receive the prompt generated by the Trusys platform as an argument. The code should submit this prompt to the AI application and return the output. Please see examples provided below.
2. Select the custom provider: When configuring the application, choose the language in which you have implemented the custom provider.
3. Choose Language: Python or JavaScript

class EchoProvider {
id() {
return 'echo';
    }
async callApi(prompt, context, options) {
return {
  output: `Echo: ${prompt}`,
      };
  }
    }
module.exports = EchoProvider;

def call_api(prompt, options, context):
"""Simple provider that echoes the prompt with a prefix."""
config = options.get("config", {})
prefix = config.get("prefix", "Echo: ")

return {
    "output": f"{prefix}{prompt}"
}

function callApi(prompt, options, context) {
  const apiKey = process.env.API_KEY;
  const response = callYourAPI(prompt, apiKey);
  return { output: response };
}

module.exports = callApi;

def call_api(prompt, options, context):
  api_key = os.getenv('API_KEY')
  response = call_your_api(prompt, api_key)
  return { "output": response }