AI Agent
The AI Agent scenario block allows you to automate purposeful interactions with customers in both voice and chat scenarios using language models from AI provider integrations. You define any number of requested outcomes and instruct the model how to behave. The AI Agent block uses the language model to conduct a voice or chat dialog with the customer, following your instructions and gathering information for the requested outcomes. Once the block has gathered enough information to reach a desired outcome, the block exits, and scenario variables are populated with data extracted from the dialog.
| Prerequisites | ||
|
| |
How the Block Holds Conversations
The AI Agent block conducts conversations with customers according to the following steps:
- Initialization: When a scenario executes the AI Agent block (dialog block), the AI provider (AI) is used to create the first message for the customer based on the setting Instructions to AI and the list of Requested Outcomes.
- Customer Interaction (Voice or Chat):
- Voice: The AI's message is played to the customer using your default text-to-speech provider. The customer's response is converted to text using the Speech-to-Text integration indicated by the Transcriber setting.
- Chat: The AI's message is sent to the customer as a chat message. The dialog block waits for the customer's text response.
- If the Reply timeout elapsed without a response from the customer, the AI is informed there was "no reply" from the customer.
- Response Evaluation: The customer's response is analyzed by the AI.
- If the customer response contains data related to any of the Requested Outcomes, the AI extracts and formats the data for further use by the dialog block.
- If all the required data has been collected for a Requested Outcome, the AI indicates which outcome the dialog block should follow.
- The AI generates the next message for the customer. The next message is only sent to the customer if the dialog block continues the conversation.
- Exit or continue: The AI Agent block determines whether to continue the conversation or exit the block.
- Outcome Achieved: If all the data for one of the Requested Outcomes has been collected and the AI indicated which outcome to follow, the dialog block exits via the corresponding conditional exit. Whether the final AI message is sent to the customer is controlled by the Trim final AI response setting.
- Continue Dialog: If there is not enough data to satisfy any of the Requested Outcomes, the dialog block returns to the customer interaction step, sending the next message to the customer and awaiting a response.
- Maximum dialog turns: If the customer does not respond after several messages from the AI, or the maximum number of dialog turns have passed, then the dialog block follows the default exit with Error
1005"Reached the maximum number of dialog turns".
Settings
Title Text
The name of the instance of the block in the scenario.
AI Provider
Select the AI provider account to use for this block, or select <<default>> if an AI account is supplied by your service provider.
Transcriber
Only for AI Agent blocks used in voice scenarios.
The speech-to-text service that will be used to transcribe the user's statements. Select from among the configured Speech-to-Text (STT) integration accounts.
End of phrase timeout
Only for AI Agent blocks used in voice scenarios.
The duration of silence after the customer's reply that indicates they have finished speaking. When the customer begins speaking, their reply is recorded until there is a pause with duration equal to the End of phrase timeout, at which point their reply is considered finished and is sent to the AI provider for evaluation.
Allow interrupting prompt by customer voice
Only for AI Agent blocks used in voice scenarios.
When enabled, the block constantly listens for the customer's voice while an AI message is playing. When a voice is detected during an AI message, the block will:
- Stop playing the AI message
- Record and transcribe the customer's response
- Continue the interaction by sending the customer's response to the AI provider for evaluation
When disabled, the block will begin to listen for the customer's voice only after the entire AI message has been played. Any voice input from the customer before the AI message ends is ignored.
Speech contexts
Only for AI Agent blocks used in voice scenarios.
An optional list of words or phrases likely to be said by the customer. Listed terms are more likely to be transcribed accurately. See the Speech context setting of the Play-Listen block for more information.
Prompt to play while waiting for AI
Only for AI Agent blocks used in voice scenarios.
The selected audio prompt will play while the block waits for the AI provider to return a response. You may find that a musical prompt or sound effect helps customers understand that the system is processing their response.
Instructions to AI
These instructions are given to the AI during every turn of the dialog, allowing you to provide guidance on how the AI should conduct the conversation with the customer. Scenario variables can be used to personalize the instructions for each customer.
Any instructions to the AI should take into account the particular model you have selected with the AI provider. You may find it useful to consult the Prompt Gallery from Vertex AI and the Prompt Engineering Guide from OpenAI for ideas on how to best instruct the AI.
Regardless of the model or the provider you have selected, typical instructions to the AI include the followingː
- Context to the AI about the conversationː "You are chatting with a customer who may want to do X, Y, or Z..."
- Explanations of the goal the AI should try to achieveː "Find out what the customer wants and collect the necessary information..."
- A description of how the AI should conduct itselfː "Be polite, ask short questions, maintain a professional tone... "
- Set limitations on what topics to talk aboutː "only talk about X, Y, or Z. Only collect information necessary to solve X, Y, or Z. The only company you represent is Bright Pattern"
- Include a persona for the AIː "Introduce yourself as Diana, a virtual assistant for Bright Pattern customers..."
Writing effective instructions for the AI will require some experimentation; it may be helpful to test your instructions with conversations that try to take the AI "off script". For example, try to get the AI to discuss a competitor, make promises on behalf of your company, make undesirable statements, ask inappropriate questions, or disobey the instructions.
Advanced (System Instruction Prompts)
Clicking the Advanced button opens the System Instruction Prompts dialog, which allows you to view and customize the following:
- System prompt: These are instructions sent along with the main Instructions to AI prompt and are used to control the AI's behavior at a system level.
- Summarization prompt: These instructions, along with the full transcript, are sent to the AI provider in order to generate the conversation summary.
Generate Outcomes
The Generate Outcomes button uses the selected AI Provider to automatically create Requested Outcomes and their associated Output Data based on your Instructions to AI. Clicking this button sends your instructions to the selected AI Provider, which analyzes them and returns a structured list of potential outcomes together with the required data. This provides a way to test how well the AI understands your instructions. Any previously configured outcomes that are not part of the newly generated set will be disabled and displayed in grey, but will remain visible for review or reuse.
Before inserting new outcomes, the system warns you of which current outcomes will be disabled, and lists any scenario variables defined by disabled outcomes.
Requested Outcomes
Defines the outcomes which you would like the AI to pursue with the customer. Each outcome adds a conditional exit to the AI Agent block, allowing you to define different paths for the scenario based on the outcome. For each outcome, you may define any number of associated data points that must be collected for the block to take that exit. The block will only take an exit when the AI chooses that outcome, and all data points for that outcome have been collected.
Each Requested Outcome has the following propertiesː
Name (Requested Outcome)
The name of the outcome and its corresponding conditional exit. The AI considers the name of each Requested Outcome when evaluating the customer response, so it is important to provide names that accurately describe the outcome.
Output Data (Requested Outcome)
The data component of each Requested Outcome can have any number of elements. Each element must have a Data description and Variable name, and optionally, an Example Value.
- Data descriptionː A description of the data in natural language. This text is used to instruct the AI on what specific piece of information to collect from the customer. You can also include formatting instructions (e.g., "The customer's date of birth, formatted as MM/DD/YYYY").
- Variable nameː Name for the scenario variable that the scenario block will populate when it takes the relevant conditional exit.
- Example value: An optional, example value to help the AI better understand
Example Requested Outcomes
Suppose you want to allow customers to change their address with the AI Agent block, and you need the customer's old address and their new address in order to make the change. To define such an outcome, you could create a Requested Outcome with the following propertiesː
- Nameː
Change Address - Output Data:
- Item 1
- Data descriptionː
Customer's previous address, formatted as <...> or <...> - Variable name:
old_address - Example value:
123 Berkshire Ln, Lafayette, CO, 80026, USA
- Data descriptionː
- Item 2
- Data description:
Customer's new address, formatted as <...> or <...> - Variable name:
new_address - Example value:
456 Bogle Way, Houston, TX, 77002, USA
- Data description:
- Item 1
This will create a conditional exit named "Change Address". If the AI Agent block takes this conditional exit, the scenario variables "old_address" and "new_address" will be populated with the data provided by the customer during the dialog.
Include conversation context
Each AI response is generated based on the conversation transcript from the current AI Agent block, the Instructions to AI (including any Advanced instructions), and the Requested Outcomes.
Enable Include conversation context to provide the AI with information from earlier in the scenario. Use the Conversation Context setting (see below) to configure what additional context is provided to the AI.
When disabled (default), no additional context is provided to the AI.
|
|
Including context can improve interaction quality, but typically increases cost and response times from the AI provider. | |
Conversation Context
When Include conversation context is enabled, clicking the view/edit Context link opens the Conversation Context dialog. Here, you can specify what additional information (e.g., a conversation summary or external data) is provided to the AI to maintain continuity across multiple AI Agent blocks.
- Include in context: A dropdown menu that specifies what to provide as conversation context:
- Conversation summary only: Default option. Include only the automatically-generated AI summary from the most recent, successfully-executed, AI Agent block.
- Additional context only: Include only your custom Additional context.
- Conversation summary and additional context: Include both the Additional context and the automatically-generated AI summary.
- Additional context: A text field for providing supplemental information to the AI. You can use any combination of plain text, structured data (like JSON), and Scenario Variables to give the AI more context about the interaction.
Trim final AI response
Controls whether the final message from the AI is sent to the customer when the block exits.
- When enabled (default), the final AI response is trimmed (removed) from the transcript and is not sent to the customer.
- When disabled, the final response is sent to the customer and included in the transcript.
Regardless of whether this option is enabled, the final response from the AI is always stored in the $(aiAgentFinalReply) variable for later use.
Max dialog turns
Enter the maximum number of back-and-forth exchanges (maximum 50) allowed between the AI and the customer before the block exits with an error.
AI Response Timeout
Enter the maximum time (in seconds) the block will wait for the AI provider to respond after receiving the customer's input. If no response is received before this time, the block will exit with an error.
Customer Reply Timeout
Enter the maximum time (in seconds) the block will wait for the customer to provide a response after the AI has presented a prompt. If no response is received, the block will request a new message from the AI provider, while indicating that the customer did not reply to the previous message. The maximum number of repeated attempts to elicit a response from the customer is defined by the Max Consecutive Reply Timeouts setting. The maximum value is 180 seconds.
Max consecutive retry attempts
The maximum number of consecutive times that the block will wait the full Customer Reply Timeout period and try to elicit a response from the customer. Once the maximum number of attempts has been made, the block will take the default path and indicate an error.
Model temperature
This setting controls the creativity of the AI's responses. The minimum value is 0.0 (default), while the maximum value depends on the AI model. Typical models have a maximum temperature of 2.0. Lower values (e.g., 0.1) make the output more deterministic and less creative, which tends to produce predictable and repetitive responses. Higher values (e.g., 0.8 to 1.2) generate more diverse and creative responses, which can make conversations feel more human-like. Note that using higher temperatures increases the risk of the AI producing nonsensical results.
Conditional Exits
The AI Agent block creates conditional exits based on the Requested Outcomes you define. When the AI detects that a specific outcome has been achieved and all the required data for that outcome has been collected, the scenario will follow the corresponding exit path.
Default Exit
The default exit is the path the scenario will follow if the AI Agent block is unable to fulfill the conditions of any of the conditional exits defined by the Requested Outcomes setting. For example, if the customer did not provide the requested information before the maximum number of dialog turns elapsed, or the AI provider returned an error.
The scenario variables aiAgentErrorCode and aiAgentErrorBody, described below, will contain the information needed to determine why the AI Agent block took the default exit.
Variables
The AI Agent block populates several scenario variables to help you understand how the block is operating.
Token Usage Variables
Tokens are the units AI providers use to measure the length of text being processed, with each token typically corresponding to a single word or part of a word. AI providers usually calculate charges based on the number of input tokens (tokens sent to the model) and output tokens (tokens generated by the model) processed during a billing period. The following scenario variables are populated by the AI Agent block during scenario execution:
- aiAgentTotalInTokens contains the cumulative number of tokens sent to the AI provider during scenario execution.
- aiAgentTotalOutTokens contains the cumulative number of tokens received by the AI provider during scenario execution.
Note that these variable values are cumulative during scenario execution (e.g., they are not reset if an AI Agent block is executed multiple times in a single scenario), but you can set them to zero using the Set Variable block.
AI Summary Variable
After each AI Agent block runs, a conversation summary is automatically generated. This summary can be used to provide context to a subsequent AI Agent block or a human agent who continues the interaction after the scenario. The format and content of the summary are configured with the Advanced (System Instruction Prompts) property.
- aiAgentSummary contains the summary from the most recently executed AI Agent block. This variable is reset by each newly executed AI Agent block.
Final Reply Variable
Whether the final AI response is sent to the customer depends on the Trim final AI response setting. Regardless, if a final reply was generated and exists, it is stored in this variable:
- aiAgentFinalReply contains the most recent final reply from the AI provider.
Error Variables
If the AI Agent block is unable to take a conditional exit, the following variables will be populated with information to help you determine the reason:
- aiAgentErrorCode contains an integer indicating the type of error. See the table below for possible error codes and their meaning.
- aiAgentErrorBody contains the full API response from the AI provider.
The following table indicates the possible values of the aiAgentErrorCode scenario variable, and the meaning of each value.
| Code | Description |
| 1000 | AI provider account is either not specified or has been deleted. |
| 1001 | Speech-to-Text account is either not specified or has been deleted. Applicable only to voice scenarios. |
| 1002 | The AI Agent scenario block is not enabled by your service provider. |
| 1003 | Authentication error from the AI provider account, indicating the API key is missing, invalid or expired. |
| 1004 | AI provider account was unable to execute the API request, indicating a problem with connectivity. |
| 1005 | Reached the maximum number of dialog turns (as configured by Max Dialog Turns), but no outcome was selected or insufficient data was received for the selected outcome. |
| 1006 | Reached the maximum number of consecutive turns without receiving a customer reply (as configured by Max consecutive retry attempts). |
| 1007 | The AI provider did not provide any response or results during the most recent turn. |
| 1008 | The transcriber was not able to recognize the customer response. Applicable only to voice scenarios. |
| 2000 | Other error from the AI provider. The entire API response payload is available in the aiAgentErrorBody variable and can be useful for understanding the error reason.
|