Agent Testing

While unit tests for your intents.Agent and intents.Intent classes can be defined as in any other Python project, testing conversation flows may require simulating multi-turn interactions within a session, and check the Agent understanding as the interaction proceeds.

This testing module provides the following abstractions to facilitate Agent testing through conversational stories:

  • AgentTester runs in the background and keeps a dev fulfillment service running.

  • TestingStory wraps a single conversation and models its steps and their assertions.

Setup

Testing frameworks can be configured to automatically instantiate these two objects in the context of a test run. In pytest this can be done by defining fixtures in conftest.py (in the example we use the internal CLI controller to read the Agent configuration from env)

import pytest

from intents.cli.agent_controller import AgentController
from intents.testing import AgentTester

@pytest.fixture(scope="session")
def agent_tester() -> AgentTester:
    controller = AgentController.from_env()
    return controller.load_agent_tester()

@pytest.fixture(scope="function")
def story(agent_tester):
    return agent_tester.story()

This will instruct pytest to inject a TestingStory instance in each test function, that can be used to easily assert on single conversation steps:

def test_order_fish_kipper_followup(story: TestingStory):
    story.step("I want a fish", OrderFish())
    story.step("kipper", OrderFishAnswerKipper())
    story.step("make it three", ChangeAmount(3))

Running a test campaign

An example of this setup can be found in /example_agent/test_stories/. From there, tests can be launched as follows:

  1. Spawn a ngrok (https://ngrok.com/) process to expose the fulfillment server globally (Dialogflow wil call it as an https://... endpoint):

    >>> ngrok start 8000

  2. Create a .env configuration from .env.template. Make sure use the same address and port as your ngrok tunnel

  3. Upload your agent to make sure that the cloud prediction service (e.g. Dialogflow) is aligned with your code.

    >>> dotenv run agentctl upload

  4. Wait until the cloud agent completes its training after the upload, and then run the test campaign as any other pytest suite:

    >>> poetry shell
>>> dotenv run pytest -vvv
class AgentTester(connector, steps_wait=0.5, dev_server=True, dev_server_port=8000, dev_server_token=None)[source]

This is meant to be instantiated at the beginning of a test campaign run.

It will automatically spawn a dev server in the background and will serve as a factory for TestingStory objects throughout the campaign. It also checks that the specified connector implements the intents.connectors.interface.testing.TestableConnector interface (which is required to make assertions on fulfillment internals)

Parameters

  • connector (Connector) – The connector that will be used to spawn the fulfillment server

  • steps_wait (float) – Waiting time (seconds) in between steps

  • dev_server (bool) – Set to false if you want to spawn your dev server manually

  • dev_server_port (int) – Dev server will run on this port

  • dev_server_token (Optional[str]) – Dev server will check this for authentication

ensure_server()[source]

Spawn a background thread running a development fulfillment server with intents.fulfillment.run_dev_server().

story()[source]

Returns a TestingStory object that is configured to run on the same connector as self.

class FulfillmentAssert(name=None, param_dict=<factory>)[source]
Parameters

  • name (Optional[str]) –

  • param_dict (dict) –

class IntentAssert(name=None, param_dict=<factory>)[source]
Parameters

  • name (Optional[str]) –

  • param_dict (dict) –

class PredictionAssert(name=None, param_dict=<factory>)[source]
Parameters

  • name (Optional[str]) –

  • param_dict (dict) –

class TestingStory(connector, steps_wait, session_id=<factory>)[source]

This class models a test conversation, where each step bears the context of the prevous ones, and can define intent assertions.

Each testing story runs in an isolated session, to prevent contexts to alter predictions across different tests.

Parameters

  • connector (Connector) – A Connector object that will be used for predictions

  • steps_wait (float) – Waiting time (seconds) in between steps

  • session_id (str) – An id for the session. If unset, a random one will be generated as intents-test-<UUID>

step(utterance_or_step, prediction_assert=None, fulfill_asserts=None)[source]

Adds a step to the story. This will run the utterance prediction and, if set, perform assertions on the result as well as on its fulfillment call.

Parameters

  • utterance_or_step (Union[str, TestingStoryStep]) – Typically, this is the message we send to the Agent

  • prediction_assert (Union[PredictionAssert, Intent, None]) – An assertion on the prediction result. This can simply be the expected Intent; in this case, an assertion will be built to check that the result matches the expectation.

  • fulfill_asserts (Union[FulfillmentAssert, Intent, Iterable[Union[FulfillmentAssert, Intent]], None]) – An utterance may require one or more fulfillment calls to get to the result. These are assertions on the sequence of fulfillment calls.

class TestingStoryStep(utterance, prediction_assert, fulfillment_asserts)[source]
Parameters