pytest-conversational

A pytest plugin for testing chat bots, voice assistants, IVR menus. Rule-based assertions, no LLM dependency.

Status: v1.1.0, released July 2026.

Most chat-bot test setups fall into one of two camps. Either a pile of requests.post calls with hand-rolled assertions, or a heavy framework that pins you to one platform. This plugin sits in the middle: a small Conversation object, a callable bot adapter, and pytest fixtures that wire them together.

You bring the bot. The plugin keeps turn order and per-conversation state, then prints a transcript when an assertion fails.

pip install pytest-conversational

Python 3.10 and above.

def my_bot(text, convo):
    if "hello" in text.lower():
        return "hi"
    return "sorry, did not get that"


def test_greeting(conversation_factory):
    convo = conversation_factory(bot=my_bot)
    convo.say("hello there")
    assert convo.last.bot == "hi"

Adapters can read convo.state and convo.turns to keep slots between turns:

def slot_filling_bot(text, convo):
    slots = convo.state.setdefault("slots", {})
    if "name" not in slots:
        slots["name"] = text
        return "got it, what city?"
    if "city" not in slots:
        slots["city"] = text
        return f"hello {slots['name']} from {slots['city']}"
    return "done"


def test_two_slot_flow(conversation_factory):
    convo = conversation_factory(bot=slot_filling_bot)
    convo.say("Mikhail")
    convo.say("Hove")
    assert convo.state["slots"] == {"name": "Mikhail", "city": "Hove"}
    assert convo.last.bot == "hello Mikhail from Hove"

Repetitive dialogues - the same shape with different inputs (greetings per language, slot variants, fallback phrasings) - load from a YAML or JSON file and run as one test per case. Mark the test with data= and take a scenario argument:

import pytest
from pytest_conversational import expect

@pytest.mark.conversational(data="tests/data/order_status.yaml")
def test_order_status(scenario, scenario_fixtures, conversation_factory):
    convo = conversation_factory(bot=scenario_fixtures["bot"])
    for turn in scenario.turns:
        convo.say(turn.user)
        if turn.expect_contains:
            expect.contains(convo.last.bot, turn.expect_contains)

# tests/data/order_status.yaml
- name: english_path
  fixtures: { bot: english_bot }     # a fixture per case (e.g. a locale bot)
  turns:
    - user: "where is my order?"
      expect_contains: "order number"
- name: russian_path
  fixtures: { bot: russian_bot }
  turns:
    - user: "gde moy zakaz?"
      expect_contains: "nomer zakaza"

Each case becomes its own test, with the scenario name as the pytest id (visible in pytest --collect-only). The file is data, not behaviour: the test keeps control of the assertions, and a case's fixtures overrides resolve to live fixtures through scenario_fixtures. A malformed file (missing name/turns, a non-mapping fixtures) fails collection with a clear message rather than a stack trace.

JSON works out of the box; YAML needs the optional extra:

pip install pytest-conversational[scenarios]

The decorator form @parametrize_scenarios("cases.yaml") is also available for tests that prefer a parametrize-style API over the marker.

If your bot lives behind an HTTP endpoint, use the bundled adapter instead of writing one by hand:

pip install pytest-conversational[http]

from pytest_conversational import Conversation
from pytest_conversational.adapters import http_webhook


def test_remote_bot():
    bot = http_webhook("https://my-bot.example.com/webhook", timeout=3.0)
    convo = Conversation(bot=bot)
    convo.say("hello")
    assert "hi" in convo.last.bot.lower()

The default contract: POST {"user": text, "history": [[u, b], ...]}, expect 200 OK with JSON {"reply": "..."}. If your endpoint speaks a different shape, pass request_builder and response_parser callbacks.

The webhook URL is passed through to httpx as-is. If your test feeds a URL it pulled from user input, fixture data, or another untrusted source, the adapter will happily hit it. That includes internal addresses like 127.0.0.1, 169.254.169.254 (cloud metadata service), or 10.x.x.x inside a VPC. Pin the URL to a hard-coded value in the test, or gate it through your own allowlist before passing it in.

If your bot is a coroutine (a FastAPI handler, an aiogram dispatcher, a LangChain ainvoke wrapper), drive it with say_async instead of say:

import asyncio
from pytest_conversational import Conversation


async def my_async_bot(text, convo):
    reply = await call_my_model(text)
    return reply


def test_async_bot():
    convo = Conversation(bot=my_async_bot)
    asyncio.run(convo.say_async("hello"))
    assert convo.last.bot

say_async keeps the same turn order, history, and partial-transcript-on-failure behaviour as say. Passing an async def bot to the synchronous say() raises a TypeError that points you here, rather than silently storing an un-awaited coroutine as the reply.

expect is a small module of assertion helpers tuned for bot replies. Each matcher raises AssertionError with the actual reply embedded in the message, so pytest output shows what the bot said versus what the test wanted.

from pytest_conversational import expect

def test_replies(conversation_factory):
    convo = conversation_factory(bot=my_bot)
    convo.say("hi")

    expect.contains(convo.last.bot, "hello")
    expect.not_contains(convo.last.bot, "error")
    expect.regex(convo.last.bot, r"^hello\s+\w+")
    expect.one_of(convo.last.bot, ["hello there", "hi there", "hey"])

    convo.say("to Brighton")
    expect.said_in_order(convo, ["your name", "where to", "confirmed"])

• contains(actual, substring, *, case_sensitive=False): substring search. Case-insensitive by default.
• not_contains(actual, substring, *, case_sensitive=False): the negative of contains. Guards against leaks, for example a bot echoing an internal error, a stack trace, or a value it was never given.
• regex(actual, pattern, *, flags=0): re.search semantics. Returns the match object so callers can inspect captured groups.
• one_of(actual, options, *, case_sensitive=False, mode="exact"): matches actual against a list of alternative options. Supports mode="exact" (full-string match, default) and mode="substring" (checks if any option is a substring of actual).

The next four matchers inspect metadata your adapter records rather than the reply text. They check what the adapter wrote; they do not classify, extract, or time anything themselves.

• has_intent(turn, intent_name): asserts turn.metadata["intent"] equals intent_name.
• has_slot(turn, slot_name, value=...): asserts slot_name is present in turn.metadata["slots"]. Pass value= to also assert the stored value.
• has_state(convo, state_name, value=...): asserts state_name is present in convo.state, the conversation-wide dict that persists across turns. Pass value= for equality.
• responds_within(turn, seconds): asserts turn.metadata["latency_ms"] is within the seconds budget (converted to milliseconds).

said_in_order reads the whole conversation instead of a single reply:

• said_in_order(convo, phrases, *, case_sensitive=False): asserts the bot's replies contain phrases in order across turns. Each phrase must appear at or after where the previous match ended, so matches are non-overlapping and two phrases share a reply only when the later one starts after the earlier one ends. Use it for wizard-style flows where the order of prompts is the contract, without pinning each prompt to a fixed turn index. Substring match, case-insensitive by default.

Use these when bare assert "hello" in convo.last.bot would give noisy failure messages across many tests. For one-off checks, plain assert is still fine.

• Conversation(bot=None, turns=[], state={})
• Conversation.say(text): drive a turn through the adapter, return the Turn.
• Conversation.say_async(text): await a coroutine adapter, return the Turn.
• Conversation.add_user(text): append a user-only turn.
• Conversation.last, .turns, .history, .transcript().
• Turn(user, bot, metadata).
• BotAdapter = Callable[[str, Conversation], str].
• expect.contains, expect.not_contains, expect.regex, expect.one_of.

Full shipped history lives in CHANGELOG.md. Recent milestones:

• v1.1 (2026-07-25): data-driven scenarios via @pytest.mark.conversational(data="cases.yaml") with per-case fixture overrides, and say_async for coroutine bot adapters.
• v1.0 (2026-06-12): first stable release. Scenario loading from JSON or YAML, not_contains matcher, Allure transcript attachments.
• v0.4 (2026-05-23): slot, state, intent, and latency matchers, plus HTTP webhook adapter hardening (allowed_hosts, reply size guard).

On main, not yet in a release:

• said_in_order matcher: assert the bot's replies contain a set of phrases in order across turns, for wizard-style flows where prompt order is the contract.

Direction, not a schedule. What gets built first depends on what real users report.

• Adapters for real bot stacks: an HTTP webhook adapter ships today. A Telegram adapter and a voice or IVR surrogate (Twilio-shaped) would remove the glue a team writes by hand before the first test.
• Transcript diff on failure: the transcript already prints when an assert fails; next is marking the exact mismatch inside it and saving the run as an HTML artifact, so a red test reads as a diagnosis.
• Composable intent and slot helpers: the intent and slot matchers cover the primitives; a few helpers on top would fold in the checks people still write by hand.

Contributions welcome. See CONTRIBUTING.md for setup and the PR workflow. A couple of good first issue slots are open in the issue tracker if you want to jump in.

MIT. See LICENSE.