Testing a Runbook: A Beginner's Guide

1. Introduction

A runbook is an automated workflow. Something happens, and the runbook reacts by doing a few steps in order. Picture a factory line: a package arrives, and each station does one job to it before passing it down the line.

Two words show up everywhere:

• A trigger is what starts the runbook. For example, "another app sent us some data over the web." The trigger catches that event and turns it into a tidy bundle of data called the trigger output.
• An action is one step the runbook performs, usually a call to another system like "create a ticket" or "send an HTTP request." Each action takes some input and produces some output. Later actions can read the trigger output and earlier actions' output.

A test lets you run the whole runbook without touching the real world. No real tickets get created, no real emails go out. You hand the runbook fake data, then check it did the right thing.

Hold onto one idea:

Fake the outside. Check the inside.
Replace anything that talks to the real world (the trigger event, and any action that calls another system) with fake data you supply. That fake data is a mock. Then confirm the runbook produced the right values. That confirmation is an expectation.

2. The example we'll use

Picture a small runbook called "New deal to ticket":

• Its trigger is JSON endpoint. When your sales app closes a deal, it sends the runbook a short JSON message like:

  { "deal_name": "Acme renewal", "amount": 5000 }

• It has one action, create-ticket, which calls a ticketing system over HTTP to open a ticket for that deal.

We'll write a test that feeds in a fake deal and confirms the runbook tries to create the right ticket. Everything below uses this one example.

3. The easy way: let the platform write the test for you

You almost never need to write a test from scratch. The smoothest path is to run the runbook(Enqueue the job in Factory) once for real, then turn that run into a test. Here is the whole flow.

Step 1. Get one real run. Trigger the runbook once the normal way, or find a run that already happened. Each run shows up as a job.

Step 2. Open Factory. Factory is the screen where you see your runbook and its jobs.

Step 3. Add the job to the dock. In the jobs list, find your run and click Add to dock. The dock is a tray that holds the jobs you're working with.

Step 4. Hover over the docked job. Two small icons appear:

• a beaker icon, labeled Extract test case
• a truck icon, labeled Ship again (that one just re-runs the job, so ignore it for testing)

Step 5. Click the beaker (Extract test case). The platform reads what happened in that run and writes a complete test for you:

• the trigger's output becomes a mock, so the test feeds the same data in without waiting for a real web request,
• each action's input becomes a check (an expectation that the action was called with those values),
• each action's output becomes a mock if the action connects to an outside network, or a check if it doesn't.

You now have a working test with every value filled in correctly. This is also the best way to learn the format, because you can read what it produced.

One thing to fix afterward: timestamps and other values that change every run. This is the most common reason an extracted test passes the first time and then fails later, so it's worth understanding before you trust an extracted test.

The problem: Extract writes every value it captured as an exact-match check (matcher: equals). That works for steady values like a name or an id, but not for a timestamp the runbook generates while it runs. Say an action sets a field to the current time (Time.now). On the real run that produced the test, the time was 2025-04-18T11:22:15.123Z, so Extract freezes the check as equals 2025-04-18T11:22:15.123Z. The next time you run the test, the action computes a new time a few milliseconds later, the equals check no longer matches that exact value, and the test fails even though nothing is actually wrong. The action's time is never the same millisecond twice, so an exact check can't pass on a later run. Anything else generated fresh each run, like a random id, breaks the same way.

The solution: after extracting, find each check that holds a timestamp (or other per-run value) and stop comparing the exact value. Pick one:

• change matcher: equals to is_present, which just confirms the field has a value,
• use starts_with on the stable part, like the date 2025-04-18,
• use a custom proc that checks the shape instead of the value (for example, that it parses as a time), or
• delete the check if that field doesn't matter to the test.

These matchers (is_present, starts_with, custom) are all explained in Section 6; if they're new to you, read that section first and come back here.

One nuance decides which fix you need. If the timestamp comes straight from the trigger's data, you can instead pin it in the trigger mock as a fixed: value: every run then replays that exact value, so downstream checks stay stable. But if the runbook computes the time itself (Time.now), no mock can pin it, so loosening the check is the only fix.

(The "Generated from job … 2025-04-18T11:22:15Z" caption shown on the test is only a label recording where it came from; it has no effect on runs.)

4. Reading the test it generated

After Extract, open the test in the Quality Assurance tab. You'll see the runbook's steps as an outline (the trigger, then each action). Click a step and its test shows in its own panel as a readable summary; click the pencil (edit) on that panel to open the YAML editor, which is what the blocks below show. You read and edit each step on its own, never as one combined document. Here are the two pieces for our example.

The trigger panel holds the fake event, as a mocked_output list:

mocked_output:
  - field_id: body
    fixed:
      deal_name: "Acme renewal"
      amount: 5000

The trigger's body field holds our fake deal. (The field_id / fixed pattern is explained in the next section.)

The create-ticket action panel holds that step's input check and mocked result:

reference: create-ticket
iterations:
  - input_expectations:
      - field_id: method
        matcher: equals
        fixed: POST
    mocked_outputs:
      - output:
          - field_id: response
            fixed:
              status: 201

Line by line:

• reference is the action's id inside the runbook; this panel is scoped to that one action.
• iterations holds one entry per time the action runs. A normal action runs once, so there is one entry. (An action inside a loop runs several times, so it would have several entries.)
• input_expectations is a check: "the action should have been called with method: POST."
• mocked_outputs is the fake result: "pretend the ticket API replied with status 201." Because create-ticket connects to an outside network, the test fakes its result instead of really calling it. This action returns a single output shape, so the mock needs no schema_reference (explained in Section 8).

5. Filling in values manually

When you edit a test yourself, you type the values. This is where people get stuck.

A mock is always a list. Every entry starts with - field_id: (the name of the field), and the value goes under a key called fixed:. fixed: isn't only for text: it takes whatever shape the field needs, including a number, true/false, a list, or a whole object with its own nested fields (like the ratelimit object in Section 8).

mocked_output:
  - field_id: body
    fixed:
      deal_name: "Acme renewal"
      amount: 5000

Three rules that save you pain:

1. The value key is fixed:. Not value:, not sample:, not anything else. If you use a different key, the platform silently throws it away when you save. Your value just disappears with no error message. This is the single most common mistake.
2. It must be a list. Each field starts with - field_id:. If you write a plain object instead, the save is rejected with a clear message: "mocked_output must be a list. Start each field with '- field_id:'."
3. A field with no value comes out empty (null). If you see fields with no values, you left off the fixed:.

When fixed: isn't enough

fixed: is a static value. Sometimes you want something else. There are four more options, and you use exactly one of them on a field, in place of fixed:.

proc: is a computed value. Write a small piece of Ruby that runs when the test runs. Good for a fresh timestamp, or a value built from other values.

- field_id: received_at
  proc: 'Time.now.iso8601'

A proc runs in a safe sandbox. You get the everyday building blocks (text, numbers, lists, dates and times, base64, hashing), but not file, network, or system access. If you call something outside the allowed set, you'll see an error like "Method '...' not allowed." (it names the method you used) For the full list of what a proc can use, see the Ruby connector docs: https://www.xurrent.com/help/ruby-connector

nested: fills an object one sub-field at a time. If a field is an object with its own sub-fields, you can set each sub-field separately. Useful when most of the object is static but one piece needs a proc:.

- field_id: body
  nested:
    - field_id: deal_name
      fixed: "Acme renewal"
    - field_id: received_at
      proc: 'Time.now.iso8601'

variable: reads a solution environment variable. Give the variable's name.

- field_id: region
  variable: DEFAULT_REGION

runbook_variable: reads a runbook variable. Give the variable's name.

- field_id: tenant_id
  runbook_variable: current_tenant

Use one key per field. If you set more than one, fixed: wins.

6. Checking results instead of faking them

So far we've faked data with mocks. The other half of testing is checking that the runbook produced the right values. That's an expectation.

You already read about input_expectations (check what an action received). There's also expected_outputs (check what an action produced). Each check names a field and a way to compare it:

input_expectations:
  - field_id: method
    matcher: equals      # how to compare
    fixed: POST          # what you expect

Checking an action's output uses the same kind of check, but the checks go under expected_outputs, grouped inside an expectations: list (use this for an action that doesn't connect to an outside network):

expected_outputs:
  - expectations:
      - field_id: priority
        matcher: equals
        fixed: high

The matcher: is how the comparison works. Leave it out and it defaults to equals.

equals (the default): passes when the value is exactly what you give.

- field_id: amount
  matcher: equals
  fixed: 5000

contains: for text; passes when the value contains this substring.

- field_id: deal_name
  matcher: contains
  fixed: "renewal"

includes: for a list or object; passes when it contains the item(s) you give.

- field_id: tags
  matcher: includes
  fixed: ["urgent"]

starts_with / ends_with: for text; passes when it starts (or ends) with this.

- field_id: deal_name
  matcher: starts_with
  fixed: "Acme"

is_present: passes when the field has a non-empty value (an empty string or empty list doesn't count). No fixed: needed.

- field_id: ticket_id
  matcher: is_present

custom: your own rule, written as a proc:. Inside the proc, actual_value is a built-in keyword that holds the value being checked, which is the field you put the check on (here, amount). The check passes when the proc returns true.

- field_id: amount
  matcher: custom
  proc: 'actual_value > 1000'

nested: check several sub-fields of an object at once. Each sub-field is its own check.

- field_id: response
  matcher: nested
  nested:
    - field_id: status
      matcher: equals
      fixed: 201
    - field_id: ticket_id
      matcher: is_present
    - field_id: priority
      matcher: equals
      fixed: high

You only check the sub-fields you list. If response has many other keys, that's fine: unlisted keys are ignored, so check just the ones that matter.

negated: You can flip any check with negated: true ("should NOT equal..."), and add a failure_message: to explain a failure in plain words.

- field_id: method
  matcher: equals
  fixed: GET
  negated: true                    # passes when method is NOT GET
  failure_message: "create-ticket must not use GET"

One note to remember: for a given action output you either mock it or check it, never both. The platform rejects a test that tries to do both on the same output.

When do you mock vs check?

• An action that connects to an outside network (an HTTP call, a ticketing API): mock its output, so the test stays offline.
• An action that doesn't connect to an outside network: check its output, to confirm the logic.

Actions that run more than once

Most actions run a single time, so their panel has one entry under iterations:. An action runs several times in two cases, and then you add one iterations: entry per run:

• it sits inside a loop (it runs once per item), or
• it paginates (a query that fetches one page at a time runs once per page).

For those, you can also mock the state the action carries from one run to the next (for a paginated query, the page cursor) under mocked_iteration_state:, and check it with iteration_state_expectations:. Section 8 shows a paginated query with two iterations and an end_cursor carried between them.

7. Running the test and reading the result

Tests live in the Quality Assurance tab of your runbook.

• Run one test: open it and click Run Test Case.
• Run all tests: click Run All Tests.

A test passes when every check matches and the runbook ran exactly the steps the test expected. If it runs extra steps or skips some, the test fails and says so. Every individual check is logged, so a failure shows you the field, the value it got, and the value it expected.

8. A complete example: a Xurrent query

Everything so far used one example, but the steps are the same for every action in every connector. Only the field names change, and Extract test case (Section 3) fills in the exact fields for whatever actions your runbook uses. Here is a real runbook that queries Xurrent, to show the method carries over.

Important: you don't write one big test document. In the QA tab each step has its own panel, and you write its test there. The trigger panel holds only mocked_output:. Each action panel holds only reference: and iterations: for that one action. So the example below is split the same way: an overview of the runbook, then the YAML for each step.

The runbook ("Log all service instances"). In the QA tab the steps show as an outline:

• Start — the trigger (a JSON Endpoint)
• 1. Retrieve service instances — a Xurrent query that fetches records
• 2. Log total count
• 3. If instances found
• 4. Each service instance — a loop
• 5. Log name

You only need to write a test for the steps you care about. Two things to know about the Xurrent query step:

• It connects to an outside network (it calls Xurrent), so you mock its output.
• It paginates: it fetches one page of records at a time, so the action runs once per page. Each page is one iteration, and end_cursor (mocked iteration state) is the marker carried into the next page.

The trigger panel

This runbook doesn't read anything from the trigger, so the mock just names the field and leaves it empty:

mocked_output:
  - field_id: body

The "retrieve-service-instances" action panel

The query returned two pages, so the action has two iterations. Each one checks the inputs the action was called with, then mocks that page of results:

reference: retrieve-service-instances
iterations:
  - input_expectations:
      - field_id: connection
        matcher: equals
        fixed: serviceInstances
      - field_id: page_size
        matcher: equals
        fixed: 100
    mocked_outputs:
      - schema_reference: page
        output:
          - field_id: total_count
            fixed: 2
          - field_id: has_next_page
            fixed: true
          - field_id: nodes
            fixed:
              - id: id1
                name: service 1
    mocked_iteration_state:
      - field_id: end_cursor
        fixed: theStart
  - input_expectations:
      - field_id: connection
        matcher: equals
        fixed: serviceInstances
      - field_id: page_size
        matcher: equals
        fixed: 100
    mocked_outputs:
      - schema_reference: page
        output:
          - field_id: total_count
            fixed: 2
          - field_id: has_next_page
            fixed: false
          - field_id: nodes
            fixed:
              - id: id2
                name: service 2

What each part does:

• reference: retrieve-service-instances — the action panel is always scoped to one action by its reference. The trigger lives in its own panel (above), which is why neither block wraps the other.
• iterations — one entry per page the query fetched. Page 1 has has_next_page: true, page 2 has false, so the action stops after two pages.
• input_expectations — confirm the action was called correctly: it queried serviceInstances with page_size: 100. (connection is the field that picks what to query.)
• mocked_outputs — the fake result for that page; the action calls Xurrent, so you mock it instead of calling out. nodes holds that page's records.
• schema_reference — some actions can return more than one shape of output, and each shape is a named output schema. schema_reference says which shape this mock is for. Its value is that schema's name: here it's page, and the panel even labels it "Schema: page". You don't guess the name; Extract fills it in, and the structured view shows it. An action with a single output, like create-ticket in Section 4, doesn't use schema_reference at all.
• mocked_iteration_state — when an action runs several times, it can carry state from one run into the next; this sets that carried-in state for the run. Its fields come from the action's iteration-state schema, here end_cursor (the marker the next page starts from). An action that runs once doesn't have it. To check the carried state instead of faking it, use iteration_state_expectations.

Two things to keep in mind:

• The field names (connection, nodes, total_count, end_cursor) come from this action's schema. A different Xurrent action, or a different connector, has different fields; Extract test case fills in the exact ones for whatever your runbook uses.
• You don't have to mock every field. This action also returns request_id, ratelimit, and costlimit; they're left out here because nothing downstream reads them. Mock only what you need.

9. Troubleshooting

• My values vanished when I saved. You used a key other than fixed:. Move the value under fixed:.
• "mocked_output must be a list" when I save. Your mocked_output is a plain object, not a list. Make every field start with - field_id:.
• All my fields show up empty (null). Those fields have no fixed: (or proc: / nested:). Add a value.
• "Method '...' not allowed" inside a proc. You called something the sandbox doesn't permit, like file or network access. Stick to text, number, list, date and time, base64, and hashing operations. The full list is in the Ruby connector docs: https://www.xurrent.com/help/ruby-connector
• "Cannot set expectations on mocked output." You put both a mock and a check on the same action output. Keep one.
• A test passed right after Extract but fails later. It's checking a value that changes between runs, almost always a timestamp (or a generated id): Extract froze that run's exact value, down to the millisecond, as an equals check, and a later run produces a different one. Loosen the check: is_present, starts_with the stable part, a custom proc, or delete it. Full explanation and the pin-the-mock alternative are in Section 3.

10. Quick reference

The value keys you can put on a field (pick one):

The matchers you can use in a check (default is equals): equals, contains, includes, starts_with, ends_with, is_present, nested, custom.

Inside an action panel, each iterations: entry can hold:

One iterations: entry per run: one for a normal action, one per item for a loop, one per page for a paginated query.