Skip to main content

Data-Driven Testing

Welcome to the Data-Driven Testing documentation!

Rumour allows you to parameterize your test suites using external data files. By providing a CSV or JSON file, Rumour will iterate through your entire workflow once for every row (or object) in the dataset, dynamically parameterizing request bodies, URLs, headers, and query parameters with the corresponding row values.

This is extremely powerful for testing boundary conditions, multiple user roles, or bulk ingestion scenarios.

1. Preparing Data Files​

Rumour supports both CSV and JSON arrays for data-driven testing.

CSV Format​

When using a CSV file, the first row must be the header. The column names will become available as variables in your .toml files.

Create a users.csv:

username,email,role
alice,alice@swahira.io,admin
bob,bob@swahira.io,user
charlie,charlie@swahira.io,guest

JSON Format​

When using JSON, the root element must be a flat array of objects.

Create a users.json:

[
  { "username": "alice", "email": "alice@swahira.io", "role": "admin" },
  { "username": "bob", "email": "bob@swahira.io", "role": "user" },
  { "username": "charlie", "email": "charlie@swahira.io", "role": "guest" }
]

2. Parameterizing Requests​

Once your data is prepared, you can reference the column/key names as variables inside your request definitions using the {{variable_name}} syntax.

create_user.toml​

name = "Data Driven Create User"

[request]
method = "POST"
url = "{{base_url}}/users"

[headers]
"Content-Type" = "application/json"

[body]
type = "json"
raw = """
{
  "username": "{{username}}",
  "email": "{{email}}",
  "role": "{{role}}"
}
"""

[assert]
status = 201

3. Running Data-Driven Suites​

To execute a workflow with a dataset, use the --data flag followed by the path to your data file. Rumour automatically detects the file format based on the extension (.csv or .json) and parses it accordingly.

Using a CSV Dataset​

rumour run ./create_user.toml --data ./users.csv -v

Using a JSON Dataset​

rumour run ./create_user.toml --data ./users.json -v

Output Trace​

Rumour automatically loops over the target workflow. To avoid terminal flooding, Rumour displays a live progress bar tracking iteration completion and displays a single consolidated run summary at the end:

$ rumour run ./create_user.toml --data ./users.json

  [00:00:00] ████████████████████████████████████████       3/3       ✓ Completed 3 iterations.

DATA-DRIVEN RUN SUMMARY

╭──────────────────────────────────────────────────────────────────────────╮
│ RUMOUR EXECUTION REPORT                                                  │
├──────────────────────────────────────────────────────────────────────────┤
│ Total Requests:      3                                                   │
│ Successful:          3                                                   │
│ Failed:              0                                                   │
│ Skipped:             0                                                   │
│ Success Rate:        100.0%                                              │
│ Total Time:          30ms                                                │
╰──────────────────────────────────────────────────────────────────────────╯

Actionable Recommendations:
  → Use -v -t to see detailed latency and response diagnostics.
  → Run with --json to export this report for your CI/CD pipeline.

If you run with -v (verbose), the individual successful, failed, or skipped requests are namespaced by their iteration prefix:

$ rumour run ./create_user.toml --data ./users.json -v

...
✓ Successful Requests:
  - Iteration 1: /home/bugsfounder/workspace/testing/advanced_examples/create_user.toml  [200] [8ms]
  - Iteration 2: /home/bugsfounder/workspace/testing/advanced_examples/create_user.toml  [200] [3ms]
  - Iteration 3: /home/bugsfounder/workspace/testing/advanced_examples/create_user.toml  [200] [2ms]

4. Advanced Data-Driven Patterns​

Iterating Over Directories​

Data-driven testing isn't limited to a single file. If you point Rumour to a directory, it will resolve the full dependency graph and execute the entire directory sequentially for the first row (or JSON object), then repeat the entire directory for the second row, and so on.

Example Scenario​

Consider the directory structure under /home/bugsfounder/workspace/testing/advanced_examples/:

advanced_examples/
├── users.json
├── directory_suite/
│   ├── 01_get_status.toml
│   └── 02_create_user.toml

Here, 01_get_status.toml checks request availability:

# directory_suite/01_get_status.toml
name = "Get Status"

[request]
method = "GET"
url = "{{base_url}}/users/1"

[assert]
status = 200

And 02_create_user.toml provisions a user by referencing variables from the dataset:

# directory_suite/02_create_user.toml
name = "Create User"

[request]
method = "POST"
url = "{{base_url}}/users"

[headers]
"Content-Type" = "application/json"

[body]
type = "json"
raw = """
{
  "username": "{{username}}",
  "email": "{{email}}",
  "password": "default_password",
  "role": "{{role}}"
}
"""

[assert]
status = 201

Using the dataset:

// users.json
[
  { "username": "alice", "email": "alice@swahira.io", "role": "admin" },
  { "username": "bob", "email": "bob@swahira.io", "role": "user" },
  { "username": "charlie", "email": "charlie@swahira.io", "role": "guest" }
]

Run the directory suite using either CSV or JSON datasets:

# Iterating a directory using a CSV file
rumour run ./directory_suite/ --data ./users.csv

# Iterating a directory using a JSON file
rumour run ./directory_suite/ --data ./users.json

This will run the entire suite (01_get_status.toml then 02_create_user.toml) sequentially for Alice, then repeat the suite for Bob, then repeat it for Charlie.

Combined with Parallel Execution​

If you have a massive dataset (e.g., 10,000 records) and want to load-test or speed up execution, you can combine --data with the --parallel (-p) flag.

# Fan out 10,000 JSON records concurrently
rumour run ./create_user.toml --data ./massive_users.json -p --concurrency 50

# Fan out 10,000 CSV records concurrently
rumour run ./create_user.toml --data ./massive_users.csv -p --concurrency 50

In this mode, Rumour will fan out the iterations concurrently, respecting the --concurrency limit, radically accelerating your test cycles.

5. Declarative Stress Testing ([stress_test_data])​

Instead of specifying the dataset path imperatively at the CLI using the --data flag, you can define it declaratively inside your request TOML file using the [stress_test_data] block.

Configuration​

Add the [stress_test_data] table with a file field pointing to your dataset (CSV or JSON):

name = "Stress test request"

[request]
method = "POST"
url = "{{base_url}}/users"

[body]
type = "json"
raw = """
{
  "username": "{{username}}",
  "email": "{{email}}",
  "role": "{{role}}"
}
"""

[stress_test_data]
file = "users.csv"

Note: The dataset file path in the file field is resolved relative to the directory containing the request TOML file.

Executing Declarative Stress Tests​

To run the full iterations of the stress test defined in the request TOML file, pass the --stress flag:

rumour run ./create_user.toml --stress

Behavior Differences​

  • With --stress: Rumour automatically loads the dataset from the path specified under [stress_test_data].file and iterates execution runs for each row in the dataset (exactly like passing --data).
  • Without --stress: Running the request normally defaults to single-request execution, automatically utilizing the first row of the dataset for variable resolution (instead of displaying a missing variable error). This allows a single request file to be used seamlessly for both single runs and full stress testing without modification.