Running collections via CLI in Postman - Test Execution Trace

Test Overview

This test runs a Postman collection using the CLI tool Newman. It verifies that the collection executes successfully and all tests inside pass.

Test Code - Python subprocess with Newman CLI

Postman

import subprocess

# Run Postman collection using Newman CLI
result = subprocess.run([
    'newman', 'run', 'sample_collection.json',
    '--reporters', 'cli',
    '--timeout-request', '5000'
], capture_output=True, text=True)

# Check if the run was successful by looking for 'run complete' in output
assert 'run complete' in result.stdout.lower(), 'Collection run did not complete successfully'

# Check that there are no failed tests reported
assert 'failed: 0' in result.stdout.lower(), 'Some tests failed in the collection run'

Execution Trace - 6 Steps

Step	Action	System State	Assertion	Result
1	Test starts	Test script is ready to run the Newman CLI command	—	PASS
2	Runs 'newman run sample_collection.json --reporters cli --timeout-request 5000' via subprocess	Newman CLI starts executing the Postman collection	—	PASS
3	Newman executes all requests and tests inside the collection	CLI output shows request results and test assertions	—	PASS
4	Test script checks if 'run complete' is in the CLI output	CLI output contains summary of the run	'run complete' found in output	PASS
5	Test script checks if 'failed: 0' is in the CLI output	CLI output shows zero failed tests	'failed: 0' found in output	PASS
6	Test ends with all assertions passed	Collection run successful with no test failures	—	PASS

Failure Scenario

Failing Condition: Newman CLI run does not complete or tests fail in the collection

Execution Trace Quiz - 3 Questions

What does the test check to confirm the collection ran successfully?

A Presence of 'run complete' in CLI output

B Presence of 'error' in CLI output

C Number of requests in the collection

D Size of the collection file

Key Result

Always verify both the completion status and the test results when running collections via CLI to ensure the entire suite executed successfully without failures.

Practice

1. What is the primary command line tool used to run Postman collections via CLI?

easy

A. npm

B. newman

C. curl

D. postman-cli

Solution

1. Step 1: Identify the official CLI tool for Postman collections

   Newman is the official command line tool designed to run Postman collections.

2. Step 2: Differentiate from other tools

   npm is a package manager, curl is a general HTTP client, and postman-cli is not an official tool.

3. Final Answer:

   newman -> Option B

4. Quick Check:

   Postman CLI tool = newman [OK]

Hint: Remember: Newman runs Postman collections in CLI [OK]

Common Mistakes:

• Confusing npm with the CLI runner
• Using curl as a Postman collection runner
• Assuming postman-cli is official

2. Which of the following is the correct syntax to run a Postman collection file named test_collection.json using Newman?

easy

A. newman execute test_collection.json

B. newman start test_collection.json

C. newman run test_collection.json

D. newman launch test_collection.json

Solution

1. Step 1: Recall the Newman command to run collections

   The correct command is newman run <collection-file>.

2. Step 2: Verify the options

   Options like execute, start, or launch are not valid Newman commands.

3. Final Answer:

   newman run test_collection.json -> Option C

4. Quick Check:

   Command syntax = newman run [OK]

Hint: Use 'newman run' to execute collections [OK]

Common Mistakes:

• Using 'execute' instead of 'run'
• Trying 'start' or 'launch' commands
• Omitting the 'run' keyword

3. Given the command newman run my_collection.json -e my_environment.json --delay-request 1000, what effect does the --delay-request 1000 option have?

medium

A. Limits the collection run to 1000 requests total

B. Sets the timeout for each request to 1000 milliseconds

C. Runs the collection 1000 times

D. Delays each request by 1000 milliseconds between executions

Solution

1. Step 1: Understand the --delay-request option

   This option adds a delay between each request in the collection, measured in milliseconds.

2. Step 2: Interpret the value 1000

   1000 milliseconds equals 1 second delay between requests.

3. Final Answer:

   Delays each request by 1000 milliseconds between executions -> Option D

4. Quick Check:

   --delay-request = wait time between requests [OK]

Hint: Delay means wait time between requests in ms [OK]

Common Mistakes:

• Confusing delay with timeout
• Thinking it repeats the collection
• Assuming it limits total requests

4. You run the command newman run collection.json -e env.json --reporters cli,html but get an error saying Unknown reporter: html. What is the most likely cause?

medium

A. The HTML reporter is not installed or supported by Newman

B. The environment file is invalid

C. The collection file is missing

D. The CLI reporter cannot be combined with others

Solution

1. Step 1: Analyze the error message

   The error states 'Unknown reporter: html', meaning Newman does not recognize the 'html' reporter.

2. Step 2: Understand reporter support

   Newman supports some reporters by default; others require installation or are unsupported.

3. Final Answer:

   The HTML reporter is not installed or supported by Newman -> Option A

4. Quick Check:

   Unknown reporter means unsupported or missing plugin [OK]

Hint: Check reporter names and installation before running [OK]

Common Mistakes:

• Blaming environment or collection files
• Assuming CLI reporter conflicts
• Ignoring reporter installation requirements

5. You want to run a Postman collection multiple times with different data sets from a CSV file named data.csv. Which Newman command correctly achieves this?

hard

A. newman run collection.json -d data.csv

B. newman run collection.json --data data.csv

C. newman run collection.json -e data.csv

D. newman run collection.json --env data.csv

Solution

1. Step 1: Identify the option for data files in Newman

   The -d or --iteration-data option is used to specify a data file for iteration.

2. Step 2: Differentiate environment and data options

   -e or --env is for environment files, not data files.

3. Final Answer:

   newman run collection.json -d data.csv -> Option A

4. Quick Check:

   Use -d for data files in Newman [OK]

Hint: Use -d to specify data file for iterations [OK]

Common Mistakes:

• Using -e or --env for data files
• Using --data, an invalid flag
• Confusing environment and data files