Running collections via CLI in Postman - Build an Automation Script

Run Postman Collection using Newman CLI

Preconditions (2)

Step 1: Open the command line interface (CLI).

Step 2: Navigate to the directory containing 'sample_collection.json'.

Step 3: Run the command: newman run sample_collection.json

Step 4: Observe the CLI output for test run results.

✅ Expected Result: The CLI displays the execution summary showing the number of requests run, passed, and failed. All tests in the collection should pass without errors.

Automation Requirements - Newman CLI

Assertions Needed:

Verify that the CLI output contains 'iterations: 1'.

Verify that the CLI output contains 'requests: X' where X is the number of requests in the collection.

Verify that the CLI output contains 'tests: Y' where Y is the total number of tests run.

Verify that the CLI output contains 'failed: 0' indicating no test failures.

Best Practices:

Use explicit command line arguments for collection file path.

Capture CLI output programmatically for assertions.

Use JSON reporter for structured output parsing.

Handle errors gracefully if collection file is missing or invalid.

Automated Solution

Postman

import subprocess
import json

# Path to the Postman collection JSON file
collection_path = 'sample_collection.json'

# Run Newman CLI with JSON reporter for structured output
command = ['newman', 'run', collection_path, '--reporters', 'json']

try:
    result = subprocess.run(command, capture_output=True, text=True, check=True)
    # Parse the JSON output from Newman
    output_json = json.loads(result.stdout)

    # Extract run summary
    run_summary = output_json['run']['stats']

    # Assertions
    assert run_summary['iterations']['total'] == 1, f"Expected 1 iteration, got {run_summary['iterations']['total']}"
    assert run_summary['requests']['total'] >= 1, "Expected at least 1 request"
    assert run_summary['tests']['total'] >= 1, "Expected at least 1 test"
    assert run_summary['tests']['failed'] == 0, f"Expected 0 failed tests, got {run_summary['tests']['failed']}"

    print('Test run passed successfully.')

except subprocess.CalledProcessError as e:
    print(f'Newman CLI execution failed: {e}')
except json.JSONDecodeError:
    print('Failed to parse Newman JSON output.')
except AssertionError as ae:
    print(f'Assertion failed: {ae}')

This script automates running a Postman collection using Newman CLI.

We use Python's subprocess module to run the command newman run sample_collection.json --reporters json. The --reporters json option outputs the results in JSON format, which is easier to parse and assert.

We capture the CLI output and parse it as JSON. Then, we check the run summary statistics:

• There should be exactly 1 iteration.
• At least 1 request should have run.
• At least 1 test should have run.
• No tests should have failed.

If all assertions pass, the test run is successful. If any assertion fails, or if the CLI command fails, we print an error message.

This approach follows best practices by using structured output, explicit command arguments, and proper error handling.

Common Mistakes - 4 Pitfalls

{'mistake': 'Not using the JSON reporter and parsing plain text output.', 'why_bad': 'Plain text output is hard to parse reliably and can break if formatting changes.', 'correct_approach': "Use the '--reporters json' option to get structured output that can be parsed easily."}

Hardcoding the collection path without checking if the file exists.

Not capturing CLI output programmatically and relying on manual inspection.

Ignoring subprocess errors and not handling exceptions.

Bonus Challenge

Now add data-driven testing by running the same collection with three different environment files.

Show Hint

Practice

1. What is the primary command line tool used to run Postman collections via CLI?

easy

A. npm

B. newman

C. curl

D. postman-cli

Solution

1. Step 1: Identify the official CLI tool for Postman collections

   Newman is the official command line tool designed to run Postman collections.

2. Step 2: Differentiate from other tools

   npm is a package manager, curl is a general HTTP client, and postman-cli is not an official tool.

3. Final Answer:

   newman -> Option B

4. Quick Check:

   Postman CLI tool = newman [OK]

Hint: Remember: Newman runs Postman collections in CLI [OK]

Common Mistakes:

• Confusing npm with the CLI runner
• Using curl as a Postman collection runner
• Assuming postman-cli is official

2. Which of the following is the correct syntax to run a Postman collection file named test_collection.json using Newman?

easy

A. newman execute test_collection.json

B. newman start test_collection.json

C. newman run test_collection.json

D. newman launch test_collection.json

Solution

1. Step 1: Recall the Newman command to run collections

   The correct command is newman run <collection-file>.

2. Step 2: Verify the options

   Options like execute, start, or launch are not valid Newman commands.

3. Final Answer:

   newman run test_collection.json -> Option C

4. Quick Check:

   Command syntax = newman run [OK]

Hint: Use 'newman run' to execute collections [OK]

Common Mistakes:

• Using 'execute' instead of 'run'
• Trying 'start' or 'launch' commands
• Omitting the 'run' keyword

3. Given the command newman run my_collection.json -e my_environment.json --delay-request 1000, what effect does the --delay-request 1000 option have?

medium

A. Limits the collection run to 1000 requests total

B. Sets the timeout for each request to 1000 milliseconds

C. Runs the collection 1000 times

D. Delays each request by 1000 milliseconds between executions

Solution

1. Step 1: Understand the --delay-request option

   This option adds a delay between each request in the collection, measured in milliseconds.

2. Step 2: Interpret the value 1000

   1000 milliseconds equals 1 second delay between requests.

3. Final Answer:

   Delays each request by 1000 milliseconds between executions -> Option D

4. Quick Check:

   --delay-request = wait time between requests [OK]

Hint: Delay means wait time between requests in ms [OK]

Common Mistakes:

• Confusing delay with timeout
• Thinking it repeats the collection
• Assuming it limits total requests

4. You run the command newman run collection.json -e env.json --reporters cli,html but get an error saying Unknown reporter: html. What is the most likely cause?

medium

A. The HTML reporter is not installed or supported by Newman

B. The environment file is invalid

C. The collection file is missing

D. The CLI reporter cannot be combined with others

Solution

1. Step 1: Analyze the error message

   The error states 'Unknown reporter: html', meaning Newman does not recognize the 'html' reporter.

2. Step 2: Understand reporter support

   Newman supports some reporters by default; others require installation or are unsupported.

3. Final Answer:

   The HTML reporter is not installed or supported by Newman -> Option A

4. Quick Check:

   Unknown reporter means unsupported or missing plugin [OK]

Hint: Check reporter names and installation before running [OK]

Common Mistakes:

• Blaming environment or collection files
• Assuming CLI reporter conflicts
• Ignoring reporter installation requirements

5. You want to run a Postman collection multiple times with different data sets from a CSV file named data.csv. Which Newman command correctly achieves this?

hard

A. newman run collection.json -d data.csv

B. newman run collection.json --data data.csv

C. newman run collection.json -e data.csv

D. newman run collection.json --env data.csv

Solution

1. Step 1: Identify the option for data files in Newman

   The -d or --iteration-data option is used to specify a data file for iteration.

2. Step 2: Differentiate environment and data options

   -e or --env is for environment files, not data files.

3. Final Answer:

   newman run collection.json -d data.csv -> Option A

4. Quick Check:

   Use -d for data files in Newman [OK]

Hint: Use -d to specify data file for iterations [OK]

Common Mistakes:

• Using -e or --env for data files
• Using --data, an invalid flag
• Confusing environment and data files