Test a Python CLI through its real interface

1. Define a small command contract

The command accepts a UTF-8 JSON file containing a list of nonnegative integer quantities. On success, stdout is one JSON object with count and total, stderr is empty and the exit code is zero.

Bad input produces a diagnostic on stderr, no result on stdout and exit code two through argparse. Booleans are rejected explicitly even though bool is an int subclass in Python. An empty list is valid and totals zero.

2. Implement the executable boundary

Save this file as sum_quantities.py. Keep main behind the usual module guard so importing the file does not execute argument parsing. The example does not write to the input file or require a network service.

python · EXAMPLE

import argparse
import json
from pathlib import Path


def main():
    parser = argparse.ArgumentParser(description="Sum nonnegative integer quantities")
    parser.add_argument("file", type=Path)
    args = parser.parse_args()
    try:
        values = json.loads(args.file.read_text(encoding="utf-8"))
    except (OSError, UnicodeError, json.JSONDecodeError) as error:
        parser.error(f"Cannot read quantities: {type(error).__name__}")
    if not isinstance(values, list) or any(
        type(value) is not int or value < 0 for value in values
    ):
        parser.error("Expected a JSON array of nonnegative integers")
    print(json.dumps({"count": len(values), "total": sum(values)}))


if __name__ == "__main__":
    main()

3. Add subprocess tests

Save this file beside it as test_cli.py. The tests use sys.executable so the child command runs under the same Python interpreter as the test runner. Temporary directories provide input files without altering a real repository’s data.

The per-command timeout fails a stuck CLI rather than hanging the test suite. Assertions cover parsed successful output, empty input, rejected types, malformed JSON, missing files and help.

python · EXAMPLE

import json
from pathlib import Path
import subprocess
import sys
import tempfile
import unittest

CLI = Path(__file__).with_name("sum_quantities.py")


class CliTests(unittest.TestCase):
    def invoke(self, *args):
        return subprocess.run(
            [sys.executable, str(CLI), *map(str, args)],
            capture_output=True, text=True, timeout=5,
        )

    def test_success_and_empty_input(self):
        for values, expected in [([2, 3, 0], {"count": 3, "total": 5}),
                                 ([], {"count": 0, "total": 0})]:
            with self.subTest(values=values), tempfile.TemporaryDirectory() as folder:
                file = Path(folder) / "quantities.json"
                file.write_text(json.dumps(values), encoding="utf-8")
                result = self.invoke(file)
                self.assertEqual(result.returncode, 0)
                self.assertEqual(result.stderr, "")
                self.assertEqual(json.loads(result.stdout), expected)

    def test_invalid_input(self):
        for raw in ["{", "{}", "[true]", "[-1]", '["2"]']:
            with self.subTest(raw=raw), tempfile.TemporaryDirectory() as folder:
                file = Path(folder) / "quantities.json"
                file.write_text(raw, encoding="utf-8")
                result = self.invoke(file)
                self.assertEqual(result.returncode, 2)
                self.assertEqual(result.stdout, "")
                self.assertIn("error:", result.stderr)

    def test_missing_file(self):
        with tempfile.TemporaryDirectory() as folder:
            result = self.invoke(Path(folder) / "absent.json")
            self.assertEqual(result.returncode, 2)
            self.assertEqual(result.stdout, "")
            self.assertIn("FileNotFoundError", result.stderr)

    def test_help(self):
        result = self.invoke("--help")
        self.assertEqual(result.returncode, 0)
        self.assertIn("usage:", result.stdout)
        self.assertEqual(result.stderr, "")


if __name__ == "__main__":
    unittest.main(verbosity=2)

4. Run in an isolated interpreter

From the folder containing the two files, create a disposable virtual environment and run unittest discovery. No third-party package is needed, so the environment can be created without pip. These commands assume a Unix-like shell.

Use an available Python 3.12 interpreter for this tested baseline. Replace python3.12 with the explicit path to that interpreter if it is not on your PATH. Do not install anything into the system interpreter for this exercise.

bash · EXAMPLE

python3.12 -m venv --without-pip .venv
.venv/bin/python -m unittest discover -s . -p test_cli.py -v

5. Read the result as a contract

The suite has four test methods, with multiple successful and invalid-input subcases. A passing suite confirms that this command preserves machine-readable stdout and a meaningful exit status for the cases listed.

Add a regression case when a real input exposes a bug. For example, file encoding, a changed JSON schema or a missing argument is a new boundary to specify deliberately, rather than a reason to swallow every exception and return a successful exit.

Verification boundary

The editorial run used a fresh Python 3.12 virtual environment in an isolated container. All four test methods and their subcases passed, including missing-file and invalid-input failures. This did not exercise a published Python package or test a production dataset.

Know the limits

This Python 3.12 example uses only the standard library. It demonstrates a CLI contract, not packaging, dependency locking or a general-purpose data validator.

Sources & further reading

A technical publication by CoreTrailKit.

Browse all articles