Test-Driven Development in Python with Pytest

How can you write tests using pytest? How does pytest work? The questions seem simple, but while teaching Python workshops, I saw many people confused by pytest.

We will start with an empty directory. All you need is pytest installed in your working environment.

Why do we write tests?

I’ve just realised that TDD is simply hypothesis-driven development.

This is what I think my code should be doing, let’s create a test to express that, followed by the code that validates that hypothesis.

Jakub Jurkiewicz

If you need convincing why you should practice Test-Driven Development, I have written several articles on TDD:

• Why should you practice TDD?
• How to learn TDD
• How to teach your team to write automated tests?
• Testing legacy data pipelines
• 4 reasons why TDD slows you down

Directory structure and test naming conventions

We need to start with naming because pytest follows the “convention-over-configuration” principle. If we name our directories and tests in the right way, we can run them without configuring anything. What is the “right way” in this case?

First, we must create a tests directory. We can have as many subdirectories as we want. I suggest having a separate directory for every feature you test.

In the tests directory (or its subdirectories) we store the Python scripts with the tests. pytest will find them if the script’s name starts with test_. We can put other files in the tests directory as long as their names have different prefixes. For example, we can have the following structure:

tests/
    test_some_name.py <- pytest will look for tests here
    another_script.py
    feature_A/
        test_feature_A_does_X.py <- and here
        mock_dependencies_for_A.py

When the directory structure is ready, pytest knows where to look for tests. But which tests will it run?

What does a test look like in pytest?

A pytest test is a Python function whose name starts with test_. By default, those functions accept no parameters (I’ll show you later how to use parameters in tests). Optionally, we can put those functions in a class. When we do it, the class name must begin with Test. Of course, all test method names must begin with test_.

Both of the following examples are valid (but empty) pytest tests:

def test_do_nothing():
    pass

and with a test class:

class TestDoingNothing:
    def test_do_nothing(self):
        pass

We have our first test! It doesn’t test anything, but we can run it. How do we do it?

How to run pytest tests

We have two options: the command line and the test runner in the IDE (PyCharm, IntelliJ, etc.).

When we want to use the command line, we should navigate to the directory with the tests directory. (don’t open the test directory!) and type the pytest command. pytest will automatically find all of the tests and run them. When you want to run a single test, type pytest -k name_of_the_test.

In your IDE, it’s even easier. Right-click the tests directory and select the Run tests command. If you want to run a single test, open the file containing the test and click the green triangle next to the test function. In the menu, choose the Run test option.

What’s the difference between both options? There is no difference when you run all of the tests at once. However, when you run a single test in the IDE, the working directory changes to the location of the test! Check the working directory path twice if your test needs to load data from files — the correct path will differ depending on how you run tests.

How to test something in pytest?

Generally, a test should consist of three parts: setup, action, and assertions. For example, if we test the append function of a Python list. The test may look like this:

def test_should_append_an_element_to_an_empty_list():
    element_to_be_added = 123
    object_under_test = []

    object_under_test.append(element_to_be_added)

    assert len(object_under_test) == 1, 'The list should contain one element'
    assert object_under_test[0] == element_to_be_added

What does happen in the function above? First, we prepare the test data and the object we want to test (by convention, we tend to name it object_under_test to distinguish the tested code from other variables in the test). After that, we use the tested object. In the end, we verify the object’s properties to check whether the action had the desired effect.

When do a test pass, and when do they fail?

A test passes when the test function doesn’t raise an exception. The test may fail for two reasons:

• AssertionError - if the condition in the assert line evaluates to False. If we add an optional comment to the assert statement (like in the first assertion in the example above), the comment will be returned as the error message.
• any other error - if the tested code raised an exception (and we don’t catch it), the test fails too.

How to read pytest test results?

Let’s assume I put the test from the previous example in a test file called test_example.py. When I run the pytest command, I’ll see the following result:

======================================== test session starts ========================================
platform darwin -- Python 3.10.0, pytest-7.2.0, pluggy-1.0.0
rootdir: THE_DIRECTORY_WITH_THE_PROJECT
collected 1 item

tests/test_example.py .                                                                       [100%]

========================================= 1 passed in 0.03s =========================================

The rootdir tells us where pytest looks for the tests. If your tests don’t run, the rootdir is the first thing you should check. Perhaps, you run the command in the wrong directory.

After that, we see the number of tests found by pytest: collected 1 item.

Next, pytest lists all of the test files one by one. In this example, I have only one file. Every test run by pytest is denoted as a single dot next to the test file name.

In the end, pytest summarizes the number of passed/failed tests and the elapsed time.

Let’s break the test. We add a single element to the list, but we’ll check if the list contains two elements in our assertion. Of course, our new assertion makes no sense, but we want to see what happens when a test fails:

def test_should_append_an_element_to_an_empty_list():
    element_to_be_added = 123
    object_under_test = []

    object_under_test.append(element_to_be_added)

    # the following line will fail
    assert len(object_under_test) == 2, 'The list should contain two elements'
    assert object_under_test[0] == element_to_be_added

Now, the test results look like this:

======================================== test session starts ========================================
platform darwin -- Python 3.10.0, pytest-7.2.0, pluggy-1.0.0
rootdir: THE_DIRECTORY_WITH_THE_PROJECT
collected 1 item

tests/test_example.py F                                                                       [100%]

============================================= FAILURES ==============================================
__________________________ test_should_append_an_element_to_an_empty_list ___________________________

    def test_should_append_an_element_to_an_empty_list():
        element_to_be_added = 123
        object_under_test = []

        object_under_test.append(element_to_be_added)

>       assert len(object_under_test) == 2, 'The list should contain two elements'
E       AssertionError: The list should contain two elements
E       assert 1 == 2
E        +  where 1 = len([123])

tests/test_example.py:7: AssertionError
====================================== short test summary info ======================================
FAILED tests/test_example.py::test_should_append_an_element_to_an_empty_list - AssertionError: The list should contain two elements
========================================= 1 failed in 0.05s =========================================

We see an F instead of a dot in the tests list. Fs denote failing tests. Below the tests, pytest reports every failure separately. It prints the test code, marks the failing line with >, prints the errors in lines starting with an E, and reports which code line failed (it’s the :7 next to the file path).

The end of the test report contains the short test summary info with names of the failed tests and the error message from the raised exception.

How to write a readable test?

pytest will run any code we put in the test function. Nothing stops us from creating a terrible, messy test code. Ideally, the test code should be easy to read and understand. We have several best practices governing writing tests. For a start, you should know about two of them.

Single business case per test

A single test may contain multiple assert statements, but all of those assert statements should test the details of a single business use case. For example, if you write code to generate an invoice, you should write separate tests for testing the tax calculations, data retrieval, tax identification number validation, etc.

When you have multiple assertions per test, consider encapsulating them in an assert object to separate the business meaning from the technical implementation.

Given/When/Then

Our test consists of three parts. I prefer explicitly separating them by putting the Given/When/Then headers as code comments. For example, our test could look like this:

def test_should_append_an_element_to_an_empty_list():
    # Given
    element_to_be_added = 123
    object_under_test = []

    # When
    object_under_test.append(element_to_be_added)

    # Then
    assert len(object_under_test) == 1, 'The list should contain one element'
    assert object_under_test[0] == element_to_be_added

It’s a controversial practice. Some people loathe adding such comments because they seem redundant. After all, you are supposed to have three blocks of code in a test. Do you need to name them when you always have the same three-block structure? Others enjoy alliterations and name their blocks Arrange/Act/Assert.

Whatever you choose, remember to have three separate parts of the test code. Don’t mix them.

How to setup test dependencies in pytest

What if multiple tests require the same data preparation code? Should we copy/paste the Given block between test functions? Do we extract the Given block to a separate function and call the setup function in every test?

In pytest, we have yet another option, which seems to be better than copying/pasting code or calling a setup function in every test. We can use pytest fixtures.

What is a fixture?

A fixture is a function we implement to prepare the test data. pytest runs them automatically and caches the values returned by the function. If we use the same fixture in multiple tests, pytest reuses the returned values without calling the setup function every time.

Assume we load a Pandas DataFrame in the given section of our test:

import pandas as pd


def test_something_here():
    # Given
    test_data = pd.read_csv('input_data.csv')

Instead of opening the file in every test (which takes time), we can read it once and pass the DataFrame to every test function where we need the same input data:

import pandas as pd
import pytest


class TestClass:
    @pytest.fixture(scope="class")
    def input_data(self):
        return pd.read_csv('input_data.csv')

    def test_something_here(self, input_data):
        # we don't need to load the input_data anymore

In the test function, we have added a parameter input_data. When we run the test, pytest will look for a fixture function with the same name and pass its returned value to the test function.

We can also use fixtures when we don’t care about the returned value. For example, when our setup function creates a file. We can also pass another fixture to a fixture. In the example below, we use a fixture to create the test directory. Then another fixture writes the input data in the directory. Ultimately, we have a test with the fixture name as a parameter. The variable contains None. We use it only to instruct pytest what setup functions to run:

import os
import pandas as pd


class TestLoadCsv:
    # this creates the `data/csv/` directory if it doesn't exist
    @pytest.fixture(scope='class')
    def test_directory(self):
        os.makedirs('data/csv/', exist_ok=True)

    # it writes a test file in the `data/csv` directory
    @pytest.fixture(scope="class")
    def csv_without_header(self, test_directory):
        df = pd.DataFrame(
            {
                "col1": [1, 2, 3],
                "col2": [4, 5, 6],
                "col3": [7, 8, 9],
            }
        )
        df.to_csv('data/csv/without_headers.csv', index=False, header=False)

    # it needs the test file to exist, hence the fixture parameter
    # note that the fixtures don't return anything
    def test_load_csv_without_headers(self, csv_without_header):
        # here is the test

What are the fixture scopes?

The scope parameter tells pytest how to reuse the value returned by the fixture. By default, the fixture function is called before every test function. However, we can change the scope if we want to reuse the returned value (or a side-effect created by the fixture). In my example, I want to run the setup code only once for every test in the test class.

You can find all supported scope values in the pytest documentation.

How to run a cleanup code after a test?

What if we wanted to remove the test file after the tests?

We need a slightly different fixture code if we want to run a cleanup code after the tests (when pytest exits the fixture scope). Instead of returning a value, we will use the yield keyword. Everything we put before the yield runs before the fixture scope. Everything after the yield runs after all of the tests in the fixture scope.

Suppose I wrote a fixture to start a Selenium session before every test. I need to stop the browser at the end of the test. We can use a function-scoped fixture to do so:

from selenium import webdriver
import pytest

@pytest.fixture(scope='function')
def driver():
    driver = webdriver.Safari()
    yield driver
    driver.quit()

def test_using_selenium(driver):
    pass

The fixture will setup the Selenium driver and pass the instance to the test. The test will run with the driver available in the parameter. After the test finishes, the fixture runs the quit function.

Reusing test code with different input data

Sometimes, instead of reusing the setup code, we want to reuse the When/Then blocks with different data. We don’t need to copy/paste test code! becausepytest supports parameterized tests. We can run the same test code with different input data values.

If we wanted to verify whether a Python list can accept values of multiple types, we could write our first example like this:

import pytest


@pytest.mark.parametrize(
    "element_to_be_added,number_of_elements",
    [(123, 1), ("123", 1), ("this is my test", 1)]
)
def test_should_append_an_element_to_an_empty_list(element_to_be_added, number_of_elements):
    object_under_test = []

    object_under_test.append(element_to_be_added)

    assert len(object_under_test) == number_of_elements
    assert object_under_test[0] == element_to_be_added

In the test function, we have two parameters. pytest will pass them there automatically from the @pytest.mark.parametrize fixture.

In the fixture, we have to define the parameter names — the fixture’s first parameter. Note that it’s a single string with names separated by commas (without spaces!).

We have a list of parameter values in the fixture’s second parameter. Each of those tuples replaces the test_should_append_an_element_to_an_empty_list parameters listed in the first argument of the fixture.

In the test report, we see three test runs (but we had to implement only one test):

======================================== test session starts ========================================
platform darwin -- Python 3.10.0, pytest-7.2.0, pluggy-1.0.0
rootdir: /Users/myszon/Projects/blog_posts/pytest
collected 3 items

tests/test_example.py ...                                                                     [100%]

========================================= 3 passed in 0.03s =========================================

If the test fails, we get a detailed error log telling us which variant of the test failed. Note the square brackets after the test name. Those brackets contain the parameter values separated by -.

====================================== short test summary info ======================================
FAILED tests/test_example.py::test_should_append_an_element_to_an_empty_list[123-10] - assert 1 == (1 + 1)
FAILED tests/test_example.py::test_should_append_an_element_to_an_empty_list[123-11] - AssertionError: assert 1 == (1 + 1)
FAILED tests/test_example.py::test_should_append_an_element_to_an_empty_list[this is my test-1] - AssertionError: assert 1 == (1 + 1)
========================================= 3 failed in 0.07s =========================================