A comprehensive guide to pytest

 Testing. A facet of software engineering evoking groans from developers everywhere since epoch. While I do not have a panacea, if you’re working with Python and you’re in need of exhaustive literature on unit testing (with pytest), look no further.

To procure such knowledge, I stumbled through the mystifying forests of Documentation, dove deep into the hostile waters of Stack Overflow, and scaled across the sheer precipice of Page 37 within the unforgiving mountains of Google. May this be the last article you will ever need on the topic.

Getting Started

Install pytest and related dependencies:

pipenv install --dev pytest pytest-cov pytest-mock

There are many more plugins, but coverage and mocks should go a long way before you need anything else.

I also use a package manager called pipenv, which I highly recommend for easy and convenient package management. Opensource.com has an excellent writeup on why you should be using it over vanilla pip.

pytest Setup

Put a conftest.py file in the root of the project. This file:

• contains “global” settings for pytest, including mark registration (more on that later) and globally shared fixtures (also more on this later)
• is responsible for giving pytest the scope; by putting the file in root, pytest will recognize the src module without needing to manually add it to PYTHONPATH

pytest does a great job of discovering tests, so the actual tests can be placed anywhere. I recommend the following folder structure:

/
|- <project name>/
|- tests/
   |- acceptance/
   |- unit/
      |- # tests go here

Out-of-the-box, all pytest test files must be prefixed or postfixed with “test” (singular). For example, if you were testing a module called handler.py, your test file should be called test_handler.py or handler_test.py. The test file doesn’t actually need to share the base name of the module to be tested — it could be called test_zzyzx_the_end_of_the_world.py and it would still work the same — but for the sake of sanity you probably shouldn’t do that.

The unit tests themselves can be organized however makes sense, but I recommend reflecting the structure of the actual code. This allows you to quickly find relevant tests to a given piece of code. Unit tests should only test the logic contained within each file (in other words: mock any imports), with a notable exception to this being custom Exception/Error classes, which may inadvertently be tested in a given module.

Writing Tests

For barebones tests, you don’t need to import the pytest module. The simplest pytest test you can write for a method is as follows:

def hello_world():
    return 'hello world'def test_function():
    assert hello_world() == 'hello world'

pytest uses standard Python asserts for most assertions. pytest can even check the context of the asserted objects, e.g.:

def hello_worlds():
    return {
        'english': 'hello world',
        'chinese': '你好世界'
    }def test_function():
    assert hello_world() == {'english': 'hello wurld',
                             'japanese': 'ハロー・ワールド'}E   AssertionError: assert {'english': 'hello world', 'chinese': '你好世界'} == {'english': 'hello wurld', 'japanese': 'ハロー・ワールド'}
E     Differing items:
E     {'english': 'hello world'} != {'english': 'hello wurld'}
E     Left contains 1 more item:
E     {'chinese': '你好世界'}
E     Right contains 1 more item:
E     {'japanese': 'ハロー・ワールド'}

That is the extent of what can be done without importing pytest. The rest of this document covers features that require import pytest.

pytest can also check for the existence and context of exceptions.

import pytestdef hello_error():
    raise NotImplementedErrordef test_function():
    with pytest.raises(NotImplementedError):
        hello_error()def test_context():
    with pytest.raises(NotImplementedError) as e:
        hello_error()
    assert e.xyz == abc
    # note that the exception context object is referenced *outside* the `with` block

It’s possible to have multiple asserts under a single test function; whether or not this should be done is entirely a semantic decision.

Fixtures

Fixtures are separate pieces of reusable code that can be called from tests and other fixtures. They’re often used for setup and teardown, but can also be used to abstract repeated actions.

Fixture Basics

Fixtures are decorated with @pytest.fixture and are requested by passing in the name of a fixture. Multiple fixtures can be requested by a single method. Fixtures can also request other fixtures.

import pytest

@pytest.fixture
def hello():
    return 'hello'

@pytest.fixture
def world():
    return 'world'

@pytest.fixture
def hello_world(hello, world):
    return hello + ' ' + world

def test_function(hello_world, hello, world):
    assert hello_world = hello + ' ' + world@pytest.fixture
def hello():
    return 'hello'@pytest.fixture
def world():
    return 'world'@pytest.fixture
def hello_world(hello, world):
    return hello + ' ' + worlddef test_function(hello_world, hello, world):
    assert hello_world = hello + ' ' + world

Fixtures have independent execution contexts except when called multiple times in a single test.

The following example has the same fixture requested in different tests, and provides distinct objects to each test.

import pytest

@pytest.fixture
def a_list():
    return ['a']

def test_str(a_list):
    a_list.append('b')
    assert a_list == ['a', 'b']

def test_int(a_list):
    a_list.append(2)
    assert  a_list == ['a', 2]

On the other hand, this example has the same fixture requested twice in the same test (once transitively and the second directly), and returns the same object.

import pytest

@pytest.fixture
def a_list():
    return ['a']

@pytest.fixture
def append_b(a_list):
    return a_list.append('b')

def test_int(b_list, a_list):
    assert  a_list == ['a', 'b']

Fancy Fixtures (officially “factory as a fixture”)

To be able to pass parameters to a fixture (e.g. to dynamically generate input data), you need to use the factory as a fixture paradigm. Essentially, the fixture itself returns a function that does what you need.

import pytest

@pytest.fixture
def gen_input():
    def _gen_input(a: bool):
        return {'a': a}

    return _gen_input

def test_input(gen_input):
    assert gen_input(True) == {'a': True}

Unfortunately, docstrings and code-completion features don’t play well with this paradigm, but that doesn’t mean you shouldn’t include documentation!

Autouse Fixtures

You can specify that a fixture be used automatically on every test. This is useful if all tests have a common setup component.

import pytest

@pytest.fixture(autouse=True)
def some_func():
    # do stuff before every test

Note, however, that autouse fixtures will not yield any objects unless they are explicitly requested by a testing function.

Sharing Fixtures & Fixture Lifecycle

To share fixtures across different modules, stick them in the conftest.py file.

By default, fixtures are invoked and torn down once per function. You can change this to have a fixture’s execution context be shared across the following scopes (shamelessly stolen from the docs):

• function(default) — the fixture is destroyed at the end of the test
• class — the fixture is destroyed after the last test in the class
• module — the fixture is destroyed after the last test in the module
• package — the fixture is destroyed after the last test in the package
• session — the fixture is destroyed at the end of the test session.

The scopes are passed in as an argument to the fixture decorator.

import pytest

@pytest.fixture(scope="session")
def test_something():
    # something

Yielding Fixtures

Sometimes you may have complicated objects involved in setup and teardown. Enter yield fixtures.

import pytest

@pytest.fixture
def read_file():
    f = open('hello_world.txt', 'r')
    yield f.readlines()
    f.close()  # teardown after yield

# alternatively, using `with`:
@pytest.fixture
def read_file():
    with open('hello_world.txt', 'r') as f:
        yield f.readlines()

def test_file(read_file):
    assert read_file = ['hello world']

yield provides an object to the requesting function and "pauses" execution of the fixture. When the requesting function completes, the fixtures are torn down in reverse order ("resuming" execution of the fixture).

Prior to pytest v3, @pytest.yield_fixture was the decorator to be used. Though we’re way past that now, some old docs and Stack Overflow questions/answers may be dated to that time.

Mocking

Unit tests should be atomic; mocking enables that. You want to mock calls to most (if not all) external functions, especially if said function is slow.

Mocking in pytest requires pytest-mock, which you'll have installed following the Getting Started section above. It provides the mocker fixture and can mock objects, methods, and variables.

Mocking Variables

Variable mocking is mainly used for mocking globals (outside of function scope). Say you have the following file that contains a Lambda handler:

# lambda_handler.pyis_cold_start = True
def handler():
    if is_cold_start:
        # do stuff
        is_cold_start = False
    # do more stuff
    return True  # arbitrary return value for illustration

You want to be able to test handler() when is_cold_start = False; to do so, you must mock is_cold_start.

# handler_test.pyimport pytest
import lambda_handler

def test_handler(mocker):
    mocker.patch.object(lambda_handler, 'is_cold_start', False)
    assert handler()

The signature is as follows:

mocker.patch.object(
    module,      # this is NOT a string
    'variable',  # this IS a string
    value        # this is whatever
)

The module name follows the import name. For example, if lambda_handler.py were in a folder src/, the module would then be src.lambda_handler.

Mocking Functions

There are some nuances with function mocking. Here are a series of examples.

Basics

# hello_world.pyimport os

def say_passphrase():
    passphrase = os.environ.get('PASSPHRASE')
    return passphrase or 'I need somebody (Help!)'

To mock the call to os.environ.get():

# hello_world_test.pyimport pytest
from hello_world import say_passphrase

def test_passphrase(mocker):
    mocker.patch('hello_world.os.environ.get', return_value='hello world')
    assert say_passphrase() == 'hello world'

Note that the first argument is the fully-qualified name of the method to be mocked as a string.

Because hello_world.py imported the os module, the target of the mock is the instance of os.environ.get() that resides in the hello_world module. Alternatively, if in hello_world.py you did from os import environ, then the fully-qualified name would be hello_world.environ.get.

Different Return Values

If the same mock is required in multiple tests, you can put the mocker in a fixture and return the mocked object. Given the same hello_world.py as above, here's what the tests would look like:

# hello_world_test.pyimport pytest
from hello_world import say_passphrase

@pytest.fixture
def mock_getenv(mocker):
    return mocker.patch('hello_world.os.environ.get')

def test_passphrase_exists(mock_getenv):
    mock_getenv.return_value = 'hello world'
    assert say_passphrase() == 'hello world'

def test_passphrase_not_exists(mock_getenv):
    mock_getenv.return_value = None
    assert say_passphrase() == 'I need somebody (Help!)'

A few things to note here:

• the call to mocker.patch() doesn't include the return_value argument — this creates a generic mock object that we can manipulate later, which the fixture then returns
• the tests don’t request mocker directly; they request the mock_getenv fixture instead — this provides the tests with the mocked object, and because the tests don’t require any additional mocks, mocker doesn’t need to be separately passed in
• the return_values can be unique per test; this is due to fixture behavior (see above)

Dynamic Return Values (via side_effect)

Let’s say the same method was called multiple times in the same function, with different arguments. How do you mock that? Let’s find out!

# hello_world.pyimport os

def say_passphrase():
    greeting = os.environ.get('GREETING')
    subject = os.environ.get('SUBJECT')
    return greeting + ' ' + subject

In this example, os.environ.get() is called twice, each with different arguments. We’ll need a way to differentiate the two calls.

# hello_world_test.pyimport pytest
from hello_world import say_passphrase

def getenv_side_effect(key, default = None):
    if key == 'GREETING':
        return 'hello'
    elif key == 'SUBJECT':
        return 'world'
    else:
        return default

@pytest.fixture
def mock_getenv(mocker):
    return mocker.patch('hello_world.os.environ.get', side_effect=getenv_side_effect)

def test_passphrase(mock_getenv):
    assert say_passphrase() == 'hello world'

The side_effect argument allows you to pass a function that will be called in place of the actual function. This allows you to specify distinct return values depending on the arguments passed in. Also note that the side effect function is not a fixture and is not requested.

Even-More-Dynamic Return Values

If you had multiple calls to the same method, and you wanted to succinctly test different return scenarios of that method, how would that work?

The idea is to combine factory fixtures with the side effect functionality, like so:

# hello_world.pyimport os

def say_passphrase():
    greeting = os.environ.get('GREETING', 'hola')
    subject = os.environ.get('SUBJECT', 'mundo')
    return greeting + ' ' + subject

This is nearly identical to the previous example, except default values are now passed in to os.environ.get().

# hello_world_test.pyimport pytest
from hello_world import say_passphrase

def getenv_side_effect(key_exists: bool):
    def _getenv_side_effect(key, default = None):
        if key == 'GREETING':
            return 'hello' if key_exists else default
        elif key == 'SUBJECT':
            return 'world' if key_exists else default
    
    return _getenv_side_effect

@pytest.fixture
def mock_getenv(mocker):
    def _mock_getenv(key_exists):
        return mocker.patch('hello_world.os.environ.get',
                            side_effect=getenv_side_effect(key_exists))
    
    return _mock_getenv

def test_passphrase_exists(mock_getenv):
    mock_getenv(True)
    assert say_passphrase() == 'hello world'

def test_passphrase_not_exists(mock_getenv):
    mock_getenv(False)
    assert say_passphrase() == 'hola mundo'

Alternatively, to get the same effect you could have multiple side effect functions for different scenarios, and in each test load a different side effect.

Mocking Objects/Classes

Mocking objects is similar to mocking functions. Here are two examples.

Implicit Class Mocking

In this example, you’re using requests to make some API call. requests.get() returns a Response object, which has some number of attributes, one of which is status_code.

# hello_world.pyimport requests

def check_status():
    response = requests.get('hello.world')  # this returns a Response class
    return response.status_code

In the test, you mock the requests.get() call and store it as a mock object (mock_response). You can then chain attributes of the mock object so that the access of status_code attribute is mocked without explicitly mocking the Response class.

# hello_world_test.pyimport pytest
from hello_world import check_status

def test_check_status(mocker):
    mock_response = mocker.patch('hello_world.requests.get')
    mock_response.return_value.status_code = 200
    assert check_status() == 200

Explicit Class Mocking

Here you’re passing in an object that’s been previously instantiated. Let’s say the name attribute is of type str, and get_employer() returns a Company object that has a name attribute as well.

# hello_world.pyfrom people import Person

def greet(person: Person):
    name = person.name
    company = person.get_employer().name
    return f'hello, {name} from {company}'

In the test, you’ll have to explicitly mock the Person class, but Company can be implicitly mocked.

# hello_world_test.pyimport pytest
from hello_world import greet

def test_greet(mocker):
    mock_person = mocker.patch('hello_world.Person')  # explicitly mocked Person
    mock_person.name = 'Rich Fairbank'
    mock_person.get_employer.return_value.name = 'Capital One'  # implicitly mocked Company
    assert greet() == 'hello, Rich Fairbank from Capital One'

Internal Class Mocking

In this scenario, you are trying to mock an object that is internal to the method being tested. Let’s say for example greet() is a part of some class (left out for simplicity) and it creates its own instance of a Person to greet you. For the sake of example, assume that the values passed in to initialize the Person are static (though in reality would be retrieved from somewhere else).

# hello_world.pyfrom people import Person

def greet():
    person = Person("Dumbledore", "Hogwarts")
    name = person.name
    company = person.get_employer().name
    return f'{name} from {company} says "Greetings."'

In the test, you’ll have to explicitly mock the internal Person method a little differently from when you pass it in as an argument.

# hello_world_test.pyimport pytest
from hello_world import greet

def test_greet(mocker):
    mock_person = mocker.patch('hello_world.Person')  # the overall class is mocked the same way
    mock_person().name = 'Dumbledore'  # however, note the parentheses here...
    mock_person().get_employer.return_value.name = 'Hogwarts'  # and here
    
    assert greet() == 'Dumbledore from Hogwarts says "Greetings."'

Marking

That’s marking with an a. Marks are a way to tag/label your tests, so that you can selectively run a subset of tests (for example, you want to run only the test you’re working on).

To mark something, just decorate it with @pytest.mark.<your mark>, e.g.:

import pytest

@pytest.mark.wip
def test_something():
    # blah

You’ll also have to register your mark with pytest in conftest.py:

# conftest.pydef pytest_configure(config):
    config.addinivalue_line("markers", "wip: mark test that is in progress")

You can add additional marks by adding another config.addinivalue_line().

To run marks, use pytest -m <mark1> <mark2> ...

Parameterization

Sometimes you want to run the same test with different sets of inputs. Enter parameterization!

# hello_world.pydef greet(name: str, is_casual: bool, use_exclamation: bool):
    greeting = 'sup' if is_casual else 'salutations,'
    ending = '!' if use_exclamation else ''
    return f'{greeting} {name}{ending}'

You want to be able to easily test the different boolean combinations without rewriting basically the same test multiple times.

# hello_world_test.pyfrom hello_world import greet

@pytest.mark.parameterize('is_casual', 'use_exclamation',
                          [(True, True), (True, False), (False, True), (False, False)])
def test_greet(is_casual, use_exclamation):
    greeting = 'sup' if is_casual else 'salutations,'
    ending = '!' if use_exclamation else ''
    assert greet('Rich', is_casual, use_exclamation) == f'{greeting} Rich{ending}'

With parameterization, you provide the arguments to be parameterized and a list of the values to use. pytest will run the test once for each set of arguments provided.

That’s All Folks!

If you’ve made it this far, you’re a trooper and I hope I was able to help you with your testing needs. If not, reach out or comment on what you’d like to accomplish and I’ll take a look! Also, if you notice any issues with the info I’ve presented, please do let me know. The last thing I want to do is spread wrong information.

from: https://levelup.gitconnected.com/a-comprehensive-guide-to-pytest-3676f05df5a0