Function Doctests

Doctests are a very handy feature for automatically testing a function by looking at its inputs and outputs.

digits_onlys(s)

As an example, the digits_only(s) function below takes in a string and returns a version of that string made of only its digits.

Doctest Syntax

Doctests are written inside the """triple quoted""" text at the top of a function. Here is the digits_only() function including Doctests:

def digits_only(s):
    """
    Given a string s.
    Return a string made of all
    the chars in s which are digits,
    so 'Hi4!x3' returns '43'.
    >>> digits_only('Hi4!x3')
    '43'
    >>> digits_only('123')
    '123'
    >>> digits_only('')
    ''
    """
    result = ''
    for i in range(len(s)):
        if s[i].isdigit():
            result += s[i]
    return result

Each Doctest is marked by a >>> for example:

    >>> digits_only('Hi4!x3')
    '43'

The first line spells out the call digits_only('Hi4!x3') and the second line shows the expected result. The syntax mimics the appearance of calling a function in the interpreter, which is why the >>> appears.

The strength of Doctests is that they are written right next to the code they test and require no extra setup, so they are easy to build and use while writing a function. Doctests are ideal for relatively simple functions. For more complex testing, Python has a more formal unit test framework. Such unit tests are more capable than simple Doctests, but are more work to maintain and run.

Running the Doctest

The easiest way to run the Doctest in an environment like PyCharm is right-clicking directly on the Doctest text, yielding a "Run Doctest digits_only()" option. There is a quirk in PyCharm where the first time you click, the Doctest menu item is not there. Wait a second and click a second time and it should be there.

If the Doctest passes, the output is minimal; in PyCharm a green checkmark appears below the code area.

If the test fails, the Doctest output gives details about the failed case. For example, adding a bug to digits_only() so it appends exclamation marks to the result produces this failed Doctest output:

Failed example:
    digits_only('Hi4!x3')
Expected:
    '43'
Got:
    '43!!!'

To run the Doctest a second time, in PyCharm just click the green-play button at the left to re-run the most recent Doctest, or look in the interface for the run Doctest keyboard-accelerator. When you are working on a function, it's normal to re-run a Doctest again and again, so it's good to have a quick way to do it.

Right click on a blank space in the python code, and there is a menu option to run all the Doctests found in the file, not just the tests for one function.

Writing Doctests

The great virtue of function tests is that just a few, simple test cases give most of the benefit. Most of the time you do not need exotic, hard to think of test cases. Here are some ways to think about test cases to write.

1. Basic Tests

Write one or two cases that are obvious, basic applications of the algorithm. Here is a basic case for digits_only() - it includes a few digits and a few non-digit chars.

    >>> digits_only('Hi4!x3')
    '43'

Writing a basic test case is also an easy way to get your thoughts organized before writing the code.

2. Edge Cases

Edge cases are cases up against a limit of the algorithm. For digits_only(), we can try inputs that are 100% digit and 0% digit. Also, the empty string is a good edge case test for a string algorithm.

    >>> digits_only('123')
    '123'
    >>> digits_only('xyz')
    ''
    >>> digits_only('')
    ''

3. Check Boundaries - Blackjack Rule

If an algorithm has a special number where its behavior changes, you should have tests on both sides of the boundary. For example in the card game blackjack, the scoring of the hand is different when the cards add to more than 21. For a blackjack algorithm, you should have both a 21 test case and a 22 test case, checking that both sides of the boundary are correct. You can imagine an off-by-one error, where logic that is supposed to act for scores of 22 or more, accidentally starts at 21 or 23 instead.

Similarly, if a function returns a boolean, be sure to have some tests where the expected is True and some where the expected is False.

4. Tiny Tests

Suppose the function is not working correctly. Debugging - thinking through line by line what the code is doing - can be a lot easier with a tiny test case. There's nothing wrong with a very small test case like this:

    >>> digits_only('a1')
    '1'

Doctests and Correctness

If a function passes all its Doctests, does that prove that the function is correct? No. As a practical matter, Doctests do an excellent job of finding many bugs, speeding up the coding and debugging process. However, passing all Doctests is not a proof that the code is correct. There can be a bug in the code that the tests simply do not hit.

Suppose we have an area of land and we are not sure if there is oil there. We sink 5 exploratory oil wells, and none of them find oil. Does that prove there is no oil? No, there could be oil at a spot we didn't check. As a matter of probability, the 5 wells with no oil suggests that there is no oil, but it's not proof. As a practical matter, function tests do a fantastic job of finding many bugs, but remember that they are not airtight. Proving that code is correct is surprisingly difficult, so the standard practice is to use tests.

Doctest Variables

The Doctest >>> code can set up variables for use by the later lines. So if there were a dictionary or some file data, you could set a variable on one line and use it on later lines, like this:

    >>> d = {'a': 4, 'b': 17}  # setting variable "d"
    >>> dict_fn(d)
    42

Doctest Printing

Usually, a function returns some Python value, and that value is what's tested in the Doctest. However, Doctests also work with functions that print. If a function prints to stdout, the Doctest will check that the lines after the >>> match what the function prints.

    >>> foo(6)
    This function prints a line with 6
    And another line down here
    >>> foo(7)
    This function prints a line with 7
    And another line down here
    >>>

Doctest Format Quirks

To verify the expected function result, the Doctest system actually formats the return value as text and compares the resulting text to the expected text after the >>> in the Doctest. The comparison will fail if there's just an extra space in the expected text, so those characters must be typed in exactly correct.

Another Doctest quirk is that a function return of None is represented by literally nothing, just the next >>> prompt in the Doctest, like this:

    >>> fuction_that_returns_none()
    >>>
    >>> # above checks for None return value

A final quirk of the Doctest data is that instances of '\n' in the input or output need an extra backslash, like this: '\\n'

    >>> digits_only('xyz\\n')
    ''

Doctest Command Line

There is a command line interface to run all the tests found in a python file, like this:

$ python3 -m doctest str1.py

If all the tests pass, it prints nothing. If there is a failure, it prints details of the expected vs. got failures. Run the Doctests in verbose mode with the form "doctest -v" to print extra information during the run.

For more information, see the official Python Doctest documentation.