Obey the Testing Goat!

Progress on the Third Edition of the Book!

2024-08-07T16:13:00+01:00

In lieu of a formal announcement about the Third Edition, how about a progress update?

Core technology updates: Django + Python

Embarrassment-Driven Development

One of the main motivations for a third edition was that the 2e is based on Django 1.11, which dropped out of support back in 2017, and that's been a big turnoff for readers for a while, and quite embarrassing really.

So, the plan is to upgrade to Django 5.x, and progress is good -- I've already updated most of the core chapters to Django 4.2, and upgrade Python to 3.12 while I was at it. Django 5 is next, and I'm hoping/assuming it will be a smaller leap that 1->4 was, so that won't be far behind.

New Deployment Technologies: Docker + Ansible

I've always been proud that the book includes several chapters on how to actually deploy our app to production, and make the app live on the actual public Internet. But the deployment process from the first and second editions--broadly speaking, SSH in to your server, hack about to figure out how to get your app deployed manually, and then automate what you did with glorified shellscripts, aka Fabric--was starting to look less and less like what modern deployment looks like, or my experience of it at least.

I uhmmed and ahhed about it for a while, but in the end I decided to go with a deployment process that looks like this:

Package up our app into a Docker container, and use our tests to confirm it really works
Use Ansible to automate pushing that container onto a server and running it.

Check out the latest version of the deployment chapters here:

I think I like how it's turned out, a lot of the fiddliness and debugging of deployment/production-readiness can now happen locally (in Docker containers on your own machine), so I think that tightens and speeds up the feedback loop a fair bit.

JavaScript

The Javascript chapter was another head-scratcher. I wanted to move away from QUnit, and include some more modern/ES6 syntax. In the end, I decided to go with Jasmine, which is old but still popular, but to keep the browser-based test runner, which is a bit of an unconventional choice, but it does mean we can avoid the whole Node.js and node_modules learning curve.

Aside from that, I've wound down the "JavaScript is such a nightmare" jokes, because they're really not fair any more, and were probably never that funny besides.

Check out the new version here:

Chapter 16: A Gentle Excursion Into JavaScript

Some changes of emphasis

The other main changes to the book are going to be around how I talk about some of the tradeoffs involved in the use of mocking, and unit vs integration vs functional/e2e tests. I think the first and second editions were perhaps a little too opinionated on this front (I still cringe to think how defensive I was when I first wrote the Hot Lava chapter, sorry CaseY!!), and my thinking has evolved a lot since I wrote my second book with Bob.

That's still very much on the drawing board though, so you'll have to watch this space for updates on that front.

Anyways, all the latest versions of the 3e chapters are live here on the site, and also as an Early Release on O'Reilly Learning, so do dive in and let me know what you think!

Cosmic Python

2020-06-03T00:00:00+01:00

Folks I've written a new book!

Along with my coauthor Bob, we are proud to release "Architecture Patterns with Python", which you can find out more about at cosmicpython.com.

The cosmic soubriquet is a little joke, Cosmos being the opposite of Chaos in ancient Greek, so we want to propose patterns to minimise chaos in your applications.

But the subtitle of the book is Enabling TDD, DDD, and Event-Driven Microservices, and the TDD part is relevant to this blog, and fans of the Testing Goat. In my two years at MADE and working with Bob, I've refined some of my thinking and some of the ways I approach testing, and I think if I were writing TTDwP again today, I might change the way I present some things.

In brief:

Mocking is not the only way to handle external (I/O et al) dependencies for your unit tests. Other techniques are possible, and often offer major benefits
If you really want to get to a test pyramid (where unit tests outnumber slow/e2e/integration tests by an order of magnitude), then you'll probably need to make some specific design choices around identifying business logic and decoupling it from infrastructure code.
When deciding what kind of unit tests to write, there's a lot to be said for writing them at the highest level of abstraction possible. It gives you more room to refactor later.

If you're curious about those questions, head on over to cosmicpython.com, and let me know what you think!

New Book Excerpt: On Coupling and Abstractions

2019-09-03T14:40:00+01:00

At MADE.com i’ve been working on a new book with the chief architect, Bob Gregory, in which we talk about application architecture and design. What follows is an excerpt, a standalone chapter in which we talk about abstractions, testability, and coupling.

Have you ever struggled with writing tests for a bit of code that has a lot of dependencies? Someting involving I/O, interacting with 3rd parties, things like that? Wondering whether end-to-end/integration tests are the only way, and/or struggling with mocks that make tests hard to understand and evolve over time?

Most of the other chapters are about specific architectural patterns, but this one kind of stands on its own. When Bob and I first talked about choosing the right abstraction in order to enable testing, it was a real lightbulb moment for me, so I’m really interested in what people think — does it help you?

You can follow the progress on the new book at https://github.com/python-leap/book/, and you can read the Early Release edition on Oreilly Safari (sign up for a free account if you need one) (and apologies for the title, we’re working hard to convince O’Reilly to change it)

Allow us a brief digression on the subject of abstractions, dear reader. We’ve talked about abstractions quite a lot. The Repository is an abstraction over permanent storage for example. But what makes a good abstraction? What do we want from them? And how do they relate to testing?

Encapsulation and Abstraction, in General

Take a look at the following two snippets of Python code, [urllib_example] and [requests_example]:

Example 1. Do a search with urllib

import json
from urllib.request import urlopen
from urllib.parse import urlencode

params = dict(q='Sausages', format='json')
handle = urlopen('http://api.duckduckgo.com' + '?' + urlencode(params))
raw_text = handle.read().decode('utf8')
parsed = json.loads(raw_text)

results = parsed['RelatedTopics']
for r in results:
    if 'Text' in r:
        print(r['FirstURL'] + ' - ' + r['Text'])

Example 2. Do a search with requests

import requests

params = dict(q='Sausages', format='json')
parsed = requests.get('http://api.duckduckgo.com/', params=params).json()

results = parsed['RelatedTopics']
for r in results:
    if 'Text' in r:
        print(r['FirstURL'] + ' - ' + r['Text'])

Both of these code listings do the same thing: they submit form-encoded values to a URL in order to use a search engine API. But the second is simpler to read and understand because it operates at a higher level of abstraction.

We can take this one step further still by identifying and naming the task we want the code to perform for us, and use an even higher-level abstraction to make it explicit:

Example 3. Do a search with the duckduckgo module

import duckduckgo
for r in duckduckgo.query('Sausages').results:
    print(r.url + ' - ' + r.text)

Encapsulating behavior using abstractions is a powerful tool for making our code more expressive and easier to maintain.

Note

This approach is inspired by the OO practice of responsibility-driven design. which would use the words roles and responsibilities rather than tasks. The main point is to think about code in terms of behavior, rather than in terms of data or algorithms. If you’ve come across CRC cards, they’re driving at the same thing.

(In a traditional OO language you might use an abstract base class or an interface to define an abstraction. In Python you can (and we sometimes do) use ABCs, but you can also rely on duck typing. The abstraction can just mean, "the public API of the thing you’re using"; a function name plus some arguments, for example.)

So far, so not-very-earth-shaking, you’re probably thinking. Abstraction and encapsulation are nice, we all knew that. So what?

Abstractions as a Way of Reducing Excessive Coupling

When we’re writing code for fun, or in a kata, we get to play with ideas freely, hammering things out and refactoring aggressively. In a large-scale system, though, we become constrained by the decisions made elsewhere in the system.

When we’re unable to change component A for fear of breaking component B, we say that the components have become coupled. Locally, coupling is a good thing: it’s a sign that our code is working together, each component supporting the others, fitting in place like the gears of a watch. In jargon, we say this works when there is high cohesion between the coupled elements.

Globally, coupling is a nuisance: it increases the risk and the cost of changing our code, sometimes to the point where we feel unable to make some changes at all. This is the problem with the ball of mud pattern: as the application grows, if we’re unable to prevent coupling between elements that have no cohesion, that coupling increases superlinearly until we are no longer able to effectively change our systems.

We can reduce the degree of coupling within a system ([coupling_illustration1]) by abstracting away the details ([coupling_illustration2]):

Lots of coupling

+--------+      +--------+
| System | ---> | System |
|   A    | ---> |   B    |
|        | ---> |        |
|        | ---> |        |
|        | ---> |        |
+--------+      +--------+

Less coupling

+--------+                           +--------+
| System |      /-------------\      | System |
|   A    |      |             | ---> |   B    |
|        | ---> | Abstraction | ---> |        |
|        |      |             | ---> |        |
|        |      \-------------/      |        |
+--------+                           +--------+

In both diagrams, we have a pair of subsystems, with the one dependent on the other. In the first diagram, there is a high degree of coupling between the two because of reasons. If we need to change system B, there’s a good chance that the change will ripple through to system A.

In the second, though, we have reduced the degree of coupling by inserting a new, simpler, abstraction. This abstraction serves to protect us from change by hiding away the complex details of whatever system B does.

But how do we pick the right abstraction? And what does this all have to do with testing?

Abstracting State Aids Testability

Time to get into the nitty-gritty. Let’s see an example. Imagine we want to write some code for synchronising two file directories which we’ll call the source and the destination.

If a file exists in the source, but not the destination, copy the file over.
If a file exists in the source, but has a different name than in the destination, rename the destination file to match.
If a file exists in the destination but not the source, remove it.

Our first and third requirements are simple enough, we can just compare two lists of paths. Our second is trickier, though. In order to detect renames, we’ll have to inspect the content of files. For this we can use a hashing function like md5 or SHA. The code to generate a SHA hash from a file is simple enough.

Example 4. Hashing a file (sync.py)

BLOCKSIZE = 65536

def hash_file(path):
    hasher = hashlib.sha1()
    with path.open("rb") as file:
        buf = file.read(BLOCKSIZE)
        while buf:
            hasher.update(buf)
            buf = file.read(BLOCKSIZE)
    return hasher.hexdigest()

Now we need to write the bit that makes decisions about what to do. When we have to tackle a problem from first principles, we usually try to write a simple implementation, and then refactor towards better design. We’ll use this approach throughout the book, because it’s how we write code in the real world: start with a solution to the smallest part of the problem, and then iteratively make the solution richer and better designed.

Our first hackish approach looks something like this:

Example 5. Basic sync algorithm (sync.py)

import hashlib
import os
import shutil
from pathlib import Path

def sync(source, dest):
    # Walk the source folder and build a dict of filenames and their hashes
    source_hashes = {}
    for folder, _, files in os.walk(source):
        for fn in files:
            source_hashes[hash_file(Path(folder) / fn)] = fn

    seen = set()  # Keep track of the files we've found in the target

    # Walk the target folder and get the filenames and hashes
    for folder, _, files in os.walk(dest):
        for fn in files:
            dest_path = Path(folder) / fn
            dest_hash = hash_file(dest_path)
            seen.add(dest_hash)

            # if there's a file in target that's not in source, delete it
            if dest_hash not in source_hashes:
                dest_path.remove()

            # if there's a file in target that has a different path in source,
            # move it to the correct path
            elif dest_hash in source_hashes and fn != source_hashes[dest_hash]:
                shutil.move(dest_path, Path(folder) / source_hashes[dest_hash])

    # for every file that appears in source but not target, copy the file to
    # the target
    for src_hash, fn in source_hashes.items():
        if src_hash not in seen:
            shutil.copy(Path(source) / fn, Path(dest) / fn)

Fantastic! We have some code and it looks okay, but before we run it on our hard drive, maybe we should test it? How do we go about testing this sort of thing?

Example 6. Some end-to-end tests (test_sync.py)

def test_when_a_file_exists_in_the_source_but_not_the_destination():
    try:
        source = tempfile.mkdtemp()
        dest = tempfile.mkdtemp()

        content = "I am a very useful file"
        (Path(source) / 'my-file').write_text(content)

        sync(source, dest)

        expected_path = Path(dest) /  'my-file'
        assert expected_path.exists()
        assert expected_path.read_text() == content

    finally:
        shutil.rmtree(source)
        shutil.rmtree(dest)


def test_when_a_file_has_been_renamed_in_the_source():
    try:
        source = tempfile.mkdtemp()
        dest = tempfile.mkdtemp()

        content = "I am a file that was renamed"
        source_path = Path(source) / 'source-filename'
        old_dest_path = Path(dest) / 'dest-filename'
        expected_dest_path = Path(dest) / 'source-filename'
        source_path.write_text(content)
        old_dest_path.write_text(content)

        sync(source, dest)

        assert old_dest_path.exists() is False
        assert expected_dest_path.read_text() == content


    finally:
        shutil.rmtree(source)
        shutil.rmtree(dest)

Wowsers, that’s a lot of setup for two very simple cases! The problem is that our domain logic, "figure out the difference between two directories," is tightly coupled to the IO code. We can’t run our difference algorithm without calling the pathlib, shutil, and hashlib modules.

Our high-level code is coupled to low-level details, and it’s making life hard. As the scenarios we consider get more complex, our tests will get more unwieldy. We can definitely refactor these tests (some of the cleanup could go into pytest fixtures for example) but as long as we’re doing filesystem operations, they’re going to stay slow and hard to read and write.

Choosing the right abstraction(s)

What could we do to rewrite our code to make it more testable?

Firstly we need to think about what our code needs from the filesystem. Reading through the code, there are really three distinct things happening. We can think of these as three distinct responsibilities that the code has.

We interrogate the filesystem using os.walk and determine hashes for a series of paths. This is actually very similar in both the source and the destination cases.
We decide a file is new, renamed, or redundant.
We copy, move, or delete, files to match the source.

Remember that we want to find simplifying abstractions for each of these responsibilities. That will let us hide the messy details so that we can focus on the interesting logic.

For (1) and (2), we’ve already intuitively started using an abstraction, a dictionary of hashes to paths, and you may already have been thinking, "why not use build up a dictionary for the destination folder as well as the source, then we just compare two dicts?" That seems like a very nice way to abstract the current state of the filesystem.

source_files = {'hash1': 'path1', 'hash2': 'path2'}
dest_files = {'hash1': 'path1', 'hash2': 'pathX'}

What about moving from step (2) to step (3)? How can we abstract out the actual move/copy/delete filesystem interaction?

We’re going to apply a trick here that we’ll employ on a grand scale later in the book. We’re going to separate what we want to do from how to do it. We’re going to make our program output a list of commands that look like this:

("COPY", "sourcepath", "destpath"),
("MOVE", "old", "new"),

Now we could write tests that just use 2 filesystem dicts as inputs, and expect lists of tuples of strings representing actions as outputs.

Instead of saying "given this actual filesystem, when I run my function, check what actions have happened?" we say, "given this abstraction of a filesystem, what abstraction of filesystem actions will happen?"

Example 7. Simplified inputs and outputs in our tests (test_sync.py)

    def test_when_a_file_exists_in_the_source_but_not_the_destination():
        src_hashes = {'hash1': 'fn1'}
        dst_hashes = {}
        expected_actions = [('COPY', '/src/fn1', '/dst/fn1')]
        ...

    def test_when_a_file_has_been_renamed_in_the_source():
        src_hashes = {'hash1': 'fn1'}
        dst_hashes = {'hash1': 'fn2'}
        expected_actions == [('MOVE', '/dst/fn2', '/dst/fn1')]
        ...

Implementing our chosen abstractions

That’s all very well, but how do we actually write those new tests, and how do we change our implementation to make it all work?

Our goal is to isolate the clever part of our system, and to be able to test it thoroughly without needing to set up a real filesystem. We’ll create a "core" of code that has no dependencies on external state, and then see how it responds when we give it input from the outside world.

Let’s start off by splitting the code up to separate the stateful parts from the logic.

Example 8. Split our code into three (sync.py)

def sync(source, dest):  #<3>
    # imperative shell step 1, gather inputs
    source_hashes = read_paths_and_hashes(source)
    dest_hashes = read_paths_and_hashes(dest)

    # step 2: call functional core
    actions = determine_actions(source_hashes, dest_hashes, source, dest)

    # imperative shell step 3, apply outputs
    for action, *paths in actions:
        if action == 'copy':
            shutil.copyfile(*paths)
        if action == 'move':
            shutil.move(*paths)
        if action == 'delete':
            os.remove(paths[0])

...

def read_paths_and_hashes(root):  #<1>
    hashes = {}
    for folder, _, files in os.walk(root):
        for fn in files:
            hashes[hash_file(Path(folder) / fn)] = fn
    return hashes


def determine_actions(src_hashes, dst_hashes, src_folder, dst_folder):  #<2>
    for sha, filename in src_hashes.items():
        if sha not in dst_hashes:
            sourcepath = Path(src_folder) / filename
            destpath = Path(dst_folder) / filename
            yield 'copy', sourcepath, destpath

        elif dst_hashes[sha] != filename:
            olddestpath = Path(dst_folder) / dst_hashes[sha]
            newdestpath = Path(dst_folder) / filename
            yield 'move', olddestpath, newdestpath

    for sha, filename in dst_hashes.items():
        if sha not in src_hashes:
            yield 'delete', dst_folder / filename

The code to build up the dictionary of paths and hashes is now trivially easy to write.
The core of our "business logic," which says, "given these two sets of hashes and filenames, what should we copy/move/delete?" takes simple data structures and returns simple data structures.
And our top-level module now contains almost no logic whatseover, it’s just an imperative series of steps: gather inputs, call our logic, apply outputs.

Our tests now act directly on the determine_actions() function:

Example 9. Nicer looking tests (test_sync.py)

    @staticmethod
    def test_when_a_file_exists_in_the_source_but_not_the_destination():
        src_hashes = {'hash1': 'fn1'}
        dst_hashes = {}
        actions = list(determine_actions(src_hashes, dst_hashes, Path('/src'), Path('/dst')))
        assert actions == [('copy', Path('/src/fn1'), Path('/dst/fn1'))]

    @staticmethod
    def test_when_a_file_has_been_renamed_in_the_source():
        src_hashes = {'hash1': 'fn1'}
        dst_hashes = {'hash1': 'fn2'}
        actions = list(determine_actions(src_hashes, dst_hashes, Path('/src'), Path('/dst')))
        assert actions == [('move', Path('/dst/fn2'), Path('/dst/fn1'))]

Because we’ve disentangled the logic of our program - the code for identifying changes - from the low-level details of IO, we can easily test the core of our code.

Testing Edge-to-Edge with Fakes

When we start writing a new system, we often focus on the core logic first, driving it with direct unit tests. At some point, though, we want to test bigger chunks of the system together.

We could return to our end-to-end tests, but those are still as tricky to write and maintain as before. Instead, we often write tests that invoke a whole system together, but fake the IO, sort of edge-to-edge.

Example 10. Explicit dependencies (sync.py)

def synchronise_dirs(reader, filesystem, source_root, dest_root): #<1>

    source_hashes = reader(source_root) #<2>
    dest_hashes = reader(dest_root)

    for sha, filename in src_hashes.items():
        if sha not in dst_hashes:
            sourcepath = source_root / filename
            destpath = dest_root / filename
            filesystem.copy(destpath, sourcepath) #<3>

        elif dst_hashes[sha] != filename:
            olddestpath = dest_root / dst_hashes[sha]
            newdestpath = dest_root / filename
            filesystem.move(oldestpath, newdestpath)

    for sha, filename in dst_hashes.items():
        if sha not in src_hashes:
            filesystem.del(dest_root/filename)

Our top-level function now exposes two new dependencies, a reader and a filesystem
We invoke the reader to produce our files dict.
And we invoke the filesystem to apply the changes we detect.

Tip

Notice that, although we’re using dependency injection, there was no need to define an abstract base class or any kind of explicit interface. In the book we often show ABCs because we hope they help to understand what the abstraction is, but they’re not necessary. Python’s dynamic nature means we can always rely on duck typing.

Example 11. Tests using DI

class FakeFileSystem(list): #<1>

    def copy(self, src, dest): #<2>
        self.append(('COPY', src, dest))

    def move(self, src, dest):
        self.append(('MOVE', src, dest))

    def delete(self, dest):
        self.append(('DELETE', src, dest))


def test_when_a_file_exists_in_the_source_but_not_the_destination():
    source = {"sha1": "my-file" }
    dest = {}
    filesystem = FakeFileSystem()

    reader = {"/source": source, "/dest": dest}
    synchronise_dirs(reader.pop, filesystem, "/source", "/dest")

    assert filesystem == [("COPY", "/source/my-file", "/dest/my-file")]


def test_when_a_file_has_been_renamed_in_the_source():
    source = {"sha1": "renamed-file" }
    dest = {"sha1": "original-file" }
    filesystem = FakeFileSystem()

    reader = {"/source": source, "/dest": dest}
    synchronise_dirs(reader.pop, filesystem, "/source", "/dest")

    assert filesystem == [("MOVE", "/dest/original-file", "/dest/renamed-file")]

Bob loves using lists to build simple test doubles, even though his co-workers get mad. It means we can write tests like assert foo not in database
Each method in our FakeFileSystem just appends something to the list so we can inspect it later. This is an example of a Spy Object.

The advantage of this approach is that your tests act on the exact same function that’s used by your production code. The disadvantage is that we have to make our stateful components explicit and we have to pass them around. DHH famously described this as "test damage".

In either case, we can now work on fixing all the bugs in our implementation; enumerating tests for all the edge cases is now much easier.

Why Not Just Patch It Out?

At this point some of our readers will be scratching their heads and thinking "Why don’t you just use mock.patch and save yourself the effort?

We avoid using mocks in this book, and in our production code, too. We’re not going to enter into a Holy War, but our instinct is that mocking frameworks are a code smell.

Instead, we like to clearly identify the responsibilities in our codebase, and to separate those responsibilities out into small, focused objects that are easy to replace with a test double.

There’s a few, closely related reasons for that:

Patching out the dependency you’re using makes it possible to unit test the code, but it does nothing to improve the design. Using mock.patch won’t let your code work with a --dry-run flag, nor will it help you run against an ftp server. For that, you’ll need to introduce abstractions.

Designing for testability really means designing for extensibility. We trade off a little more complexity for a cleaner design that admits novel use-cases.
Tests that use mocks tend to be more coupled to the implementation details of the codebase. That’s because mock tests verify the interactions between things: did I call shutil.copy with the right arguments? This coupling between code and test tends to make tests more brittle in our experience.

Martin Fowler wrote about this in his 2007 blog post Mocks Aren’t Stubs
Over-use of mocks leads to complicated test suites that fail to explain the code.

We view TDD as a design practice first, and a testing practice second. The tests act as a record of our design choices, and serve to explain the system to us when we return to the code after a long absence.

Tests that use too many mocks get overwhelmed with setup code that hides the story we care about.

Steve Freeman has a great example of over-mocked tests in his talk Test Driven Development: That’s Not What We Meant.

You should also check out this Pycon talk on Mocking Pitfalls by our esteemed tech reviewer, Ed Jung, which addresses these issues directly.

So Which Do We Use in this Book? FCIS or DI?

Both. Our domain model is entirely free of dependencies and side-effects, so that’s our functional core. The service layer that we build around it (in [chapter_04_service_layer]) allows us to drive the system edge-to-edge and we use dependency injection to provide those services with stateful components, so we can still unit test them.

See [chapter_12_dependency_injection] for more exploration of making our dependency injection more explicit and centralised.

Wrap-up: "Depend on Abstractions."

We’ll see this idea come up again and again in the book: we can make our systems easier to test and maintain by simplifying the interface between our business logic and messy IO. Finding the right abstraction is tricky, but here’s a few heuristics and questions to ask yourself:

Can I choose a familiar Python datastructure to represent the state of the messy system, and try to imagine a single function that can return that state?

Where can I draw a line between my systems, where can I carve out a seam, to stick that abstraction in?

What are the dependencies and what is the core "business" logic?

Practice makes less-imperfect!

And now back to our regular programming…

Python Architecture Stuff: do we need more?

2019-02-11T15:28:00+00:00

I've been learning lots of new stuff about application architecture recently, and I want to get a sense from you, dear reader, of whether these ideas are of interest, and whether we should try and build some more resources (blogs, talks, etc) around it.

How should we structure an application to get the most out of our tests?

For me it all started with this question. At the end of my book I concluded on a chapter discussing how to get the most out of your tests, on the tradeoffs between unit, integration and end-to-end tests, and made some vague, flailing gestures towards topics I didn't really understand: ports and adapters, hexagonal architecture, functional core imperative shell, the clean architecture, and so on.

Since then I've managed to fall in with a tech team that are actively implementing these sorts of patterns, in Python. And the thing is, these architectural patterns are nothing new, people have been exploring them for years in the world of Java and C#. They were just new to me... and I may be over-reaching from my own experience here (I'm interested in your reaction), but they are perhaps new to the Python community in general?

It does feel like, as we mature, more and more of what were once small projects and plucky startups turn into complex business and (whisper it) enterprise software, so this stuff is likely to get more and more salient.

I came to it initially from the angle of testing, and the right kind of architecture really can help you to get the most out of your tests, by separating out a core of business logic (the "domain model") and freeing it from all infrastructure dependencies, allowing it to be tested entirely through fast, flexible unit tests. At Made it finally felt like the test pyramid was an achievable goal rather than an impossible aspiration.

Classic books on the topic (All Java.)

The classic books (Evans on DDD and Fowler on Architecture Patterns, are classic, and anyone with an interest in this stuff should read them, but if you're anything like me, then wading through all that public static void main AbstractFactoryManager gubbins is a bit wearing. Maybe some more lightweight, Pythonic intros would make it all feel a bit more plausible, a bit less enterprise-architecture-astronaut-ey?

Some existing resources in the Python world:

Made's chief architect, the Venerable Bob, has written a 4-part blog series on the way we do things here, which I really enjoyed when I first started. They're a quick, practical intro to the basic concepts of DDD, ports and adapters / dependency inversion, and to some extent, event-driven architecture. And all in Python. (trigger warning: type hints).

There's more on io.made.com but those are the main four. We'd love to get some feedback on them, what's covered well, what could do with more explanation, and so on...

And: a very timely release from last Christmas, check out Clean Architectures in Python by Leonardo Giordani. It's really two books in one, part one being an intro to TDD, but part 2 has four chapters introducing very similar patterns to the ones I'm talking about here.

I also enjoyed a talk from about a year ago by David Seddon The Rocky River, how to architect your Django monolith, showing someone else starting to think about how we go beyond the basic Django models/views/templates architecture.

There's lots more at this listing of DDD resources by Valentin Ignatev, which I came across recently on twitter. It feels like something is in the air.

Call to action: is this stuff interesting?

Bob's already had some good feedback to his blog posts, and Leonardo has had some good initial sales, so I'm sensing some interest from the Python community out there, but I'd like to sanity-check it.

Does this stuff sound interesting or relevant? Do you want to hear more?
Are you doing Python stuff that's getting beyond the bounds of "basic webapp development" or "data pipeline"? Are you finding it difficult to write fast unit tests? Are you starting to want to disentangle your business logic from whichever framework you use?
Are you doing DDD or using any of these classic patterns with Python already? Do you maybe have all the answers and want to tell me about it? Or maybe just some answers and things that have worked well for you?
Do you think this stuff all sounds hella abstract and pointless? Maybe Made.com is a bit of an outliner in the Python world, in that we're writing logistics/ERP/enterprisey software in Python, and it all feels very different from what you do day-to-day?
What do you think the Python / dynamic languages community would most benefit from in terms of new guides to these topics?

I'd love to hear from you. Comments are open, or hmu on twitter, @hjwp

Did I say read the classics? Read the classics.

Patterns of Enterprise Architecture by Martin Fowler amazon.com / .co.uk
Domain-Driven Design by Eric Evans amazon.com / .co.uk

A Pytest pattern: using "parametrize" to customise nested fixtures.

2018-10-19T14:50:00+01:00

The problem: customisable fixtures in pytest

Let's say you're running along merrily with some fixtures that create database objects for you:

@pytest.fixture
def supplier(db):
    s = Supplier(
        ref=random_ref(),
        name=random_name(),
        country="US",
    )
    db.add(s)
    yield s
    db.remove(s)



@pytest.fixture()
def product(db, supplier):
    p = Product(
        ref=random_ref(),
        name=random_name(),
        supplier=supplier,
        net_price=9.99,
    )
    db.add(p)
    yield p
    db.remove(p)

And now you're writing a new test and you suddenly realise you need to customise your default "supplier" fixture:

def test_US_supplier_has_total_price_equal_net_price(product):
    assert product.total_price == product.net_price

def test_EU_supplier_has_total_price_including_VAT(supplier, product):
    supplier.country = "FR" # oh, this doesn't work
    assert product.total_price == product.net_price * 1.2

For whatever reason, maybe because you need to set the supplier.country before you add things to the DB, or before you instantiate product objects, you need to be able to adjust the country field on your supplier feature.

Option 1: more fixtures

We can just create more fixtures, and try do do a bit of DRY by extracting out common logic:

def _default_supplier():
    return Supplier(
        ref=random_ref(),
        name=random_name(),
    )

@pytest.fixture
def us_supplier(db):
    s = _default_supplier()
    s.country = "US"
    db.add(s)
    yield s
    db.remove(s)

@pytest.fixture
def eu_supplier(db):
    s = _default_supplier()
    s.country = "FR"
    db.add(s)
    yield s
    db.remove(s)

That's just one way you could do it, maybe you can figure out ways to reduce the duplication of the db.add() stuff as well, but you are going to have to have a different, named fixture for each customisation of Supplier, and eventually you may decide that doesn't scale. us_supplier, eu_supplier, asia_supplier, ch_supplier, etc etc, too many fixtures! I'd like just one, customisable fixture please.

Option 2: factory fixtures

Instead of a fixture returning an object directly, it can return a function that creates an object, and that function can take arguments:

@pytest.fixture
def make_supplier(db):
    s = Supplier(
        ref=random_ref(),
        name=random_name(),
    )

    def _make_supplier(country):
        s.country = country
        db.add(s)
        return s

    yield _make_supplier
    db.remove(s)

The problem with this is that, once you start, you tend to have to go all the way, and make all of your fixture hierarchy into factory functions:

def test_EU_supplier_has_total_price_including_VAT(make_supplier, product):
    supplier = make_supplier(country="FR")
    product.supplier = supplier # OH, now this doesn't work, because it's too late again
    assert product.total_price == product.net_price * 1.2

And so...

@pytest.fixture
def make_product(db):
    p = Product(
        ref=random_ref(),
        name=random_name(),
    )

    def _make_product(supplier):
        p.supplier = supplier
        db.add(p)
        return p

    yield _make_product
    db.remove(p)


def test_EU_supplier_has_total_price_including_VAT(make_supplier, make_product):
    supplier = make_supplier(country="FR")
    product = make_product(supplier=supplier)
    assert product.total_price == product.net_price * 1.2

That works, but firstly now everything is a factory-fixture, which makes them more convoluted, and secondly, your tests are filling up with extra calls to make_things, and you're having to embed some of the domain knowledge of what-depends-on-what into your tests as well as your fixtures.

Option 3: "normal" fixture parametrization

This is a pretty cool feature of Pytest. You probably already know that you can parametrize tests, injecting different values for arguments to your test and then running the same test multiple times, once for each value:

@pytest.mark.parametrize('n', [1, 2, 3])
def test_doubling(n):
    assert n * 2 < 6 # will pass twice and fail once

A slightly less well-known feature is that you can parametrize fixtures as well. You need to use the special request fixture to access your parameters:

@pytest.fixture(params=['US', 'FR'])
def supplier(db, request):
    s = Supplier(
        ref=random_ref(),
        name=random_name(),
        country=request.param
    )
    db.add(s)
    yield s
    db.remove(s)

Now any test that depends on supplier, directly or indirectly, will be run twice, once with supplier.country = US and once with FR.

That's really cool for checking that a given piece of logic works in a variety of different cases, but it's not really ideal in our case. We have to build a bunch of if logic into our tests:

def test_US_supplier_has_no_VAT_but_EU_supplier_has_total_price_including_VAT(product):
    # this test is magically run twice, but:
    if product.supplier.country == 'US':
        assert product.total_price == product.net_price
    if product.supplier.country == 'FR':
        assert product.total_price == product.net_price * 1.2

So that's ugly, and on top of that, now every single test that depends (indirectly) on supplier gets run twice, and some of those extra test runs may be totally irrelevant to what the country is.

Presenting: using pytest parametrization to override nested default-value fixtures

We introduce an extra fixture that holds a default value for the country field:

@pytest.fixture()
def country():
    return "US"


@pytest.fixture
def supplier(db, country):
    s = Supplier(
        ref=random_ref(),
        name=random_name(),
        country=country,
    )
    db.add(s)
    yield s
    db.remove(s)

And then in the tests that need to change it, we can use parametrize to override the default value of country, even though the country fixture isn't explicitly named in that test:

@pytest.mark.parametrize('country', ["US"])
def test_US_supplier_has_total_price_equal_net_price(product):
    assert product.total_price == product.net_price

@pytest.mark.parametrize('country', ["EU"])
def test_EU_supplier_has_total_price_including_VAT(product):
    assert product.total_price == product.net_price * 1.2

Amazing huh? The only problem is that you're now likely to build a teetering tower of implicit dependencies where the only way to find out what's actually happening is to spend ages spelunking in conftest.py, but, hey, if you didn't like crazy nested fixture magic, why are you using pytest in the first place, right?

Reactions and alternative suggestions on a postcard please :)

This blog post inspired by a pattern I first explored at PythonAnywhere, which came up again recently; I found myself writing two successive answers to this SO post

Cross-posted at the made.com dev blog

Code samples can be found here

Speeding up Django unit tests with SQLite, keepdb and /dev/shm

2017-11-23T18:32:00+00:00

Here's the tldr version:

DATABASES = {
    'default': {
        'ENGINE': 'your db settings as normal',
        [...]
        'TEST': {
          # this gets you in-memory sqlite for tests, which is fast
          'ENGINE': 'django.db.backends.sqlite3',
        }
    }
}

if 'test' in sys.argv and 'keepdb' in sys.argv:
    # and this allows you to use --keepdb to skip re-creating the db,
    # even faster!
    DATABASES['default']['TEST']['NAME'] = '/dev/shm/myproject.test.db.sqlite3'

More context

For day-to-day development, running your tests needs to be as fast as possible to keep you in a good workflow. Although you're unlikely to be using SQLite as your production database, using it for tests in dev is often a nice shortcut, particularly since Django will use an in-memory sqlite database for tests, which is even faster than one on disk.

But, especially if you have a large and complicated database, re-creating it with each test run can take quite a bit of time. That's where the --keepdb step comes in. Of course, normally if you're using an in-memory database, keepdb doesn't make any sense because memory disappears between runs. That's where the sneaky trick of using /dev/shm comes in. In linux, /dev/shm is actually a filesystem against your machine's RAM, and it will persist between processes, until you reboot your machine.

So you get all the speed of an in-memory SQLite database, with the extra boost of not having to re-create the database.

What if the database changes,

... I hear you ask? Django is smart enough to apply any new migrations to the keepdb database if it notices them. Docs here. This works pretty well in my experience, although I have had to blow away the test db in /dev/shm manually once or twice...

Don't do this in CI

But I'm only advocating this for use in development! Ultimately, Postgres or whichever database you're using will behave differently from SQLite. Django does a good job of abstracting away 90% of those differences, but that still leaves plenty of strange edge case behaviours to do with default values, ordering and transactions that can easily trip you up.

Make sure you always run your test suite in CI against the real database.

You could use an environment variable for example, to make sure:

if os.environ.get('CI'):
    del DATABASES['default']['TEST']

Some numbers:

Here's a subset of the PythonAnywhere tests running, first without keepdb:

»»»» time ./manage.py test console
Creating test database for alias 'default'...
................................................................................
................................................................................
...........................................................
----------------------------------------------------------------------
Ran 219 tests in 5.261s

OK
Destroying test database for alias 'default'...
10.86user 0.25system 0:11.12elapsed 99%CPU

And now with keepdb:

»»»» time ./manage.py test console --keepdb
Using existing test database for alias 'default'...
................................................................................
................................................................................
...........................................................
----------------------------------------------------------------------
Ran 219 tests in 5.557s

OK
Preserving test database for alias 'default'...
6.28user 0.36system 0:06.66elapsed 99%CPU

Notice the time that Django reports is the almost the same in both cases, but the actual elapsed time is quite different -- that's because Django isn't counting time spent re-creating the database at the beginning of the test run.

(also, be aware if you're running your own tests here that you will only see an improvement the second time you run with keepdb. On the first run it's still creating the database)

Your mileage may vary

On a different project with simpler models I see very different results:

»»»» time ./manage.py test opera.tests.test_pages
[...]
Ran 65 tests in 7.620s
15.44user 0.08system 0:15.53elapsed 99%CPU

»»»» time ./manage.py test opera.tests.test_pages --keepdb
[...]
Ran 65 tests in 7.535s
14.91user 0.10system 0:15.02elapsed 99%CPU

Really not that much in it! Although that's on my laptop with a nice fast processor and SSD. Differences are more pronounced (on both projects) on a system with a slower CPU and filesystem:

$ time ./manage.py test opera.tests.test_pages
Creating test database for alias 'default'...
[...]
Ran 65 tests in 13.620s
real    0m25.720s

$ time ./manage.py test opera.tests.test_pages --keepdb
Using existing test database for alias 'default'...
[...]
Ran 65 tests in 10.648s
real    0m20.632s

Linux only!

/dev/shm only exists on Unixey operating systems. A bit of googling might help you find alternatives on Windows and MacOS though -- I can't vouch for these, but here are the first two links I found while duckduckgoing:

https://stackoverflow.com/questions/2033362/does-os-x-have-an-equivalent-to-dev-shm#2033417
https://stackoverflow.com/questions/3011464/what-is-the-dev-shm-equivalence-in-windows-system

More tips

Changing the password hashing algorithm (for tests only) can speed up user creation. This, and more tips, here.
If you really want to use Postgres on your dev machine, you can also hack your postgres config to speed things up, although, again, you'd want to make sure your config in CI was more (exactly?) like production.

Let me know if these help you!

Second edition is out! Reviews please!

2017-09-12T13:52:00+01:00

I'm still not great at selfies... Here you get two for the price of one tho.

The second edition is officially out!

Thanks to everyone involved!

If you've been reading the free online edition and feel like showing your support, then now's the time! There's not a huge amount in it for me financially (I get about a dollar off every sale) but it does add up, and O'Reilly also deserve your support, they've been great.

Reviews, please!

More importantly, I really need some reviews for the book, on Amazon and elsewhere. They make a real difference to sales, and they really help other people to make the decision over whether they will find the book useful. So if you've enjoyed the book at all, do write a review for me online. That would be very, very much appreciated.

Thanks once again, dear readers, for your kind words and encouragement throughout the process. I couldn't do it without you.

Latest release (the last big one?): Python 3.6, Django 1.11 beta

2017-03-13T06:12:00+00:00

Hi all! A brief overview of changes in the latest version, with hopefully a few pointers for people half-way through the book and looking to adapt.

Python 3.6 and Django 1.11b: minimal changes, other than f-strings

The main feature in Python 3.6 that's made an impact (for the book and, arguably, elsewhere) is f-strings.

I have to say when I first heard of them I was pretty skeptical, they sounded a bit like YOLO-ing **locals() all over your code, scary, but those concerns have occurred to the brighter minds than mine that develop Python, and they figured there was nothing truly to be scared about, so I should relax.

I'll tell you what really made me relax, which was using them! I can't count the number of times I replaced a 3-line string.format() or an incoprehensible %s substitution with an f-string one-liner, and felt my code was nicer and more readable as a result.

The move to 3.6 also forced me to upgrade to Django 1.11 at the same time, since it's the only version of Django that officially supports Python 3.6, and that was pretty much entirely painless too. No major code changes as a result, and since it's going to be an LTS, I'm hoping I'll be able to stick with it for a long while over the lifetime of this 2e of the book.

Other minor improvements: fabric3

While I was at it I decided to upgrade to a Python 3 fork of fabric (thanks to all the readers who encouraged me to do so), and that saves us from a pretty awkward hop via subprocess to Python 2. Here's the old ascii-art diagram, preserved for posterity:

## Locally:

MyListsTest
.create_pre_authenticated_session  →   .management.commands.create_session
                                       .create_pre_authenticated_session

## Against staging:

MyListsTest
.create_pre_authenticated_session      .management.commands.create_session
                                       .create_pre_authenticated_session
     ↓
                                                      ↑
server_tools
.create_session_on_server                run manage.py create_session

     ↓                                                ↑
subprocess.check_output    →    fab   →   fabfile.create_session_on_server

the old, bad way of doing it

That's now much neater in the server-side debugging chapter. Huge thanks to the fabric3 developers.

What to do if you're half-way through the book

I'm always a fan of upgrading! Because the changes are all quite minor, you should be able to just:

Install Python 3.6 (downloadable on windows + macos, can use deadsnakes ppa on Ubuntu)
rmvirtualenv superlists
re-create the virtualenv with Python 3.6 (see the pre-reqs chapter for help, particularly on windows)
pip install django==1.11b1 selenium fabric3

What about Selenium 3?

If you actually started the book before Jan 30th or so, and you haven't made the upgrade to Selenium 3 yet, that's a bigger deal, and for that I'd recommend starting again at the beginning of the book. Don't worry, you'll zip through the chapters much quicker the second time, and it should really help bed in the learning.

What did I miss?

Honestly, it does feel like it was all a bit too easy. Is there anything I missed? Any new features from Python 3.6 or Django 1.11 that you think I should have included in the book? Let me know!

Thanks, as ever, for reading.

Upgraded to Selenium 3! (and Geckodriver)

2017-02-11T10:12:00+00:00

Selenium 3 and Geckodriver

Selenium 3 came out earlier this year, and the Mozilla project, those wonderful technohippies, have been pushing the standard forward (did someone say "bleeding edge"?), and the lastest versions of Firefox will only work with the new Selenium 3 and its "geckodriver" client driver.

The project it still young and has plenty of minor bugs still, but it's functional enough, and I don't want to have to continue recommending people downgrade to Firefox 45 ESR, or (Saint RMS forbid!) switch to one of the other evil browsers, compromised as they are by corporate interests.

So selenium 3 it is!

Explicit waits all the way

"implicit waits", whereby Selenium automatically tries to wait for things to happen (eg for the page to load after a click) have always been a bit flakey, and the official line is to prefer explicit waits at all times. And unfortunately with Selenium 3 implicit waits have become all the more unreliable (or simply not implemented).

So while in the first edition I was able to avoid the topic of explicit waits all the way until about chapter 20, in the new one I have to introduce them upfront. And as a result I've had to insert a new chapter after chapter 5: the first explicit waits chapter

It did allow me to introduce two wait helpers I'm quite fond of though:

a generic wait_for function
a generic wait decorator (mmmh, Pythonic!)

Functional programming goodness!

Overall I think the book is better for it.

Lesson: avoid numerical chapter names

Inserting a new chapter after chapter 5 proved to be lots of "fun". I certainly learned the lesson of avoiding numerical names for my chapter source files! Up until this point, all the files were called things like chapter_05.html. And of course all the associated source code examples were in branches named similarly. And all the book's tests (the meta-tests if you will) have similar dependencies. So everything was off by one, and I've spent the last few weeks shaving that yak.

Also: computers.

I've enabled comments on each chapter using Disqus. By default it uses URLs as the identifier for my threads. So the recent renumbering meant about half the comments ended up on the wrong pages. Thankfully Disqus have a tool to deal with this called the "URL mapper", and "hooray", I thought, problem solved, and so I submitted a mapping a bit like this:

page7 -> page8
page8 -> page9
page9 -> page10
etc

Turns out that was a disaster. The disqus migration tool took me at my word I guess, and went down the list in order, moving all the comments from page 7 onto page 8 (can you guess what happens next?) Then it took all the comments on page 8 (which now included the ones from page 7), and moved them onto page 9... and the end result is that almost all the pages have no comments, except for the last page, which has ended up with all the 600 comments from all the other pages glommed together in a massive mess. oh no. computers eh?

Pending help from disqus (which doesn't seem to be forthcoming) it'll have to be a fresh start on comments then!

As always, your feedback is enthusiastically solicited

Follow the links above for some of the content that's specifically new to Selenium 3, and let me know what you think!

Second Edition update: Virtualenvs, Django 1.10, REST APIs, cleaner FTs...

2016-12-06T17:12:00+00:00

A brief update on my progress for the second edition.

Getting there!

Virtualenvs all the way down.

In the first edition, I made the judgement call that telling people to use virtualenvs at the very beginning of the book would be too confusing for beginners. I've decided to revisit that decision, since virtualenvs are more and more de rigueur these days. I mean, if the djangogirls tutorial is recommending one, given that it's the most beginner-friendly tutorial on Earth, then it really must be a good idea. So there's new instructions in the pre-requisite installations chapter. Let me know if you think they could be clearer.

Django 1.10

Django 1.10 doesn't introduce that many new features over 1.8, but upgrading was still pretty fiddly. Thank goodness for my extensive tests (tests for the tests in the book about testing, yes. because of course.) The main changes you'll likely to notice is in Chapter 4 where I introduce the Django Test Client, much earlier than I used to (which, through a long chain of causes, is actually because of a change to the way csrf tokens are generated). Other than that, Django 1.10 was pretty much a drop-in replacement. The main thing I'm preparing for really is the upgrade to 1.11LTS early next year.

REST APIs

I was thinking of having a couple of in-line chapters on building a REST API, but for now I've decided to have them as appendices. It starts with how to roll your own, including an example of how to test client-side ajax javascript with sinon, and then there's a second appendix on Django Rest Framework. These are both very much just skeleton outlines at the moment, but, still, feedback and suggestions appreciated.

A cleaner flow for Chapter 6

Chapter 6 is all about rewriting an app that almost works, to be one that actually works, but trying to work incrementally all along, and using the FTs to tell us when we make progress, and warn us if we introduce regressions. I used to have just the one FT, and track progress/regressions by "what line number is the FT failing at? is it higher or lower than before?". Instead I've split out one FT that tests that the existing behaviour still works, and one FT for the new behaviour, and that's much neater I think.

Next: geckodriver and Selenium 3 (uh-oh!)

There are plenty more little tweaks and nice-to-have additions I can think of (React? Docker? Oh yeah, I got your trendy topics covered), but the main task that's really outstanding is upgrading to Selenium 3 and geckodriver. And the reason that's scary is because the current status of implicit waits is up for debate, and I rely on implicit waits a lot. Introducing explicit waits earlier might be a good thing (they're currently only mentioned in Chapter 20), but it would definitely add to the learning curve in the early chapters (I think they'd have to go in chapter 4 or 5, which feels very early indeed). So I'm kinda in denial about this at the moment, hoping that maybe Mozilla will reintroduce the old behaviour, or maybe I'll build some magical wrapper around selenium that just does implicit waits for you (maybe using my stale element check trick) (in my copious spare time), or maybe switch to chromedriver, or I don't know I don't want to think about it. Suggestions, words of encouragement, moral support all welcome here.

In the meantime, I hope you enjoy the new stuff. Keep in touch!

Plans for the second edition

2016-09-25T18:52:00+01:00

The second edition was mostly prompted by the announcement by Mozilla that they were shutting down Persona in November 2016. Given that it would affect almost all the chapters from 15 thru to 21, it seemed a good excuse to do a full second edition rather than just an update.

Here, in brief, is an outline of the plan:

Chapter rewrites:

Rewrite chapters 15 + 16, replace persona with passwordless auth: first draft done
Update chapters 17+ for persona changes: in progress
Update JavaScript chapter for new version of QUnit: done
Update deployment chapters to use Systemd instead of Upstart: started but only in ansible appendix.
Two new chapters on REST APIs and Ajax: code spiked, but chapters not yet written

Minor updates + changes:

Switch to using a virtualenv from the very beginning
Upgrade to latest Django (1.10?)
Use less HTML ids and more classes
Use more early returns in FTs when refactoring partially finished user stories.

That's it, in very brief. You can read more on the google group, and feel free to join in the discussion there too, or here. Let me know what you think!

EuroPython 2016 call for beginners' day coaches.

2016-07-10T11:33:00+01:00

Coming to EuroPython? Will you be there by the Sunday before it all starts? (the 17th) Want to help some beginners to get to grips with Python, and get the most out of the conference? I need your help!

It's always surprising how many beginners come to Python conferences. The beginners' day is a full-day event aimed at helping them to get the best out of the conference -- getting them up to speed on the basics of Python, but also giving them a bit of a tour around the Python ecosystem, Python in-jokes and jargon, and what the conference is all about.

Here's an example of a happy beginner from last year:

The day is mainly structured around self-directed learning, where attendees follow one of several selected tutorials, at their own speed. Most imporantly of all, they sit at tables with coaches -- more experienced Python programmers ready to help them when they get stuck, explain things when they get confusing, and help them find solutions to problems.

(if you've ever attended a Djangogirls workshop, you'll see where I've shamelessly stolen the idea from ;)

I already have several coaches signed up, but it'd be great to have a few more. You don't have to be a Python expert, just someone that remembers what it was like to be a beginner, someone patient and kind and keen to help. Here's more on how to be a good coach.

The workshop starts at 10AM on the Sunday before the conference, the 17th.

Helping beginners is a great feeling, I hope you'll join me! Emails to obeythetestinggoat@gmail.com please :-)

Coaches needed for Python Bootcamp @ Pycon US in Portland next week.

2016-05-18T14:33:00+01:00

Coming to Pycon? Want to help some beginners to get to grips with Python, and get the most out of the conference? I need your help!

Having observed the surprising number of beginners to be found at Python conferences, I organised a "beginners' day" at last year's EuroPython in Bilbao. We had about 30 beginners attend (from a crowd of 1200 at the conference) -- quite a few science types, some people who already knew another language but wanted to learn Python, and a few total beginners. Everyone seemed to leave with lots of nice things to say about it.

Here's a couple of clips:

I'm running something similar at Pycon this year, on the first day of the tutorials:

Python Bootcamp at Pycon USA 2016, May 28th

The basic idea is shamelessly stolen from the awesome DjangoGirls, and is mainly structured around:

self-directed learning based on a selection of Python tutorials (one for total beginners, one for sciencey types, one for web developers, one for programmers who already know another language, and so on)
coached by friendly Python developers sitting near each attendee
supplemented by a few short lectures on the Python "ecosystem" (what is the cheese shop / the bdfl / a pep / pip / requests / IPython notebook / etc etc)

There's more info in the repo.

I need more coaches! I would love to get to a 3-to-1 ratio of attendees to mentors. You don't have to be an expert Python developer, you just have to be a friendly person, that remembers how tough it can be as a beginner, and that wants to help. I expect quite a few "data science" people in particular, so if you have skills in that area (even if you've only just learned them yourself), I want you!

Check out the DjangoGirls coach manual for a flavour of the kind of attitude I'm trying to cultivate. And don't forget you'll have to swear the "Oath of the Mentor", as per the link above.

Helping beginners is a great feeling, I hope you'll join me! Emails to obeythetestinggoat@gmail.com please :)

PS: This is not a free event for attendees, but I am donating my entire fee for it back to the PSF. Minus, perhaps, a small budget set aside for "thankyou" drinks and cakes for coaches :)

Your feedback required re: a new edition

2016-04-18T10:22:00+01:00

Mozilla has announced that they are retiring the Persona project, which I use in the book from chapter 15 onwards. This is sad news, because it was a great project, but also because it's going to break those chapters! So O'Reilly have suggested I write up a new edition of the book, and I'd like to get your suggestions and feedback.

It's not meant to be a wholesale rewrite, mainly just find a replacement for Persona, and then address a few improvements if I have time. So my questions to you are:

What would you replace Persona with?
What other changes or improvements would you like to see in the book?
Are there any totally new topics you'd love to see covered? (and I'm not making any promises here!)

I've kicked off a discussion on the book's mailing list, and there's talk of JavaScript MVC frameworks, a REST API, docker, and much else.

Please join the discussion!

Intermediate TDD workshops in London and Portland

2016-03-31T13:22:00+01:00

After many successful years of running my beginners' TDD/Django tutorial, I thought it might be time to have a crack at some more intermediate-level topics, so I'm announcing a couple of workshops.

the first is in London, on Monday the 4th of April, at Skillsmatter
the second is in Portland, Oregon, on Sunday the 29th of May, at Pycon

The aim will be to cover some intermediate-level topics. It assumes you've already started doing a bit of TDD, you've wrapped your head around unit testing, and some basic mocking, but you're starting to ask questions like: What order should I write tests in? When should I mock, and when should I not? What are the pros and cons of isolated unit tests vs integration tests? What do people mean when they say "let the tests drive the design"?

I'm not claiming that I'll come up with perfect answer to these questions, or solve all your problems, but we will go through a worked example that will help to illustrate some of the tradeoffs, and hopefully help you to think about how to apply this for your own projects. It's based on chapters 17 and 18 in my book.

That London one is next Monday, so book up quick!

And come prepared! You'll need:

git clone https://github.com/hjwp/book-example/ tdd-workshop
cd tdd-workshop
git fetch --tags
git checkout chapter_17
mkvirtualenv --python=python3 tdd-workshop  # or however you like to create virtualenvs
pip install -r requirements.txt
pip install selenium
python manage.py test  # these should all pass

Ask me if you have any problems! obeythetestinggoat@gmail.com

[update monday PM]

If the functional tests are being strange, try switching from firefox to chrome.

Download chromedriver from here: https://sites.google.com/a/chromium.org/chromedriver/downloads
Extract it and add it to your path
Switch webdriver.Firefox() to webdriver.Chrome() in functional_tests/base.py

Testing, async, asyncio, and performance

2015-12-27T13:29:00+00:00

I recently did some experimenting with asyncio, and wanted to report back on how I got on with writing tests for it. While I was at it I was also able to compare its performance with a couple of other approaches to mutlitasking in Python, namely threads and gevent, so I'll report on that here too. (tl;dr: it's much of a muchness).

"hobbling" naughty user processes

At PythonAnywhere we have a "tarpit" where we put users who exceed their usage limits. Their processes still run, but slower. We use cgroups for this, and it works pretty well, but I was in a mean mood and I wanted to see whether I could make our tarpit even more mean -- essentially "hobbling" naughty users's processes using OS stop and restart signals.

In synchronous/pseudo code, something like this:

def main():
    while True:
        naughty_processes = find_new_naughty_processes()
        for process in naughty_processes:
            hobble_process(process)  # parallelised somehow
        time.sleep(10)

def hobble_process(process):
    while True:
        try:
            os.kill(process.pid, signal.SIGSTOP)
            time.sleep(0.25)
            os.kill(process.pid, signal.SIGCONT)
            time.sleep(0.01)

Every 10 seconds, go fetch a list of "naughty" processes, and then "hobble" each one of them, by using OS signals to stop and start the process at short intervals. The naughty program still runs, but its execution is suspended for 95% of the time.

A first cut using asyncio

The only question was: how to do the "parallelise somehow" part. Asyncio is the hot new thing in the world of Python async stuff, and this seemed like a good potential candidate -- I have a fairly simple algorithms, and there are lots of places where I use "time.sleep", which are good places to give back control to some sort of event loop or task manager.

And sure enough, my first cut of the same code with asyncio was pleasingly similar to the normal procedural code -- I just add a few "yield froms" to signify where each function can yield control back to the event loop, ready to be woken up again when there's something for it to do:

def main():
    loop = asyncio.get_event_loop()
    loop.create_task(hobble_processes_forever())
    loop.run_forever()


@asyncio.coroutine
def hobble_processes_forever():
    already_hobbled = set()
    while True:
        yield from hobble_current_processes(already_hobbled)
        yield from asyncio.sleep(2)


@asyncio.coroutine
def hobble_current_processes(already_hobbled):
    pids = yield from get_naughty_pids()
    for pid in pids:
        if pid in already_hobbled:
            continue
        already_hobbled.add(pid)
        asyncio.get_event_loop().create_task(
            hobble_process(pid)
        )


@asyncio.coroutine
def hobble_process(pid):
    while True:
        os.kill(pid, signal.SIGSTOP)
        yield from asyncio.sleep(0.2)
        os.kill(pid, signal.SIGCONT)
        yield from asyncio.sleep(0.01)

feel free to skip this next section if you already know asyncio

What is a coroutine, I hear you ask? Or, at least, I hear some of you ask. I'm relying on the ones who don't ask to improve the amateurish definition that follows. Ahem. A coroutine is a function that defines some points at which it's happy to suspend and resume execution, or wait until some data or device becomes available. wikipedia

In asyncio, those places are marked by the yield from keywords, where the function says: I know this next thing will take some time, I'm happy to wait and let the rest of the program (as controlled by the event loop) get on with something else.

As I learned while trying to build this thing, yield from on its own won't make your code asynchronous (check out this gist for an illustration). You also need a special way of invoking functions that you want to start off asynchronously, and that's the purpose of create_task. create_task tells the event loop to start a function "in the background".

testing async code with a functional test

Once I'd more or less wrapped my head around that, and built a prototype that works, I started to think about testing. Or, how to turn my manual testing into automated testing

Some wag recently said "When people tell me they don't do TDD, I usually see them driving development with a bunch of manual tests which they're going to throw away, instead of automating them.

It felt like some "real" tests were in order, tests that would actually start some real processes and see if they really did get slowed down -- an end-to-end test, if you will -- so that's what I went for. Here was my first cut:

def test_hobbled_process_is_slow(tarpit_pids_file, start_hobbler_in_subprocess):
    timer = "; ".join([
        "import time",
        "time.sleep(0.4)",  # give hobbler a chance to spot us
        "start = time.time()",
        "list(range(int(1e6)))",  # do some work
        "print(time.time() - start)",
    ]
    normal = subprocess.check_output(['python', '-c', timer])

    add_self_to_tarpit = (
        "import os;"
        " open({pidsfile}, 'w').write(str(os.getpid()));"
    ).format(pidsfile=tarpit_pids_file)
    slow = subprocess.check_output(
        [ 'python', '-c', add_self_to_tarpit + " "  + timer],
    )

    normal = float(normal)
    slow = float(slow)
    assert normal < slow
    assert normal * 10 < slow
    assert normal * 100 > slow

The test depends on two fixtures, one to create a file containing process ids (pids) that we want to hobble, and one to actually launch the hobbler.py process

@pytest.yield_fixture
def tarpit_pids_file():
    yield tempfile.NamedTemporaryFile()


@pytest.yield_fixture
def start_hobbler_in_subprocess(tarpit_pids_file):
    process = subprocess.Popen(
        ['python3', 'hobbler.py', tarpit_pids_file],
        stdout=subprocess.PIPE, stdin=subprocess.PIPE, 
        stderr=subprocess.STDOUT, universal_newlines=True
    )
    first_line = process.stdout.readline()
    if 'Traceback' in first_line:
        raise Exception(process.stdout.read())

    yield process

    process.kill()
    print('full hobbler process output:')
    print(process.stdout.read())

If you're not familiar with py.test fixtures, they're like things you might put in a unittest setUp / tearDown; special functions that get called with each test that names them as an argument. The yield pattern I'm using here allows me to inject a resource into the test.

Running the code under test as a totally separate Python process has two benefits -- first, it lets me test the program as it will actually be used, and secondly, it neatly sidesteps one of the difficulties of testing async code, which is how to deal with the event loop itself, which has to be launched as a blocking call..

From that point onwards, I found it was relatively easy to use similar tests to drive my development, alongside a few manual checks. Here's my final list of tests:

def test_tarpit_process_is_slow(fake_tarpit, hobbler_process):
def test_spots_process(fake_tarpit, hobbler_process):
def test_spots_multiple_processes(fake_tarpit, hobbler_process):
def test_doesnt_hobble_any_old_process(fake_tarpit, hobbler_process):
def test_stops_hobbling_dead_processes(fake_tarpit, hobbler_process):
def test_hobbles_children(fake_tarpit, hobbler_process):
def test_lots_of_processes(fake_tarpit, hobbler_process):
def test_get_top_level_processes_returns_list_of_parents_and_with_chidren():

You can explore these, and the implementation, in the repo on GitHub

Performance comparison: aysncio vs gevent vs threads

Maybe the most interesting test was the "lots of processes" test, which is a performance test -- since the hobbler is meant to reduce the load on our servers, by reducing the resource usage of user processes, it's important that the hobbler itself shouldn't chew up all the CPU! So I wanted to see how it performs with lots of processes to hobble:

def test_lots_of_processes(fake_tarpit, hobbler_process):
    start_times = Process(hobbler_process.pid).cpu_times()
    print('start times', start_times)
    procs = []
    for i in range(200):
        p = subprocess.Popen(['sleep', '100'], universal_newlines=True)
        _add_to_tarpit(p.pid, fake_tarpit)
        procs.append(p)

    time.sleep(7) # time for 3 iterations

    end_times = Process(hobbler_process.pid).cpu_times()
    print('end times', end_times)

    assert end_times.user > start_times.user
    assert end_times.system > start_times.system

    Process(hobbler_process.pid).cpu_percent(interval=0.1)  # warm-up
    assert Process(hobbler_process.pid).cpu_percent(interval=2) < 10

All that boils down to starting 100 processes, telling the hobbler to hobble all of them, and then measuring the CPU usage of the hobbler -- I wanted it to be less than 10% of CPU. Unfortunately, it was far from that, taking up over 100% CPU in my first test.

So I thought I'd compare asyncio with a couple of other popular Python multitasking solutions: gevent and plain old threads

Switching from asyncio to gevent was actually very simple -- their programming patterns are quite similar -- a matter of changing a few aysncio.sleeps to gevent.sleeps, and loop.create_tasks to gevent.spawns:

-            yield from asyncio.sleep(0.01)
+            gevent.sleep(0.01)

-        loop.create_task(
-            hobble_process_tree(parent)
-        )
+        gevent.spawn(hobble_process_tree, parent)

-        yield from hobble_current_processes(loop, already_hobbled, cgroup_dir)
+        gevent.spawn(hobble_current_processes, already_hobbled, cgroup_dir)

Check out the full diff here, if you're curious:

https://github.com/hjwp/process-hobbler-experiment/commit/gevent

It turns out that didn't buy me any real performance improvement though. So I thought I'd try good ol' fashioned threads

-            yield from asyncio.sleep(0.01)
+            time.sleep(0.01)

-        loop.create_task(
-            hobble_process_tree(parent)
-        )
+        threading.Thread(
+            target=lambda: hobble_process_tree(parent)
+        ).start()

Again, not a massive change to the programming model -- a loop.create_task becomes a Thread().start(), and the asyncio.sleep() can just be a normal time.sleep(), since we're off the main thread.

Check out the full diff here if you're curious

https://github.com/hjwp/process-hobbler-experiment/commit/threads

But, performance-wise, it was no better. Worse, in fact!

Final test results: asyncio vs gevent vs threads

WARNING: NO SCIENCE HERE!

This is the results of just a single run on a single machine, not to be taken as a general indication of the true, intrinsic performance characteristics of any of these libraries, etc etc.

Library	CPU usage
asyncio:	85.4
gevent:	100.4
threading:	172.8

Feel free to try and replicate these tests yourself, using the code here

Call to action (1): can this be made more efficient?

I'm inclined to think that this whole process hobbler was just a bad idea, but maybe you know more about multitasking stuff, and you can see some obvious improvements for my code, or have some suggestions for different approaches? Any thoughts on the theoretical reasons for why asyncio should seem so much quicker than threads in this instance? Answers on a postcard please...

Call to action (2): better ways of testing async code and/or process behaviour?

I use functional tests here for a few reasons:

The code under test crosses a lot of system boundaries -- it affects other processes, it relies on operating system signals, and it reads from a file. At least one end-to-end / integrationey test felt necessary. (this feels like a good reason)
Once I'd started, it was easier to just continue in that model, rather than figure out the subtleties of factoring out my code for testability, dealing with testing the event loop, and mocking all the different layers involved. This, maybe, is a bad reason.

The price I paid was a fairly slow test suite -- the basic test of process hobbling takes 8 or 9 seconds, and any of the other tests still take on the order of a second, since they involve writing to a temp file, starting a hobbler process and a test process, waiting for the hobbler to do something, and cleaning up. The performance test takes a a good 30 seconds on its own.

The tests are also a little more flakey than I'd like -- the exact amount that a process gets hobbled is subject to quite a lot of random noise, so just checking that it's "between 10 and 100 times slower" is a bit more vague than you might want, and it took me a while to set up the test in such a way that it reliably passed and failed as appropriate.

On the other hand, of course, I get the benefit you always get from a functional test, which is that I'm sure my system really does work.

So, I'd be interested -- would you write this test suite differently? Can you see any candidates for more "unitey" tests?

You'll see I already made a start on a slightly more isolated test that aims at just the get_top_level_processes function. The other main candidate that seems like it could do with a better, more granular test is hobble_process_tree function. Specifically, it's important that the process tree should be hobbled in the correct order, starting with the top level, then down to children, then children's children, and so on, and then re-started in the reverse order, starting from the bottom-level processes and going back up to the top. (otherwise, if you're hobbling a users' interactive console session, which has a nested tree of, say, bash and ipython, the user sees really weird things if you pause the ipython before the bash.)

That's not well tested at the moment. Can you think of a way of doing so which can handle the interspered asyncio yield froms, and that still feels like testing behaviour, not implementation?

Better tests for Redis integrations with redislite

2015-12-01T08:29:00+00:00

A colleague and I were staring at some ugly, mocky tests for our redis integration the other day, when I remembered someone at Pycon last year showing me a cool library called redislite -- basically, a lightweight, self-contained, pip installable version of redis, that can be installed almost anywhere and run totally separately from the system redis. (That was Dwight, who I now realise is the main author of redislite. Yay Pycon.)

He'd said "it's great for testing", so we thought we'd give it a go.

pip installs and compiles in 20 seconds

Here's the sample usage from the docs:

>>> import redislite
>>> r = redislite.StrictRedis()
>>> r.set('foo', 'bar')
True
>>> r.get('foo')
'bar'

Couldn't be simpler! You can initiate a lightweight version of redis just like that, have one for each test even, and throw it away at the end, without having to worry about ports, passwords, or any interaction with the system redis.

The code under test is meant to parse a web server log line, and then update some hit counts in redis. Here's how our tests looked before:

@patch('hit_counter.redis')
def test_write_to_redis_updates_appropriate_keys(self, mock_redis):
    write_to_redis('www.nowhere.com', datetime(2013, 11, 14, 16, 54, 21, 0), 2323)

    self.assertItemsEqual(
        mock_redis.method_calls,
        [
            call.incr('count|www.nowhere.com'),
            call.incr('count|www.nowhere.com|2013'),
            call.incr('count|www.nowhere.com|2013-11'),
            call.incr('count|www.nowhere.com|2013-11-14'),
            call.incr('count|www.nowhere.com|2013-11-14 16'),
            call.incr('count|www.nowhere.com|2013-11-14 16:54'),
            call.incrby('bytes|www.nowhere.com', 2323),
            call.incrby('bytes|www.nowhere.com|2013', 2323),
            call.incrby('bytes|www.nowhere.com|2013-11', 2323),
            call.incrby('bytes|www.nowhere.com|2013-11-14', 2323),
            call.incrby('bytes|www.nowhere.com|2013-11-14 16', 2323),
            call.incrby('bytes|www.nowhere.com|2013-11-14 16:54', 2323),
        ]
    )

And here's how they look after:

class WriteToRedisTest(unittest.TestCase):

    def setUp(self):
        self.fake_redis = redislite.StrictRedis()
        self.patcher = patch('hit_counter.redis', self.fake_redis)
        self.patcher.start()

    def tearDown(self):
        self.patcher.stop()


    def test_increments_all_counts(self):
        timestamp = datetime(2013, 11, 14, 16, 54, 21, 0)
        expected_counts = [
            'count|www.nowhere.com',
            'count|www.nowhere.com|2013',
            'count|www.nowhere.com|2013-11',
            'count|www.nowhere.com|2013-11-14',
            'count|www.nowhere.com|2013-11-14 16',
            'count|www.nowhere.com|2013-11-14 16:54',
        ]
        for key in expected_counts:
            self.fake_redis.set(key, 11)

        write_to_redis('www.nowhere.com', timestamp, 2323)

        for key in expected_counts:
            self.assertEqual(self.fake_redis.get(key), '12')


    def test_increments_all_bytes(self):
        timestamp = datetime(2013, 11, 14, 16, 54, 21, 0)
        expected_bytes = [
            'bytes|www.nowhere.com',
        # etc

Look at that!

Instead of mocking the library, I'm mocking out a particular Redis instance, with a real, functioning redis that I have full control over
I now no longer need to care about exactly how I use the redis API in my code -- no need to check whether I'm calling incr or incrby, I can just set an actual value before, and check the actual value after. It allows me to obey one of the key rules of testing, "test behaviour, don't test implementation".

The approach only continued to deliver benefits. The reason we were looking at this code was because we wanted to start setting some expiry times on keys. As a result of using redislite, the tests for expiry came out really sane and readable:

def test_hourly_data_expires_after_one_week(self):
    timestamp = datetime(2013, 11, 14, 16, 54, 21, 0)
    write_to_redis('www.thing.com', timestamp, 123)
    ttl = int(self.fake_redis.ttl('count|www.thing.com|2013-11-14 16'))
    self.assertAlmostEqual(
        ttl,
        7 * 24 * 60 * 60,
        delta=2,
    )

Imagine that test with mocks!

    self.assertItemsEqual(
        mock_redis.method_calls,
        [
            call.incr('count|www.thing.com'),
            call.incr('count|www.thing.com|2013'),
            # ... 
            call.incr('count|www.thing.com|2013-11-14 16'),
            call.expire('count|www.thing.com|2013-11-14 16', expected_expiry),
            # and so on

Ugh. And that's glossing over the complexities of timestamping -- the assertAlmostEqual works nicely in the redislite tests, but here I'd have to either mock out datetime, or pull all of the call instances out of .method_calls one by one... Oh I just don't even want to think about it.

So, remember kids, friends don't let friends use mocks when there's a better alternative around. Definitely check out redislite if you ever find yourself writing some tests for a redis integration in your own codebase.

[thanks to Nicole for prompting me to write this post]

[OT] Autumn Leaves

2015-10-30T14:29:00+00:00

To my usual readers that come hear for TDD, apologies, this is off-topic, normal service will resume shortly.

I was in an autumnal mood yesterday, and found myself listening to 3 different versions of the jazz standard "Autumn Leaves", and since it's that sort of season, I thought I'd share them with you.

I realised I was surprisingly affected by the song, because actually I've been listening to it, in different forms, for much longer than I thought. Here are the three that stand out, in reverse chronological order of my life encounters with them.

The first version is Eva Cassidy's. The fact that I'm listening to folk singers is not something I would ever have guessed when I was twenty, call it my wife's bad influence, but I think this version is incredibly beautiful, almost perfect.

(if you like the sounds of that, be sure to check out Eva Cassidy's rendition of "somewhere over the rainbow", which is absolutely heartbreaking, particularly if you know how she died)

That was the version of the song I encountered in my 30s. Here's the version from my 20s:

Quite different! The original Coldcut mix is well worth checking out too.

I was always surprised by how affecting I found this song, whether it was in my increasingly-soppy, middle-aged 30s, or in my considerably cooler 20s, and searching around yesterday I realised why.

Autumn Leaves has been around for ages, but it's actually originally a French song, the canonical rendition of which is by classic French crooner Yves Montand, and this is the one I remember from my youth.

Now we're back to my childhood in France, and apologies to non-French speakers, I'm not sure if this will connect with y'all, but I find this one almost unbearable. It's actually based on a poem by Jacques Prevert, one famous enough that we studied it in school.

Les feuilles mortes, Jacques Prévert

In French, it's not "Autumn Leaves" though, it's "Dead Leaves" -- "Les Feuilles Mortes". Already that sets a different tone, but it's the final stanza that I think is so devastating:

Mais la vie sépare ceux qui s'aiment,

Tout doucement, sans faire de bruit.

Et la mer efface sur le sable,

Les pas des amants désunis.

Here's my rough translation:

But life separates those who love each other,

Gently, quietly,

And the sea washes away

the sandy footprints of lovers, now disunited.

From French lessons you might remember you have to do an elision between the "des" and the "amants", making it "desamants", anticipating the "desunis", making it crushingly inevitable. I think it's about the saddest thing in the world.

Enjoy!

P.S. Everything's fine between me and my wife by the way! Don't worry. Just inspired by the season...

Image credit: kat squirrelmuffins

[OT] HiDPI on Ubuntu with a Samsung Ativ 9

2014-12-14T23:39:00+00:00

This post isn't about TDD, but I just wanted to gather some notes into one place about how to get Hi-DPI working on Ubuntu. I found other sources but my hope is that this will bring all the important tips into one place.

I have a character flaw as regards overly shiny laptops, and when my golden Sony Vaio Z series finally packed in after 3 years of service, it was time to get a new one. The sensible choice would have been a Thinkpad or a Galago from System 76, but then I saw this thing, and I had to have it. It has a ridiculous 3200x1600 Hi-DPI screen, which I knew for sure was never going to work properly under Linux. But then, GNU/Linux is meant to be a bit rough on the desktop. It's all part of the fun.

Hi-DPI on Ubuntu in brief

Set two Gnome UI settings:

gsettings set org.gnome.settings-daemon.plugins.xsettings overrides "{'Gdk/WindowScalingFactor': <2>}"

gsettings set org.gnome.desktop.interface scaling-factor 2

(You can also install gnome-tweak-tool to fiddle with these, cf the "Window" section for the most important one).

In Firefox and Thunderbird, open about:config and change layout.css.devPixelsPerPx to somewhere between 1.6 and 2

(Do not touch layout.css.dpi, that's a red herring).

Then fix up your grub prompt and TTY consoles, more info on that below.

Credit to the ever-impressive Arch linux documentation for those tips.

Adjusting the grub boot menu to make it readable

You'll have noticed that the GRUB boot menu is in a ridiculously small font because of all our teeny-tiny pixels. Fix it by generating a new font in 30-point:

sudo grub-mkfont -s 30 -o /boot/grub/DejaVuSansMono.pf2 /usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf

DejaVu works well because it has all the glyphs for the line art, others may not work so well.

Edit /etc/default/grub and add a line saying GRUB_FONT=/boot/grub/DejaVuSansMono.pf2, and run update-grub to commit. There's more info here. While you're at it, why not add a delightful background image to your boot screen? Check out the Ubuntu wiki for details.

Adjusting your TTY console

One final thing that's pretty much unusable out of the box are the TTY consoles you get from pressing, eg, CTRL+ALT+F1. To change their font, you'll want to do a:

sudo dpkg-reconfigure console-setup

I picked the VGA font in 32x16, and it looks fine, if somewhat retro.

And that's your lot!

Loads of things will still look pretty wrong. Chromium is a bit broken, but I don't use it enough to warrant a lengthy investigation. I haven't tried any non-gnome apps, but presumably they will suffer.

Next challenge is getting reasonably good touchscreen support!

I leave you with a screenshot...

So. Many. Pixels.

BDD with Django and Behave

2014-10-24T20:30:00+01:00

I've started working on a new appendix for the book, which is meant to be an intro to BDD. Will you head on over to the online version of the book and tell me what you think?

BDD appendix on Chimera

If you're an existing reader, do you think it makes a good addition to the book? If you're a BDD expert, is there anything I'm getting hideously wrong?

Decorators!

2014-10-23T00:00:00+01:00

Someone recently wrote to me asking about decorators, and saying they found them a bit confusing. Here's a post based on the email I replied to them with.

The best way to understand decorators is to build a couple of them, so here are two examples for you to try out. The first is in the Django world, the second is actually a simpler, pure-python one.

Challenge: build a decorator in a simple Django app

We've built a very basic todo lists app using Django. It has views to deal with viewing lists, creating new lists, and adding to existing lists. Two of these views end up doing some similar work, which is to retrieve a list object from the database based on its list ID:

def add_item(request, list_id):
    list_ = List.objects.get(id=list_id)
    Item.objects.create(text=request.POST['item_text'], list=list_)
    return redirect('/lists/%d/' % (list_.id,))


def view_list(request, list_id):
    list_ = List.objects.get(id=list_id)
    return render(request, 'list.html', {'list': list_})

(Full code here)

This is a good use case for a decorator.

A decorator can be used to extract duplicated work, and also to change the arguments to a function. So we should be able to build a decorator that does the list-getting for us. Here's the target:

@get_list
def add_item(request, list_):
    Item.objects.create(text=request.POST['item_text'], list=list_)
    return redirect('/lists/%d/' % (list_.id,))


@get_list
def view_list(request, list_):
    return render(request, 'list.html', {'list': list_})

So how do we build a decorator that does that? A decorator is a function that takes a function, and returns another function that does a slightly modified version of the work the original function was doing. We want our decorator to transform the simplified view functions we have above, into something that looks like the original functions.

(you end up saying "function" a lot in any explanation of decorators...)

Here's a template:

def get_list(view_fn):

    def decorated_view(...?):
        ???
        return view_fn(...?)

    return decorated_view

Can you get it working? Thankfully, our code has tests, so they'll tell you when you get it right...

git clone -b chapter_06 https://github.com/hjwp/book-example
python3 manage.py test lists # dependencies: django 1.7

Some rules of thumb for decorators:

they usually contain an "internal" function definition, which ends up being what the decorator returns.
that internal function usually calls the original function.
that internal function also needs to return something.

Decorators definitely are a bit brain-melting, so it may take a bit of effort to wrap your head around it. Once you get the hang of them, they're dead useful though,

A simpler decorator challenge:

If you're finding it impossible, you could start with a simpler challenge... say, building a decorator to make functions return an absolute value:

def absolute(fn):
    # this decorator currently does nothing
    def modified_fn(x):
        return fn(x)
    return modified_fn


def foo(x):
    return 1 - x

assert foo(3) == -2


@absolute
def foo(x):
    return 1 - x

assert foo(3) == 2  # this will fail, get is passing!

Try it out:

git clone https://gist.github.com/2cc523b66d9c0fe41c4b.git deccy
python3 deccy/deccy.py

Enjoy!

[update 2014-10-23 at 3pm, see also @baroque, the decorating decorator decorator]

Test-Driving a docker-based Postgres service using py.test

2014-09-11T16:59:00+01:00

[Cross-posted on the PythonAnywhere blog]

We've been working on incorporating a Postgres database service into PythonAnywhere, and we decided to make it into a bit of a standalone project. The shiny is that we're using Docker to containerise Postgres servers for our users, and while we were at it we thought we'd try a bit of a different approach to testing. I'd be interested in feedback -- what do you like, what might you do differently?

Context: A Docker-based Postgres service

The objective is to build a service that, on demand, will spin up a Docker container with Postgres running on it, and listening on a particular port. The service is going to be controlled by a web API. We've got Flask to run the web service, docker-py to control containers, and Ansible to provision servers.

A single loop of integrated tests

Normally we use a "double-loop" TDD process, with an outside loop of functional tests that use selenium to interact with our web app, and an inner loop of more isolated unit tests. For our development of the Postgres service, we still have the outer loop of functional tests -- selenium tests that log into the site via a browser, and test the service from the perspective of the user -- clicking through the right buttons on our UI and seeing if they can get a console that connects to a new Postgres service.

But for the inner loop we were in a green field -- this wasn't going to be another app in our monolithic Django project, we wanted it to be a standalone service, one that you could package up and use in another context. It would provide all its services via an API, and need no knowledge of the rest of PythonAnywhere. So how should we write the self-contained tests for this app? Should it, in turn, have a double loop? Relying on isolated unit tests only felt like a waste of time -- after all, the whole app was basically a thin wrapper that hooks up a web service to a series of Docker commands. All boundaries. Isolated unit tests would end up being all mocks. And from a TDD-process point of view, because we'd never actually used docker-py before, we didn't know its API, so we wouldn't know what mocks to write before we'd actually decided what the code was going to look like, and tried it out. And trying it out would involve either running one of the PythonAnywhere FTs (super-slow, so a tediously and onerous feedback loop), or with manual tests, with all the uncertainty that implies.

So instead, it felt like starting with an intermediate-level layer of integrated tests might be best: we've already got our top-level UI layer full-stack tests in the form of functional tests. The next level down was the API level -- does calling this particular URL on the API actually give us a working container?

An example test

def test_create_starts_container_with_postgres_connectable(docker_cleanup):
    response = post_to_api_create()

    port = response.json()["port"]
    assert port > 1024

    connection = psycopg2.connect(
        database="postgres",
        user="pythonanywhere_helper", password="papwd",
        host="localhost", port=port,
    )
    connection.close()

Where

def post_to_api_create():
    response = requests.post(
        "http://localhost:5000/api/create",
        {"admin_password": "papwd"}
    )
    assert response.status_code == 200
    assert response.json()["status"] == "OK"
    return response

So you can see that's a very integration-ey, end-to-end test -- it does a real POST request, to a place where it expects to see an actual webapp running, and it expects to see a real, connectable database spun up and ready for it.

Now this test runs in about 10 seconds - not super-fast, like the milliseconds you might want a unit test to run in, but much faster than our FT, which takes 5 or 6 minutes. And, meanwhile, we can actually write this test first. To write an isolated, mocky test, we'd need to know the docker-py API already, and be sure that it was going to work, which we weren't.

To illustrate this point, take a look at the difference between an early implementation and a later one:

A first implementation

USER_IMAGE_DOCKERFILE = '''
FROM postgres
USER postgres
RUN /etc/init.d/postgresql start && \\
    psql -c "CREATE USER pythonanywhere_helper WITH SUPERUSER PASSWORD '{hashed}';"
CMD ["/usr/lib/postgresql/9.3/bin/postgres", "-D", "/var/lib/postgresql/9.3/main", "-c", "config_file=/etc/postgresql/9.3/main/postgresql.conf"]
'''

def get_user_dockerfile(admin_password):
    hashed = 'md5' + md5(admin_password + 'pythonanywhere_helper').hexdigest()
    return USER_IMAGE_DOCKERFILE.format(
        hashed=hashed,
    )

def create_container_with_password(password):
    tempdir = tempfile.mkdtemp()
    with open(os.path.join(tempdir, 'Dockerfile'), 'w') as f:
        f.write(get_user_dockerfile(password))

    response = docker.build(path=tempdir)
    response_lines = list(response)

    image_finder = r'Successfully built ([0-9a-f]+)'
    match = re.search(image_finder, response_lines[-1])
    if match:
        image_id = match.group(1)
    else:
        raise Exception('Image failed to build:\n{}'.format(
            '\n'.join(response_lines)
        ))

    container = docker.create_container(
        image=image_id,
    )
    return container

(These are some library functions we wrote, I won't show you the trivial flask app that calls them).

This was one of our first attempts -- we needed to be able to customise the Postgres superuser password for each user, and our initial solution involved building a new image for each user, by generating and running a custom Dockerfile for them.

We were never quite sure whether the Dockerfile voodoo was going to work, and we weren't really Postgres experts either, so having the high-level integration test, which actually tried to spin up a container and connect to the Postgres database that should be running inside it, was a really good way of getting to a solution that worked.

Imagine what a more isolated test for this code might look like:

@patch('containers.docker')
def test_uses_dockerfile_to_build_new_image(mock_docker):
    expected_dockerfile = USER_IMAGE_DOCKERFILE.format(
        'md5sekritpythonanywhere_helper'
    ).hexdigest()
    def check_dockerfile_contents(path):
        with open(os.path.join(path, 'Dockerfile')) as f:
            assert f.read() == expected_dockerfile

    mock_docker.build.side_effect = check_dockerfile_contents

    create_container_with_password('sekrit')

    assert mock_docker.build.called is True

 @patch('containers.docker')
def test_creates_container_from_docker_image(mock_docker):
    create_container_with_password('sekrit')
    mock_docker.create_container.assert_called_once_with(
        mock_docker.build.return_value
    )

There's no way we could have written that test until we actually had a working solution. And, on top of that, the test would have been totally useless when it came to evolving our requirements and our solution

A later implementation -- but minimal change to the main test

To give you an idea, here's what our current implementation looks like:

def start_new_container(storage_dirname, password, requested_port):
    prep_storage_dir(storage_dirname)
    run_command_on_temporary_container_with_mounts(
        command=['chown', '-R', 'postgres:postgres', POSTGRES_DIR],
        storage_dirname=storage_dirname,
        user='root',
    )
    run_command_on_temporary_container_with_mounts(
        command=[
            'bash', '-c', 
            INITIALISE_POSTGRES_AND_SET_PASSWORD.format(password)
        ],
        storage_dirname=storage_dirname
    )
    user_container = create_postgres_container(name=storage_dirname)
    start_container_with_storage(
        user_container, storage_dirname, 
        ports={POSTGRES_PORT: requested_port},
    )
    with open(port_file_path(storage_dirname), 'w') as f:
        f.write(str(requested_port))
    return requested_port

I won't bore you with the details of run_command_on_temporary_container_with_mounts, but one way or another we realised that building separate images for each user wasn't going to work, and that instead we were going to want to have some permanent storage mounted in from outside of Docker, which would contain the Postgres data directory, and which would effectively "save" customisations like the user's password.

So a radically different implementation, but look how little the main test changed:

def post_to_api_create(storage_dir=None, port=None):
    if storage_dir is None:
        storage_dir = uuid.uuid4()
    if port is None:
        port = random.randint(6000, 9999)
    response = requests.post(
        "https://localhost/api/containers/",
        {
            "storage_dir": storage_dir,
            "admin_password": OUR_PASSWORD,
            "port": port,
        },
        verify=False,
    )
    return response

def test_create_starts_container_with_postgres_connectable(docker_cleanup):
    response = post_to_api_create(port=6123)
    # rest of test as before!

And now imagine all the time we'd have had to spend rewriting mocks, if we'd decided to have isolated tests as well.

Aside: py.test observations

One py.test selling point is "less boilerplate". Notice that none of these tests are methods in a class, and there's no self variable. On top of that, we just use assert keywords, no complicated remembering of self.assertIn, self.assertIsNotNone, and so on. Absolutely loving that.

py.test fixtures

Another thing you may be interested in is the docker_cleanup argument to the test. py.test will magically look for a special fixture function named the same as that argument, and use it in the test. Here's how it looks:

from docker import Client
docker = Client(base_url='unix://var/run/docker.sock')

@pytest.fixture()
def docker_cleanup(request):
    containers_before = docker.containers()

    def kill_new_containers():
        current_containers = docker.containers()
        for container in current_containers:
            if container not in containers_before:
                print('killing {}'.format(container['Names'][0]))
                docker.kill(container)

    request.addfinalizer(kill_new_containers)
    return kill_new_containers

The fixture function has a couple of jobs:

It adds a "finalizer" (the equivalent of unittest addCleanup or tearDown) which will run at the end of the tests, to kill any containers that have been started by the test
It provides that same finalizer, and a helper method to identify new containers, to the tests that use the fixture, as a helper tool (I haven't showed any examples of that here though)

As it's illustrated here, there are no obvious advantages over the unittest setUp/tearDown ideas, although you can see it would make it a little easier to share setup and cleanup code between tests in different files and tests. There's a lot more to them, and if you really want to get #mindblown, go checkout out pytest yield fixtures

Incidentally, until I started using py.test I'd always associated "fixtures" with Django "fixtures", which basically meant serialized versions of model data, but really py.test is using the word in a more correct usage of the term, to mean "state that the world has to be in for the test to run properly".

The pros & cons of the "integrated-tests-only" workflow

Pros:

Allowed us to experiment freely with an API that was new to us, and get feedback on whether it was really working
Allowed us to refactor code freely, extracting helper functions etc, without needing to rewrite mocky unit tests

Cons:

Being end-to-end tests, they ran much slower than unit tests would - on the order of seconds, and later, a minute or two, once we grew from three or four tests to a dozen or two. And, on top of that...
Being integrated tests, they're not designed to run on a development machine. Instead, each code change means pushing updated source up to the server using Ansible, restarting the control webapp, and then re-running the tests in an SSH session.
Because the tests call across a web API, the code being tested runs in a different process to he test code, meaning tracebacks aren't integrated into your test results. Instead, you have to tail a logfile, and make sure you have logging set up appropriately.

Conclusions and next steps

I can potentially imagine a time when we might start to see value in a layer of "real" unit tests... So far though, there's really no "business logic" that we could extract and write fast unit tests for. Or at least, there's no business logic that I identify as such, and I'd be very pleased for someone to come along and school me about it?

On the other hand, I can definitely see a time where we might want to split out our tests for the web API from the tests for the Postgres and Docker stuff, and I can see value in a setup where a developer can run these tests locally rather than having to push code up to a dev box. Vagrant and VirtualBox might be one solution, but, honestly, installing Docker and Postgres on a dev box doesn't feel that onerous either, as long as we know we'll be testing on a "real" box in CI. Or at least, it doesn't feel onerous until we start talking about my poor laptop with its paltry 120GB SSD. No room here!

And the bonus of being able to see honest-to-God tracebacks in your test run output feels like it might be worth it.

But, overall, at this stage in development, given the almost total lack of "business logic" in our app, and given the fact that we were working with a new API and a new set of technologies -- I've found that doing without "real" unit tests has actually worked very well.

How to get Selenium to wait for page load after a click

2014-09-03T00:00:00+01:00

Oft-heard is the folorn cry...

Every so often you get bitten by a weird behaviour in one of your Selenium tests. You tell it to click a link, and then you ask it something about the new page, and it returns you something from the old page:

old_value = browser.find_element_by_id('my-id').text
browser.find_element_by_link_text('my link').click()
new_value = browser.find_element_by_id('my-id').text
assert new_value != old_value ## fails unexpectedly

(There's another example, in chapter 20 of my book)

You scratch your head, and eventually conclude Selenium must be fetching the element from the old page. "Why would it do that?!", you exclaim in a programmer-rage, "In real life, when you click on a link, you see the browser starts to load a new page, and you wait for it to load, right? That's obviously what you'd want Selenium to do too, and it should be totally trivial to implement!"

"Selenium should just wait until the page has completed loading after you click!"

browser.find_element_by_link_text('my link').click() # should just block until the next page has loaded

... with a sane timeout perhaps. There's even a document.readyState API for checking on whether a page has loaded! Grrr...

The thing is that, from the Selenium point of view, it's not that simple (and I'm grateful for David from Mozilla (@AutomatedTester) for patiently explaining this to me, more than once.)

You see, Selenium has no way of telling whether you've asked it to click on a "real" hyperlink that goes to a new URL, or whether the link goes to the same page, or whether the click is going to be intercepted by some sort of JavaScript to do some rich UI stuff on the same page.

More than that, since Selenium webdriver has become more advanced, clicks are much more like "real" clicks, which has the benefit of making our tests more realistic, but it also means it's hard for Selenium to be able to track the impact that a click has on the browsers' internals -- it might try to poll the browser for its page-loaded status immediately after clicking, but that's open to a race condition where the browser was multitasking, hasn't quite got round to dealing with the click yet, and it gives you the .readyState of the old page.

So, instead, Selenium does its best. The implicitly_wait argument will at least put a little retry loop in if you try and fetch an element that doesn't exist on the old page:

browser.implicitly_wait(3)
old_value = browser.find_element_by_id('thing-on-old-page').text
browser.find_element_by_link_text('my link').click()
new_value = browser.find_element_by_id('thing-on-new-page').text # will block for 3 seconds until thing-on-new-page appears
assert new_value != old_value

But the problem comes when #thing-on-new-page also exists on the old page. So what to do?

The "recommended" solution is an explicit wait:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions

old_value = browser.find_element_by_id('thing-on-old-page').text
browser.find_element_by_link_text('my link').click()
WebDriverWait(browser, 3).until(
    expected_conditions.text_to_be_present_in_element(
        (By.ID, 'thing-on-new-page'),
        'expected new text'
    )
)

Several problems with that though:

HIDEOUSLY UGLY *
It's not generic -- even if I do write a nice wrapper, it's tedious to have to call it every time I click on a thing, specifying a different other thing to wait for each time
And it won't work for the case when I want to check that some text stays the same between page loads.

Really, I just want a reliable way of waiting until the page has finished loading after I click on a thing. I totally understand that David and pals aren't going to provide that for me by default because they can't tell what's a Javascript click and what's a click that goes to a new page, but I know. But how to do it?

Some things that won't work

The naive attempt would be something like this:

def wait_for(condition_function):
    start_time = time.time()
    while time.time() < start_time + 3:
        if condition_function():
            return True
        else:
            time.sleep(0.1)
    raise Exception(
        'Timeout waiting for {}'.format(condition_function.__name__)
    )


def click_through_to_new_page(link_text):
    browser.find_element_by_link_text('my link').click()

    def page_has_loaded():
        page_state = browser.execute_script(
            'return document.readyState;'
        ) 
        return page_state == 'complete'

    wait_for(page_has_loaded)

The wait_for helper function is good, but unfortunately click_through_to_new_page is open to the race condition where we manage to execute the script in the old page, before the browser has started processing the click, and page_has_loaded just returns true straight away.

Our current working solution

Full credit to @ThomasMarks for coming up with this: if you keep some references to elements from the old page lying around, then they will become stale once the DOM refreshes, and stale elements cause selenium to raise a StaleElementReferenceException if you try and interact with them. So just poll one until you get an error. Bulletproof!

def click_through_to_new_page(link_text):
    link = browser.find_element_by_link_text('my link')
    link.click()

    def link_has_gone_stale():
        try:
            # poll the link with an arbitrary call
            link.find_elements_by_id('doesnt-matter') 
            return False
        except StaleElementReferenceException:
            return True

    wait_for(link_has_gone_stale)

Or, here's a genericised, sanitized version of the same thing, based on comparing Selenium's internal "IDs" for an object, and made into a nice Pythonic context manager:

[update 2014-09-06 -- see the comments, it's possible that comparing ids is not as efffective as waiting for stale reference exceptions. Will investigate, but bewarned that YMMV for now.]

class wait_for_page_load(object):

    def __init__(self, browser):
        self.browser = browser

    def __enter__(self):
        self.old_page = self.browser.find_element_by_tag_name('html')

    def page_has_loaded(self):
        new_page = self.browser.find_element_by_tag_name('html')
        return new_page.id != self.old_page.id

    def __exit__(self, *_):
        wait_for(self.page_has_loaded)

And now we can do:

with wait_for_page_load(browser):
    browser.find_element_by_link_text('my link').click()

And I think that might just be bulletproof!

And for bonus points...

(credit to Tommy Beadle for this solution)

It turns out selenium has a built-in condition called staleness_of, as well as its own wait-for implementation. Use them, alongside the @contextmanager decorator and the magical-but-slightly-scary yield keyword, and you get:

from contextlib import contextmanager
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support.expected_conditions import \
    staleness_of

class MySeleniumTest(SomeFunctionalTestClass):
    # assumes self.browser is a selenium webdriver

    @contextmanager
    def wait_for_page_load(self, timeout=30):
        old_page = self.browser.find_element_by_tag_name('html')
        yield
        WebDriverWait(self.browser, timeout).until(
            staleness_of(old_page)
        )

    def test_stuff(self):
        # example use
        with self.wait_for_page_load(timeout=10):
            self.browser.find_element_by_link_text('a link')
            # nice!

Note that this solution only works for "non-javascript" clicks, ie clicks that will cause the browser to load a brand new page, and thus load a brand new HTML body element.

Let me know what you think!

The print book is available!

2014-07-08T10:38:00+01:00

My copy of the print edition arrived in the post!

I am my own customer...

(OK that actually happened last month. This post is a little late. I was distracted by Oscons and stuff!)

Anyway, that's it, it's officially available. If you've been reading the free online version and have been holding out until the print version comes out, now's the time!

Or, if you've been waiting for the print version to come out before recommending the book to a friend, or buying a book for your local usergroup, or whatever it'll be, you know...

The online version will always be free and licensed under creative commons. I'll just link to the acknowledgements chapter, since it really wouldn't have been possible without all the help I got:

CC-licensed free version

It's been a crazy 18 months. Happy reading everyone!

Kent Beck on the limits of TDD

2014-06-03T11:28:00+01:00

For anyone that's been following the DHH / #isTDDDead controversy, or for anyone interested in what the limits of TDD are. Kent Beck (the godfather) had some really good reflections on what he was hoping to learn from the discussions:

I'm puzzled by the limits of TDD--it works so well for algorithm-y, data-structure-y code. I love the feeling of confidence I get when I use TDD. I love the sense that I have a series of achievable steps in front of me--can't imagine the implementation? no problem, you can always write a test. I recognize that TDD loses value as tests take longer to run, as the number of possible faults per test failure increases, as tests become coupled to the implementation, and as tests lose fidelity with the production environment. How far out can TDD be pushed? Are there special cases where TDD works surprisingly well? Poorly? At what point is the cure worse than the disease? How can answers to any and all of these questions be communicated effectively?

Read the full post on his facebook, and follow the next episode of #isTDDDead

Book upgraded to Django 1.7!

2014-04-02T12:23:00+01:00

In a (futile) attempt to future-proof the book, I decided to upgrade it to Django 1.7. Here's how that went down.

Overview

Unsurprisingly, the biggest change was to do with migrations. Like any new change, my initial reaction was dislike, and I resented the new things, but I think overall it's a definite improvment.

In brief, here's what happened:

The new migrations framework means 'any' change to models needs a migration, or tests won't pass.
This meant introducing the concept of migrations much earlier in the book; in fact, at the same time as I introduce the ORM. I resented this because it made the learning curve of the book steeper.
On the other hand, because the new version of migrations essentially forces you to have them from the very beginning, I was able to drop an entire chapter that was devoted to retrospectively building migrations after the first deployment, which included all sorts of checking out of old versions, and using --fake, and so on. So that's a big win.
Using step-by-step TDD also forces you to make more migrations than you really want to, if you're adding fields and constraints step-by-step. It means I have to introduce the reader early on to the idea of replacing existing migrations too, another steepening of the learning curve.

Here's some detail on the changes.

Migrations make the introduction of models.py more complex

In chapter 5, where we build the first model, the narrative used to go:

Write a test
See it fail
Add code in models.py, step by step
Get the tests further, see a different failure
Add more code in models.py
Get the tests passing

Now it goes:

Write a test
See it fail
Add code in models.py, step by step
See a database error
Create a migration
See the tests get further, see a different failure
Add more code in models.py
See a database error
Explain the concept of squashing migrations into one
Delete the existing migration and re-create it.
See the tests pass

So you can see it's more complicated. On the other hand, understanding how Django gets from models.py to the database is important. I had been just hand-waving and saying "use syncdb, and just delete the database if anything goes wrong", so maybe it's better to address this stuff head-on, rather than wait for a complicated later chapter.

If you're curious, you can view the whole narrative here

(If you're a Django core developer and you're reading this, I'd love to hear your thoughts btw. There's a few weeks before the book goes to print yet, so there's still time to tell me I'm doing it all wrong!)

I still kinda wish I could have kept my nice shallow learning curve - I expended a lot of effort with the book, in trying to make sure concepts are introduced one at a time and gradually, and now I feel I'm slightly forced to lump two concepts onto the reader at the same time. But, there's clearly an upside.

But they save me from a fairly horrible chapter 13...

It was always going to be an unlucky chapter wasn't it. Because I'd glossed over the concept of migrations until then, I would get the readers to deploy their code to a server in chapter 8 or so, and then code some new stuff, including a new database feature.

Then I had a chapter 13 in which we would try and deploy to the staging site, and see the new feature wouldn't work. So then I had to explain migrations, and go through this process:

Find the old commit that matches the point at which we did the last deployment, and check out the old version of models.py from it.
Do a manage.py schemamigration, and create a migration to match live
Check out the latest version of models.py, and do another schemamigration to get the migration we want to apply.
Test it out locally. Check out the old models.py again, delete the databse, syncdb, then run migrate 0001 --fake, then check out the new code, and run migrate, check it works
Adjust the deploy script to include migrate 0001 --fake followed by a migrate
Test deploying to staging... OK
Deploy to live
And, don't forget to now remove the migrate 001 --fake from your deploy script.

Ouch! Quite a lot of pain there! Especially when you consider that the new procedure is:

Run the deploy script. It just works, because we've had migrations all along.

:-)

Other thoughts.

I found the fact that tests would fail if you didn't have migrations intriguing, but unfortunately it's not something you can rely on. For example, in chapter 12 I introduce a unique_together constraint and test it thusly:

def test_duplicate_items_are_invalid(self):
    list_ = List.objects.create()
    Item.objects.create(list=list_, text='bla')
    with self.assertRaises(ValidationError):
        item = Item(list=list_, text='bla')
        item.full_clean()

To get that passing, I just add my unique_together constraint:

class Item(models.Model):
    text = models.TextField()
    list = models.ForeignKey(List)

    class Meta:
        unique_together = ('list', 'text')

And at this point Django doesn't warn me that I need a migration, because the test is actually happening at the validation layer.

I think that's a bit of a shame, but there's probably nothing to be done about it. It's all because the concepts of data validation and database integrity constraints are separate in Django, even though their implementation in models.py actually often happens in a single place...

One last thing...

I love the pretty colours!

How to log exceptions to stderr in Django

2014-01-23T13:35:00+00:00

Here's a common set of questions about Django:

How do I get django to log errors to stderr?
Why can't I see Django exceptions in the console?
How do I get Django to log exceptions?
How to print debug messages in Django?

I know they're common because I've often found asking myself some kind of variant on these questions, at some point or other. Then I saw that, in Django 1.6/1.7, the default logging configuration actually does send logging messages to the console

So does this mean that exceptions in Django are going to start appearing in the console then? No.

Just because you wish for it doesn't make it so

I think my own difficulties with this issue stemmed from the fact that, obviously, in my mind, the thing you'd want to log would be exceptions, and that, since Django will show us a clever debug page (if DEBUG is True), or a standard server 500 page, then I sort of assumed it would log that exception as well. But it doesn't

The reason I struggled with this because I wasn't clear on the fact that there are actually two systems involved here:

Django's logging system (based on the Python logging module)
Django's exception handling layer (middleware)

By default (as of Django 1.6), Django is configured to send logging messages to the console. There are two gotchas however:

Django's default logging level is set to WARNING, which means any attempt to use logging.debug or logging.info will fail
Django has a middleware layer that automatically catches exceptions, and handles them differently depending on whether you have DEBUG = True or not, but it doesn't explicitly log them.

Unit testing Django's logging config (if you want)

This is a blog about TDD, so let's write a test first. It works by monkey-patching in an extra view function into the project's url config (mwahahaha). The view then tries to do various calls to, eg, logging.debug, just we'd like to be able to in our real views.

Finally it explodes with an exception, so we can test whether any information about the exception ends up in the logs

We mock out sys.stderr to pick up on what was actually sent to the console.

# test assumes django 1.6 and project called 'myproj'
# also assumes this file is saved in myproj/myproj/test_logging.py,
# ie in the same folder as settings.py
from django.test import TestCase
from django.conf.urls import url
from unittest.mock import patch
import logging
import sys
from django.http import HttpResponse
from .urls import urlpatterns
from .settings  import DEBUG # django.conf.settings are messed with by test runner


def do_logging(request):
    # dummy view that tries to log at all levels, and then raises an exception
    logging.debug('debug logged')
    logging.info('info logged')
    logging.warning('warning logged')
    logging.critical('critical logged')
    raise Exception('arg')
urls_with_logging_test = urlpatterns + [url(r'^testlogging/$', do_logging)]


class LoggingTest(TestCase):

    def test_level(self):
        root_log_level = logging.getLogger().level
        if DEBUG:
            self.assertEqual(logging.getLevelName(root_log_level), 'DEBUG')
        else:
            self.assertEqual(logging.getLevelName(root_log_level), 'INFO')


    @patch('myproj.urls.urlpatterns', urls_with_logging_test)
    @patch('myproj.test_logging.sys.stderr')
    def test_logs_to_stderr(self, mock_stderr):
        try:
            self.client.get('/testlogging/')
        except:
            pass
        written = ''.join(c[0][0] for c in mock_stderr.write.call_args_list)
        print(written)
        self.assertIn('critical logged', written)
        self.assertIn('warning logged', written)
        self.assertIn('info logged', written)
        if settings.DEBUG:
            self.assertIn('debug logged', written)
        else:
            self.assertNotIn('debug logged', written)
        self.assertIn('arg', written, 'Could not see exception in logs')
        self.assertIn('do_logging', written, 'Could not see traceback info')

Setting the base log level (if you want):

Here's what you need to add to settings.py if you want to set the log level lower, so that, say, logging.info actually works. You don't actually need this to get the exception logging to work, but I discovered it while investigating this problem, so I thought I'd write it up here, since it's not obvious from the docs.

LOGGING = {
    'version': 1,
    'root': {'level': 'DEBUG' if DEBUG else 'INFO'},
}

I have a feeling someone will tell me that resetting the root log level is a silly thing to do, but it does work. By all means enlighten me if this isn't a good idea.

Using middleware to catch and log exceptions

Onto the real answer to this problem. Normally, to log an exception, you'd have something like this

try:
    do_something_that_might_explode()
except:
    logging.exception('Oh noes, it exploded')

And then logging.exception will automagically print a full traceback as well as our little message.

But how to get this to just happen, by default, for any exceptions in our code? The answer is what Django calls "middleware", code that can get run while handling any request.

import logging

class ExceptionLoggingMiddleware(object):

    def process_exception(self, request, exception):
        logging.exception('Exception handling request for ' + request.path)

Cf the Django docs on middleware, but that's really all you need!

If this is saved to, say, myproj/myproj/exception_logging_middleware.py, you would then add it to your project in settings.py by:

MIDDLEWARE_CLASSES = (
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'myproj.exception_logging_middleware.ExceptionLoggingMiddleware',
)

If you've actually used the test, you'll find it now passes.

Summary

In summary "how do I get Django to log all exceptions to stderr", which seems such a straightforward question, actually does involve several different components:

the logging module, and Django's logging config
Django's exception-handling middleware.

Whilst the former is where youd' go to choose where things get logged to (eg stderr or a file), and what the minimum log level is, the latter is the place you actually need to go to if you want to log exceptions.

Unit testing Ajax calls: if you're not using a mocking library, it's a world of pain

2013-10-19T16:05:00+01:00

tl;dr: I found myself going through increasing contortions trying to TDD some JavaScript code with Ajax in. Once I started using sinon.js, all the pain went away. Folks, don’t try to roll your own JavaScript mocks.

I’ve been playing around with Mozilla Persona as an authentication platform, and I knocked together this basic code to interact with their API. You can see it’s quite dense, but fairly readable:

var currentUser = '{{ user.email }}' || null;
var csrf_token = '{{ csrf_token }}';

navigator.id.watch({
  loggedInUser: currentUser,
  onlogin: function(assertion) {
    $.post('/accounts/login', {assertion: assertion, csrfmiddlewaretoken: csrf_token})
    .done(function() { window.location.reload(); })
    .fail(function() { navigator.id.logout();});
  },
  onlogout: function() {
    $.post('/accounts/logout')
    .always(function() { window.location.reload(); });
  }
});

We call a function called watch, passing it in an email address string, and two callbacks for login and logout. Login does a post, refreshes the page if it succeeds, and calls a logout if it fails. Logout just does a post and a refresh. Typical JavaScript, 3 levels of nested callbacks, but it actually reads through quite well

So off I go on my merry way, planning to de-spike this code and re-write it with TDD. Mockin' libraries? We don’t need no stinkin' mockin' libraries. Im’a roll my own, cos you can totes do that in JavaScript:

test("initialize binds sign in button to navigator.id.request", function () {
    var requestWasCalled = false;
    var mockRequestFunction = function() { requestWasCalled = true; };
    var mockNavigator = {
        id: {
            request: mockRequestFunction,
            watch: function () {}
        }
    };

    Superlists.Accounts.initialize(mockNavigator);
    equal(requestWasCalled, false, 'check request not called before click');

    $('#id_login').trigger('click');
    equal(requestWasCalled, true, 'check request called after click');
});

So far, so slightly-awkward-but-not-too-bad. But look how badly things go wrong once you start to try and write tests for more deeply nested callbacks:

test("initialize calls navigator.id.watch", function () {
    var user = 'current user';
    var token = 'csrf token';
    var urls = { login: 'login url', logout: 'logout url'};

    var watchFunctionCalled = false;
    var mockWatchFunction = function (params) {
        equal(params.loggedInUser, user, 'check user');
        equal(params.onlogin, Superlists.Accounts.submitAssertion, 'check login fn'); //<1>
        equal(params.onlogout, Superlists.Accounts.logOut, 'check logout fn'); //<1>
        watchFunctionCalled = true;
    };
    var mockNavigator = { id: { watch: mockWatchFunction } };

    Superlists.Accounts.initialize(mockNavigator, user, token, urls);

    equal(watchFunctionCalled, true, 'check watch function called');

});

You can see I ended up rewriting my anonymous callbacks as named functions in order to make them available to test.

Now, was this the unit tests being useful, forcing me to break up my code into smaller, more self-contained components? I’ll let you judge for yourself what you think of the readability of the new code, compared to the old code:

$(document).ready(function() {

    var accountUrls;
    var csrfToken;
    var personaNavigator;

    var initialize = function (navigator, user, token, urls){
        accountUrls = urls_;
        csrfToken = token;
        $('#id_login').on('click', function () {
            navigator.id.request();
        });

        navigator.id.watch({
            loggedInUser: user,
            onlogin: submitAssertion,
            onlogout: logOut,
        });
    };

    var submitAssertion = function (assertion) {
        $.post(
            accountUrls.login,
            { assertion: assertion, csrfmiddlewaretoken: csrfToken }
        ).done( Superlists.Accounts.refreshPage )
        .fail( Superlists.Accounts.onLoginFailure );
    };

    var logOut = function () {
        $.post(accountUrls.logout).done( Superlists.Accounts.refreshPage );
    };
    var onLoginFailure = function () {
        personaNavigator.id.logout();
    });
    var refreshPage = function () { window.location.reload(); };

    $.extend(window.Superlists, {
        Accounts: {
            initialize: initialize,
            logOut: logOut,
            onLoginFailure: onLoginFailure,
            refreshPage: refreshPage,
            submitAssertion: submitAssertion
        }
    });

});

What the heck happened? At each stage I just tried to make sane, self-contained unit tests, and I end up with this long and, I think, much less readable code! Look at all that painful yanking of variables up into a higher scope, and the contortions I’m putting myself to give them sensible names! Look at all that mess on the Superlists.Accounts namespace!

Doctor, Doctor, it hurts when I do this!

So stop doing that. Near the end of this adventure, I decided it was time to investigate a proper mocking library. sinon.js seemed popular, and guess what — it totally solved all my problems.

Look how much more readable the tests are:

test("initialize calls navigator.id.watch", function () {
    Superlists.Accounts.initialize(mockNavigator, user, token, urls);
    equal(mockNavigator.id.watch.calledOnce, true, 'check watch function called');
});


test("watch sees current user", function () {
    Superlists.Accounts.initialize(mockNavigator, user, token, urls);
    var callArgs = mockNavigator.id.watch.firstCall.args[0];
    equal(callArgs.loggedInUser, user, 'check user');
});

test("onlogin does ajax post to login url", function () {
    Superlists.Accounts.initialize(mockNavigator, user, token, urls);
    var onloginCallback = mockNavigator.id.watch.firstCall.args[0].onlogin;
    onloginCallback();
    equal(requests.length, 1, 'check ajax request');
    equal(requests[0].method, 'POST');
    equal(requests[0].url, urls.login, 'check url');
});

test("onlogin sends assertion with middleware token", function () {
    Superlists.Accounts.initialize(mockNavigator, user, token, urls);
    var onloginCallback = mockNavigator.id.watch.firstCall.args[0].onlogin;
    var assertion = 'browser-id assertion'
    onloginCallback(assertion);
    equal(
        requests[0].requestBody,
        $.param({ assertion: assertion, csrfmiddlewaretoken: token }),
        'check POST data'
    );
});

test("onlogin post failure should do navigator.id.logout ", function () {
    mockNavigator.id.logout = sinon.spy();
    Superlists.Accounts.initialize(mockNavigator, user, token, urls);
    var onloginCallback = mockNavigator.id.watch.firstCall.args[0].onlogin;
    server = sinon.fakeServer.create();
    server.respondWith([403, {}, "permission denied"]);

    onloginCallback();
    equal(mockNavigator.id.logout.called, false, 'should not logout yet');

    server.respond()
    equal(mockNavigator.id.logout.called, true, 'should call logout');
});

That last one is testing a callback nested fully 3 levels deep, and it’s still totally readable. Sinon’s fakeServer makes checking callbacks on Ajax requests a breeze. Sure, there’s still a bit too much boilerplate, the fact that .requestBody is URL-encoded and the only way to check send POST params is a little annoying for example, but with this kind of testing I can get right back to writing code the way I had it in the first place.

var initialize = function (navigator, user, token, urls) {
    $('#id_login').on('click', function () {
        navigator.id.request();
    });

    navigator.id.watch({
        loggedInUser: user,
        onlogin: function (assertion) {
            $.post(
                urls.login,
                { assertion: assertion, csrfmiddlewaretoken: token }
            )
            .done(function () { window.location.reload(); })
            .fail(function () { navigator.id.logout(); } );
        },
        onlogout: function (assertion) {
            $.post(urls.logout)
            .always(function () { window.location.reload(); });
        }
    });
};

Perfectly readable.

But isn’t TDD supposed to make you break up monolithic code blocks?

So that’s a question - one of the supposed advantages of unit testing is that, when you find yourself struggling to write tests, you often find yourself re-writing your code so that is uses several, smaller, self-contained, more easily testable components, and your code is improved as a result.

In this case though, my code was definitely not being improved — or at least, that’s what I think. Would anyone disagree?

Ultimately I think it was a matter of using the wrong tool for the job. Once I had a decent mocking library, I was able to get back to good-looking code and good-looking tests.

Another thing is that nested callbacks are just quite a natural part of client-side JavaScript. GUI / UI / Async code just isn’t the same as server-side code I guess, so I shouldn’t be surprise if it follows slightly different patterns of readability. What do you think?

Anyway folks, if you’re not using a mocking library to test your Ajax code, you should definitely try one. sinon.js worked well for me, there are others out there in the wonderfully diverse JS testing ecosystem.

More info on Mozilla Persona, unit testing javascript and TDD in my book, chapter 14 of which prompted this write-up. You should check it out. It’s got a great joke with Henry Ford in it, which I’m particularly proud of.

Testing Django Class-Based (Generic) Views

2013-09-27T16:30:00+01:00

This blog post is a first rough draft of a planned appendix to my book. It follows on from Chapter 9, which is all about forms and validation. You can take a look at it here

If you want to check out the code to have a play with the examples, you’ll find them on GitHub under the chapter_09 branch and the appendix_II branch

As you’ll see, the content starts out sounding a lot like a "proper" chapter for a book, and turns into more of a blog post and request for comments. Please do let me know what you think!

Update 2013-10-05

There’s been some interesting discussion with minds much greater than my own, such as those of Messrs Russell Keith-MaGee and Trey Hunner, which, for one reason or another, has taken place as line comments on github. Do check it out:

https://github.com/hjwp/www.obeythetestinggoat.com/commit/c5857b416e404c9af5a9205fbf1b10dccaf8161d

Update 2013-10-16

My basic conclusion for how to test CBGVs is now: make sure you have lots of short, single-assertion tests for your views, and it will be easy to adjust to using class-based views from function-based ones, and vice-versa. Cf the re-cap at the end of chapter 11:

http://chimera.labs.oreilly.com/books/1234000000754/ch11.html#_using_the_existing_lists_item_form_in_the_list_view (scroll right to the end)

And the updated version of this post / appendix:

http://chimera.labs.oreilly.com/books/1234000000754/apb.html

Appendix II: Django Class-based views

This appendix follows on from Chapter 9, in which we implemented Django forms for validation, and refactored our views. By the end of that chapter, our views were still using functions.

The new shiny in the Django world, however, is class-based views. In this chapter, we’ll refactor our application to use them instead of view functions. More specifically, we’ll have a go at using class-based generic views.

Warning

this appendix is currently more of a blog post / request for comments than a final appendix. Use at your own peril.

Class-based generic views

It’s worth making a distinction at this point, between class-based views and class-based generic views. Class-based views are just another way of defining view functions. They make few assumptions about what your views will do, and they offer one major benefit over view functions, which is that they can be subclassed. This comes, arguably, at the expense of being less readable than traditional function-based views. The main use case for plain class-based views is when you have several views that re-use the same logic. We want to obey the DRY principle. With function-based views, you would use helper functions or decorators. The theory is that using a class structure may give you a more elegant solution.

Class-based generic views are class-based views that attempt to provide ready-made solutions to common use cases: fetching an object from the database and passing it to a template, fetching a list of objects, saving user input from a POST request using a ModelForm, and so on. These sound very much like our use cases, but as we’ll soon see, the devil is in the detail.

I should say at this point that I’ve not used either kind of class-based views much. I can definitely see the sense in them, and there are potentially many use cases in Django apps where CBGVs would fit in perfectly. However, as soon as your use case is slightly outside the basics — as soon as you have more than one model you want to use, for example, I’ve found that using class-based views becomes much more complicated, and you end up with code that’s harder to read than a classic view function.

Still, because we’re forced to use a lot of the customisation options for class-based views, implementing them in this case can teach us a lot about how they work, and how we can unit tests them.

My hope is that the same unit tests we use for function-based views should work just as well for class-based views. Let’s see how we get on.

The home page as a FormView

Our home page just displays a form on a template:

def home_page(request):
    return render(request, 'home.html', {'form': ItemForm()})

Looking through the options, Django has a generic view called FormView — let’s see how that goes:

lists/views.py (ch21l001)
from django.views.generic import FormView
[...]

class HomePageView(FormView):
    template_name = 'home.html'
    form_class = ItemForm

We tell it what template we want to use, and which form. Then, we just need to update urls.py, replacing the line that used to say lists.views.home_page:

superlists/urls.py (ch21l002)
    url(r'^$', HomePageView.as_view(), name='home'),

And the tests all check out! That was easy..

$ python3 manage.py test lists
Creating test database for alias 'default'...
......................
 ---------------------------------------------------------------------
Ran 22 tests in 0.134s

OK
Destroying test database for alias 'default'...

$ python3 manage.py test functional_tests
Creating test database for alias 'default'...
....
 ---------------------------------------------------------------------
Ran 4 tests in 15.160s

OK
Destroying test database for alias 'default'...

So far so good. We’ve replaced a 1-line view function with a 2-line class, but it’s still very readable. This would be a good time for a commit…

Using form_valid to customise a CreateView

Next we have a crack at the view we use to create a brand new list, currently the new_list function. Looking through the possible CBGVs, we probably want a CreateView, and we know we’re using the ItemForm class, so let’s see how we get on with them, and whether the tests will help us:

lists/views.py
class NewListView(CreateView):
    form_class = ItemForm

def new_list(request):
    form = ItemForm(data=request.POST)
    if form.is_valid():
        list = List.objects.create()
        Item.objects.create(text=request.POST['text'], list=list)
        return redirect(list)
    else:
        return render(request, 'home.html', {"form": form})

I’m going to leave the old view function in views.py, so that we can copy code across from it. We can delete it once everything is working. It’s harmless as soon as we switch over the URL mappings, this time in:

lists/urls.py
    url(r'^new$', NewListView.as_view(), name='new_list'),

Now running the tests gives 3 errors:

$ python3 manage.py test lists
Creating test database for alias 'default'...
...................EEE
======================================================================
ERROR: test_redirects_after_POST (lists.tests.test_views.NewListTest)
 ---------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/harry/Dropbox/book/source/appendix_II/superlists/lists/tests/test_views.py", line 33, in test_redirects_after_POST
    data={'text': 'A new list item'}
    [...]
  File "/usr/local/lib/python3.3/dist-packages/django/forms/models.py", line 370, in save
    fail_message, commit, construct=False)
  File "/usr/local/lib/python3.3/dist-packages/django/forms/models.py", line 87, in save_instance
    instance.save()
  File "/home/harry/Dropbox/book/source/appendix_II/superlists/lists/models.py", line 26, in save
    self.full_clean()
  File "/usr/local/lib/python3.3/dist-packages/django/db/models/base.py", line 926, in full_clean
    raise ValidationError(errors)
django.core.exceptions.ValidationError: {'list': ['This field cannot be null.']}

======================================================================
ERROR: test_saving_a_POST_request (lists.tests.test_views.NewListTest)
 ---------------------------------------------------------------------
[...]
django.core.exceptions.ValidationError: {'list': ['This field cannot be null.']}

======================================================================
ERROR: test_validation_errors_sent_back_to_home_page_template (lists.tests.test_views.NewListTest)
 ---------------------------------------------------------------------
[...]
django.template.base.TemplateDoesNotExist: No template names provided

 ---------------------------------------------------------------------
Ran 22 tests in 0.114s

FAILED (errors=3)
Destroying test database for alias 'default'...

TODO: talk through decoding traceback.

Let’s start with the third — maybe we can just add the template?

lists/views.py
class NewListView(CreateView):
    form_class = ItemForm
    template_name = 'home.html'

That gets us down to just two failures. They’re both to do with dealing with valid POST requests. CBGVs that deal with forms want you to put any custom code for valid forms in a method called form_valid. We can just copy across some of the code from the old view function:

lists/views.py
class NewListView(CreateView):
    template_name = 'home.html'
    form_class = ItemForm

    def form_valid(self, form):
        list = List.objects.create()
        Item.objects.create(text=form.cleaned_data['text'], list=list)
        return redirect(list)

That gets us a pass!

$ python3 manage.py test lists
Ran 22 tests in 0.117s
OK
$ python3 manage.py test functional_tests
Ran 4 tests in 15.157s
OK

And we can even save two lines (DRY) by taking advantage of the real point of CBVs: inheritance!

lists/views.py
class NewListView(CreateView, HomePageView):

    def form_valid(self, form):
        list = List.objects.create()
        Item.objects.create(text=form.cleaned_data['text'], list=list)
        return redirect('/lists/%d/' % (list.id,))

And all the tests still pass.

How does it compare to the old version? I’d say that’s not bad. We save some boilerplate code, and the view is still fairly legible. So far, I’d say we’ve got one point for CBGVs, and one draw.

A more complex view to handle both viewing and adding to a list

This took me several attempts. And I have to say that, although the tests told me when I got it right, they didn’t really help me to figure out the steps to get there… Mostly it was just trial and error, hacking about in functions like get_context_data, get_form_kwargs and so on.

One thing I did do which improved my codebase was to add a new unit test:

lists/tests/test_views.py

class ListViewTest(TestCase):
    [...]

    def test_list_view_displays_form_for_existing_lists(self):
        correct_list = List.objects.create()
        response = self.client.get('/lists/%d/' % (correct_list.id,))
        self.assertIsInstance(response.context['form'], ExistingListItemForm)

It’s another good example of the "each test should test one thing" heuristic: that check on the form class could very easily have been tacked onto the end of a different test, but having it separate means I’m immediately told exactly what’s wrong, rather than potentially having the error masked by an earlier failure.

TODO: consider moving this test into ch. 9?

Anyway, after much hacking and swearing, this is the solution I eventually got to work:

lists/views.py
class ViewAndAddToList(CreateView, SingleObjectMixin):
    template_name = 'list.html'
    model = List
    form_class = ExistingListItemForm

    def get_form(self, form_class):
        self.object = self.get_object()
        if self.request.method == 'POST':
            data={
                'text': self.request.POST['text'],
                'list': self.object.id
            }
        else:
            data = None
        return form_class(data=data)

I also had to add a get_absolute_url on the Item class:

(I did try to use get_form_kwargs instead of get_form, but it didn’t want to work for me. Perhaps some CBGV expert out there has a neater solution??)

lists/models.py
class Item(models.Model):
    [...]

    def get_absolute_url(self):
        return self.list.get_absolute_url()

Compare old and new

Let’s see the old version for comparison?

def view_list(request, list_id):
    list = List.objects.get(id=list_id)

    if request.method == 'POST':
        form = ExistingListItemForm(data={
            'text': request.POST['text'],
            'list': list.id
        })
        if form.is_valid():
            form.save()
            return redirect(list)
    else:
        form = ExistingListItemForm()

    return render(request, 'list.html', {'list': list, "form": form})

Not a great improvement. Same number of lines of code, 15. If anything, the function version is better because it has one more line of whitespace. And it’s definitely more readable.

Best practices for unit testing CBGVs?

As I was working through this, I felt like my "unit" tests were sometimes a little too high-level. They told me whether I was getting things right or wrong, but they didn’t offer many clues on exactly how to fix things.

I occasionally wondered whether there might be some mileage in a test that was closer to the implementation — something like this:

def test_as_cbv(self):
    our_list = List.objects.create()
    view = ViewAndAddToList()
    view.kwargs = dict(pk=our_list.id)
    self.assertEqual(view.get_object(), our_list)

But the problem is that it requires a lot of knowledge of the internals of Django CBVs to be able to do the right test setup for these kinds of tests. And you still end up getting very confused by the complex inheritance hierarchy.

I’d be interested to hear how other people out there are testing their CBVs?

Unit testing fabric scripts for deployment

2013-09-19T07:30:00+01:00

In chapter 8 of my book I introduce a fabric script as a way of automating the deployment of our example app. You can see the section where I do so here.

After 7 chapters of building everything step-by-step with TDD and detailed unit tests, this fairly large script leaps out fully-formed, It prompted one of my readers to write in (Thanks Nick):

It didn't work the first time I used it (I retype, rather than copy-paste, to better understand the code. Occasional typos are inevitable, and unit testing is a great way to catch these small errors). With the rest of the project, there was substantial testing that went into the creation of each source file, meaning small errors were swiftly located and corrected. However, running the fabfile often caused errors in code that wasn't tested, which took me much longer to find a fix.

There should be a way to test the deployment script as thoroughly as the rest of the project code. Especially considering how critical deployment is to a web app, and how often it will likely need to be done, maintaining error-free consistency in this process seems like a critical part of the workflow, and worth writing tests for.

Nick is right isn't he? There's a bit of a disjoint here. Surely we need some kind of testing for our deployment scripts?

Admittedly in this chapter I'm explaining how to run Selenium tests against a staging site to make sure that our deployment procedure works, so our deployment script is tested indirectly, but everywhere else in the book I've also written unit tests for all my code.

At PythonAnywhere, we decided not to write tests for our fabric scripts, and we now regret it. There's about 3000 lines of messy code in there, which we're scared of refactoring.

But how to write some kind of low-level tests for a fabric script? Here's a sample function:

def _update_virtualenv(source_folder):
    virtualenv_folder = path.join(source_folder, '../virtualenv')
    if not exists(path.join(virtualenv_folder, 'bin', 'pip')): #<11>
        run('virtualenv --python=python3.3 %s' % (virtualenv_folder,))
    run('%s/bin/pip install -r %s/requirements.txt' % (
            virtualenv_folder, source_folder
    ))

It seems to me there's no great ways of testing this sort of stuff? Here's three I can think of:

1. Mock out Fabric

You could mock out the Fabric API, and write a bunch of tests that say things like:

self.assertEqual(
    mock_run.method_calls,
    [
        call('virtualenv --python=python3.3 /path/to/virtualenv'),
        call('/path/to/virtualenv/bin/pip install -r /path/to/my/folder/requirements.txt'),
    ]
)

But I hate that kind of unit test! It's just duplicating your code with a bunch of mocks. It's not that it has no value -- it's a bit like double-entry accounting, so it might help Nick or I catch the occasional typo -- but it doesn't really feel like testing, you know? It's so tightly coupled to the implementation as to be almost identical, and we'd definitely be testing implementation rather than behaviour.

I mean, would you really test-drive writing your fabric scripts using these kinds of tests?

2. Monkeypatch fabric to work in /tmp

You could write a sort of hacked-up integration test, which redirects all the fabric calls to a temp folder on the machine you're using for testing, something like this:

# assumes we have some recognisable prefix for path given
def mock_run(cmd_given):
    cmd_redirected = cmd_given.replace(
        path_prefix, tmp_path + path_prefix
    )
    # use fabric local() function to run command locally instead
    local(cmd_redirected)
#...
self.assertTrue(os.path.exists(
    tmp_path + given_path + '../virtualenv'
))

So we can check the effects of that function on the temp folder - more of an integration test than a unit test, but at least we're now testing behaviour rather than testing the implementation. But it'll involve a lot of logic to correctly mock out the fabric API... run may not be too hard, but I'm also using append and sed... I'm worried I'll spend as much time debuggin test code as the real code!

3. Run the actual commands against a test VM

You could spin up some kind of lightweight VM / linux container (docker?) and let the fabric script run against that, checking the effects using fabric too.

That might be kinda cool, but it definitely feels very heavyweight -- it's certainly more than I want to tell my readers to set up on their machines (what about Windows users!), and it ends up being a lot like what I'm doing with Selenium, running the acceptance tests against the staging server...

So what to do? One possibility would be to consign all this stuff to an appendix, in which I demonstrate all three approaches, and then tell the reader that it's up to them to choose which they like... But can I do better? And should I even bother?

So folks, what do you think? Is it worth writing unit tests for fabric scripts? If so, how?

Fast tests are useless, hot lava be damned

2013-09-11T04:00:00+01:00

tl;dr: I think there's a real danger that striving for an ultrafast test suite, will lead you to overly mocky, disjointed tests which don't help you catch bugs and don't help you code well.

At this year's DjangoCon, Casey Kinsey gave a talk in which he (at least ostensibly) advocated striving for faster test suites. As we'll find out, I've often thought that striving for fast tests suites can be a bit misguided, so I went along preparing to disagree loudly -- as anyone who knows me can attest, disagreeing loudly is just about my favourite thing to do, and a major character flaw.

As it turns out I found very little to disagree with, aside from one particularly tortured use of a Mock object which definitely made baby test sanity Jesus cry. It was an excellent talk, and full of good advice, as well as the memorable quote "The Database is Hot Lava"

But, is it?

The traditional argument: slow tests are bad

Here's the traditional argument for fast tests:

Tests that take ages to run won't get run, which will break your development process in several ways, and reduce the benefits you get from testing. You won't pick up defects early, because no-one is running the full test suite before committing, and you won't be able to get the secondary benefit which you hope to gain from TDD, which is better designed code, because you can't do TDD with slow tests.

The reference for this point of view is Gary Bernhardt's talk from Pycon 2013, entitled Fast Tests, Slow test If you haven't already, I strongly encourage you to watch that talk, Gary knows what he's talking about, whereas I suspect I don't.

update 2013-09-17 -- everyone should also watch Gary's follow-up talk, Boundaries in which he expands on the concepts of test isolation, and how it fits with trying to adopt a more functional paradigm in the core business logic of your application.

Why I think this is wrong: fast tests don't help you code, and they don't help you find bugs

I'm being deliberately provocative here, and this is a bit of a strawman, but I think the problem with turning slow tests into fast tests is that you end up with worse tests. Why is that?

faster tests mean more "unittey" tests -- tests with less dependencies
that means using more mocks
that leads to a very granular tests, which are highly decoupled from each other
and that, ultimately, doesn't help you find unexpected bugs
and it doesn't help you find regressions when you're refactoring either

So let's look at an example. Imagine our site does a bit of setup for each user: it makes a temp folder for them, and then sets a flag on their user profile:

Imagine this:

def setup_user_environment(user):
    setup_temp_storage(user)
    profile = user.get_profile()
    profile.environment_setup = True
    profile.save()

Here's the kind of test I'm inclined to write:

def test_sets_up_temp_storage_and_sets_flag_on_profile(self):
    user = User.objects.create(username='jim')
    setup_user_environment(user)
    self.assertTrue(os.path.exists('/tmp/jim')
    self.assertTrue(user.get_profile().environment_setup)

Arg! That's not a unit test at all! It touches the database, which is hot lava! Worse still, it touches the filesystem! It'll be really slow! And it has too many dependencies!

That, at least, is what the purists would say. They would prefer a "fast", test that looks like this - a "real" unit test:

def test_calls_setup_temp_storage_and_sets_flag_on_profile(self):
    mock_user = Mock()
    with patch('myapp.setup_temp_storage') as mock_setup_temp_storage:
        setup_user_environment(mock_user)
    mock_setup_temp_storage.assert_called_once_with(mock_user)
    mock_profile = mock_user.get_profile.return_value
    self.assertEqual(mock_profile.environment_setup, True)

Well, I would argue that you have a much less readable test there, and it's also a test that's very closely coupled to the implementation. It discourages refactoring, because something as simple as changing the name of the helper method setup_temp_storage involves changing the test code in 4 places -- three of which (eg mock_setup_temp_storage) won't be found by automated refactoring tools.

IMO, there's no point in a test that duplicates the implementation, line for line, with a series of mocks

Imagine I change setup_temp_storage to take a username instead of a user. I go and find its unit tests and change it, then change its implementation. What will happen next is that my unit test for setup_user_environment would break, because it uses the real function, and so that's my reminder to change the place it gets used.

In contrast, in the "fast" test, setup_user_environment is mocked, so that test will still pass, even though my code is broken.

Sure you could argue that my os.path.exists call is tightly coupled to the implementation of setup_temp_storage, and that if I end up using it in lots of tests, they'll be annoying to change, if I ever change the location of temp storage, for example. But I could factor it out into a test helper method, if I notice myself duplicating test code a lot.

What is the correct balance of unit to integration tests?

Now unit test purists and I would probably agree that this example doesn't prove you should never mock anything, or that "proper" unit tests are useless. Clearly, both are useful, and as my example clearly shows, you definitely need some level of integration tests to check that all your pieces fit together.

What I am saying is that, in a case where you can test a piece of code with either a mocky or a non-mocky test, I prefer non-mocky tests. Gary Bernhardt says you should aim for 90% unit tests vs 10% "integration" tests (which is what you'd call the kind of test I write), and I tend to think the ratio is more like 50/50.

Unit tests are definitely better for nailing down code which has lots of edge cases and possible logical pitfalls -- like the classic example of a roman numeral converter. But my assertion is: in web development, that kind of code is rare:

helper methods on models tend to be quite simple
view functions tend to be simple: get a post request, instantiate a form, save a database item if the form validates and redirect, return a template with errors if not
code tends to be hard to unit test without mocking -- the database, the Django request/response stack, or the template layer, or whatever.

I've just tended to find that there aren't many places where I find myself spelling out more than 2 or 3 tests for any given function -- in which case, I tend to find, unit tests don't offer any substantial advantage. In the example above, there's probably only one two cases, maybe a second one for the case where setup_temp_storage raises an exception.

But what about the fact that integration tests are slow? The database is hot lava! Isn't it?

The database is, at worst, lukewarm lava

NB - this is Django-specific

Remember, Django uses an in-memory Sqlite database when you're running its unit tests. It's pretty fast. Here's a test suite which hits the DB for some tests and not for others, with 1,000 tests of each:

from django.db import models
class Car(models.Model):
    colour = models.TextField()
    def get_colour(self):
        return self.colour.capitalize()

import unittest
from django.test import TestCase
from myapp.models import Car

class FastTest(unittest.TestCase):
    def create_car(self):
        return Car(colour='blue')

class SlowTest(TestCase):
    def create_car(self):
        return Car.objects.create(colour='blue')

for i in range(1000):
    method_name = 'test_car_{0}'.format(i)
    def testfn(self):
        car = self.create_car()
        self.assertEqual(car.get_colour(), 'Blue')
    setattr(FastTest, method_name, testfn)
    setattr(SlowTest, method_name, testfn)

So what's the difference in speed between the two?

$ python manage.py test myapp.FastTest
Ran 1000 tests in 0.108s

$ python manage.py test myapp.SlowTest
Ran 1000 tests in 0.311s

A factor of 3. YMMV. At most one order of magnitude, but certainly not two orders of magnitude. And notice that's 1000 tests, still running in less than a second. So, is shifting from tests that run in microseconds to tests that run in tens of microseconds really worth all the losses in terms of readability and ease of refactoring?

Unsurprisingly, it's all down to your own circumstances

I think we all have a tendency to take the solutions we've applied to our own particular circumanstances, and want to generalise them to universal rules, saying they should apply to everybody.

Casey's team didn't have a CI setup, so their only way of preventing regressions was for the individual developer to run the full suite before checking in code. Their test suite was taking 45 minutes to run, leading to developers skipping the test run and checking in broken code. You might argue that their real problem was a problem of process, but they couldn't fix that, so instead they put effort into making their tests faster, and in the process made them more efficient and better, so it was a win for them.

At work, we have also have a unit test suite that takes 45 minutes to run (many of the tests aren't very unittey, and are in fact very integrationey). So there's no way that we run all the unit tests as we do TDD (which we do for everything). Instead, we run a subset of the tests (usually the Django app we're working on), and we leave the CI system to run the full unit test suite overnight.

And you know what? The full unit test suite almost never picks up any bugs.

That's because our code is well compartmentalised, and, even though they're not very unit-ey unit tests, they are still quite granular and independent from each other.

Instead, we have a suite of 400-odd functional tests that run with Selenium; these are definitely integration/system tests, or what some people would call acceptance tests. They check every part of the application -- and they do find unexpected bugs. They take about 8 hours for a full run, so you bet we only run one or two individual FTs during day-to-day TDD.

Now, we're building a PaaS, so we have a lot of what Gary B. would call "boundaries" - a lot of dependencies on external systems: the filesystem, the database (we run a shared-hosting database-as-a-service too), Tornado + websockets, Paypal, Dropbox, github, pypi, linux chroots and cgroups, CRON, Nginx and uWSGI, and many more. There's a lot of moving parts, and ultimately the only thing that's going to reassure us that everything really works is a full-stack test.

So that's what works for us. We are pretty much forced to have a lot of slow tests, so maybe I'm just trying to justify our own specific circumstances and try and force a generalisation onto the world.

My turn to generalise from my own circumstances!

But I'm not so sure. I really think there's something to it. I think you really do lose a lot from using mocks everywhere, and I think that the price you pay in terms of test speed is sometimes worth paying if you want more "realistic" tests.

I would say this: don't optimise prematurely. Start by writing tests in the way that seems most obvious to you, tests that are as readable as possible, and don't couple themselves too tightly to the implementation with a lot of mocks. If test speed becomes a problem at some point down the road, there are plenty of smart people out there that will give you tips on how to speed them up -- just remember that mocky unit tests don't really test your application, so you'll need to keep a few end-to-end tests in there no matter what.

Over to you folks! I'm far from an expert, have only been writing tests for about 3 years, and all for the same company. Have you ever been bitten by a bug because your tests were too mocky? What ratio of unit tests to integration tests works for you?

TDD videos from EuroPython 2013, mine and others

2013-07-03T10:43:00+01:00

I can't believe the videos are already up! Here's a few links:

My usual TDD/Django Tutorial now featuring the example from the book! People's favourite bit seemed to be when I stand on the chair...

Me in a faster TDD/Python talk in Italiano. Still can't believe the organisers put me on as the first Italian talk of the whole week. I think it was some kind of evil joke.

Rob Collins' Test-Driven-Madness which I couldn't see because it was on at the same time as my tutorial, but from what he told me about it while we were sitting at the same desk doing prep around 1AM the previous day, it was pretty entertaining. I have had independent confirmation of this too.

Holger Krekel on pytest again, haven't seen this, but any man that can speak that passionately about the politics of P2P cannot possibly ever go wrong.

Great conference so far! Any more out there?

How to unit test tornado ioloop callbacks

2013-06-10T19:11:00+01:00

WARNING: this is not battle-tested wisdom of a massively experienced tornado tester. Today was the first time we ever tried to test something that actually uses the ioloop, and we've probably got it all totally backwards. Still, in case it helps...

Async. It's always hard to wrap your head around, so perhaps it's not surprising that it took us a few goes at work today before we got the hang of it.

Here's a bit of code that adds a callback to the tornado ioloop:

def sort_that_out(mess):
    IOLoop.instance().add_callback(mess.sort)

How might one naively write a test for it?

class TestSortingStuffOut(unittest.TestCase):

    def test_stuff_get_sorted(self):
        stuff = [3,2,1]
        sort_that_out(mess)
        self.assertEqual(mess, [1, 2, 3])

Well, that doesn't work:

AssertionError: Lists differ: [3, 2, 1] != [1, 2, 3]

A little head-scratching will get you to the fact that it's because the tornado IOLoop hasn't actually been started, so our callback never gets run. So, let's fix that:

def test_stuff_get_sorted(self):
    stuff = [3,2,1]
    sort_that_out(stuff)
    IOLoop.instance().start()
    self.assertEqual(stuff, [1, 2, 3])

What about now? The test hangs, and a little Ctrl-C based profiling tells us where the busy loop is:

  File "/tmp/t.py", line 13, in test_stuff_get_sorted     
    IOLoop.instance().start()
  File "/usr/local/lib/python2.7/site-packages/tornado/ioloop.py", line 627, in start
     event_pairs = self._impl.poll(poll_timeout) 
KeyboardInterrupt

Right. start() on the IOLoop is a blocking call, and just assumes the loop should be run forever. At this point we ventured over to the official tornado testing docs but they seem to suggest a lot of overcomplicated things: using a self.wait, overriding get_new_ioloop to return the singleton...

Actually, all you really need to do is this:

def test_stuff_get_sorted(self):
    stuff = [3,2,1]

    sort_that_out(stuff)

    IOLoop.instance().add_callback(IOLoop.instance().stop)
    IOLoop.instance().start()

    self.assertEqual(stuff, [1, 2, 3])

We just add our own callback, telling the loop to shut itself down, making sure that it's the last callback added before we start the loop. Voila!

.
 ---------------------------------------------------------------------- 
Ran 1 test in 0.001s

OK

Well, that was our first foray into writing a test for tornado that actually used the IOLoop (all our other tests have just mocked everything). No doubt the tornado tools come in useful for other use cases. And you'd probably want to use a tearDown or addCleanup that made sure the IOLoop got shut down even when your test doesn't behave as expected....

But I though I'd post this in case anybody else has a simple requirement to test a tornado async callback, and finds the docs a little hard-going. Hope it helps!

Using the built-in views and forms for new user registration in Django

2013-06-09T01:10:00+01:00

Have been digging into the built-in forms and views from django.contrib.auth. I always knew you could get generic views for login, logout, even password reset, but I didn't know you could actually handle new user creation as well!

There's a form in django.contrib.auth.forms, and a class-based view for creating new objects called CreateView, and I believe this is all you need for a working registration form/view:

from django.views.generic.edit import CreateView
from django.contrib.auth.forms import UserCreationForm

urlpatterns = patterns('',
    url('^register/', CreateView.as_view(
            template_name='register.html',
            form_class=UserCreationForm,
            success_url='/'
    )),
    url('^accounts/', include('django.contrib.auth.urls')),

    # rest of your URLs as normal
)

Then, in register.html (which you do have to create yourself), you get a {{ form }} you can use, including a username, password and password confirmation, and it handles validation errors and EVERYTHING.

Obviously it doesn't include registering (and validating) an email address for the user, but still, I'm pretty excited. Did everyone else already know about this?

What to say about deployment?

2013-05-15T07:37:00+01:00

My book has got to the stage of a minimum viable site. I want the next chapter to be about actually deploying the site, even though it's ridiculously early -- to encourage the habit of "deploy early, deploy often".

But how to introduce deployment in a beginner-friendly way? It's a very simple site, so we don't need to cover all the complexities of deployment, but what's the minimum? Here's what I've got so far..

Deploying a site to a live web server can be a tricky topic. Oft heard in office corridors, IRC and tech forums is the forlorn cry -- "but it works on my machine"

Some of the danger areas of deployment include:

static files (CSS, javascript, images etc): web servers usually need special configuration for serving these
the database: there can be permissions and path issues, and we need to be careful about preserving data between deploys
dependencies: we need to make sure that the packages are software relies on are installed on the server, and have the correct versions

But there are solutions to all of these. In order:

Using a staging site, on the same infrastructure as the production site, can help us test out our deployments and get things right before we go to the "real" site
We can also run our functional tests against the staging site. They could include some smoke tests that, eg, CSS is loaded correctly.
We can write a special functional test that checks the deploy process, for example making sure database data is preserved (later, we can talk about South
data migrations...)
Virtualenvs are a useful tool for managing packages and dependencies on a server that's not entirely under your own control
And finally, automation, automation, automation. By using an automated script to deploy new versions, and by using the same script to deploy to staging and production, we can reassure ourselves that staging is as much like live as possible.

So far, so good (at least, I think. feel free to pick holes!)

But the question is: now what? What platform to choose to deploy to? What should be in my automated deploy script?

Platform:

Obviousy I think PythonAnywhere is the natural choice and the easiest and the best and stuff. But I would say that. So I can't say that
Heroku / Dotcloud et al are an option, but they involve quite a lot of specific config. The precise tools used to deploy are all different, and might change by the time the book comes out
How about "a generic VPS"? Assuming the user has SSH access, I could use fabric to deploy... That would let people use AWS, Digital Ocean, Linode or whoever. But then I get into the mess of apache / nginx / uwsgi config... and the trend these days seems to be to try and let users avoid that sort of hassle... Also, I really want to make sure people actually do this, and deploy their site, and I'd rather they were able to do it somewhere free...
Can I make some instructions generic enough that they apply to all platforms? I'm not sure...

So, your suggestions much appreciated! How do you do your deploys? Fabric? FTP? Git hooks? Something else? What do you think would work as a simple solution for beginners?

[update 2013-06-09] I've had a first bash at it, and the solution involves the reader starting up their own server, provisioning nginx + gunicorn + upstart, and deploying using fabric, git and virtualenvs. comments welcomed! You can find the current draft here]

A lovely review! "like sitting down to pair-program with the author"

2013-05-13T08:35:00+01:00

Jason, whom I met at PyCon this year, wrote a lovely review of my book on the O'Reily site. I'm very grateful because I think he actually helped me to see some things I didn't realise myself about the book. I'll reproduce it here, for which I hope my publisher will forgive me:

Programming books often fall into a few, easy to define categories. There are beginner's books, advanced books, reference books, books on specific libraries/frameworks, cookbooks, etc. "Test-Driven Web Development with Python" is none of the above; it's best described as a conversation with a developer.

Reading the book is like sitting down to pair-program with the author, starting from scratch and building a basic web app. Along the way you'll obviously use Django, unit tests, and Selenium but you'll also use Git and other tools.

I find far too many programming books compartmentalize the material, each section is separate and abstract. Rather, this book's strength is in the broad use and application of these tools. By the end of this book you won't be a master with these tools but you will have used them enough to build a good foundation for starting your own projects and into the habit of test, code, refactor, commit.

For example, if you are new to open source development you'll hear about writing unit tests or using Git and want to use them yourself. But what's the next step, read a the unit test documentation or ProGit cover to cover? I don't need to be an expert, I just need someone to show me enough to get up and running. At that point I'll have completed a project and have the confidence and skills to tackle problems on my own.

Like the answer to a job interview question, the book's biggest strength is also it's biggest weakness. Writing a book that covers so many tools the author must, by necessity, make assumptions about what the reader knows, and that's very, very hard.

I came to the book without ever touching Django before. It wasn't always obvious how things worked nor why were doing them, particularly later in the book where there's more magic going on. I had to sit back and blindly follow along. At one point I got hung up because I mis-typed my URL, omitting a a forward-slash ("/") at the beginning, and my tests kept failing but I couldn't figure out why.

Do URLs start with a forward slash? As a beginner, I don't know, and it's not obvious because Django does a lot of stuff for you. The code in the book was right, but I mis-typed it. I think had I come to the table with more Django experience I wouldn't have made that mistake.

So while that's a particularly nasty typo, which really frustrated me, I can't say that a beginner should go off and read a Django book before attempting this one. In fact, quite the opposite. Because I got a survey of how everything works, covering so many tools so quickly, I'm in a much better position to troubleshoot and solve the problem myself. For example, "we've been writing a lot of unit tests, how can I write a unit test to help debug this unexpected failure?"

Lastly, as I mentioned, the book is written like a conversation with a fellow developer at your side. There are numerous jokes and cultural references (e.g. refactoring cat, testing goat, etc.). This really lightens the tone (who doesn't joke with a co-worker?) as well as introduce the reader into the culture of open source development so that when you are walking around PyCon and someone has a stuffed goat or makes a "baaah" goat noise you'll fit right in!

Writing the book in the style of a pair-programming conversation came naturally, since it's what I do every day. It's also the way that I learnt pretty much everything I'm writing about, so it's natural that I should use it as a teaching style. Where Jason really has a point is that my book isn't enough -- it's not a reference book, so you won't be able to use it as a reference book. It won't teach you everything about Django, or even everything about unit testing -- on several occasions in the book I say "you should go and read the Django documentation", or "you should take a look at the other assertion methods from the unittest module", and I think some readers have found that frustrating. "Why should I go elsewhere to learn?"...

So I think this is something I need to set expectations about better, perhaps in the intro. "This isn't a reference book".

Any thoughts? From current readers, do you agree with Jason? Do you think the current style and content of the book are working?

Day against DRM and this book's copyright

2013-05-03T18:35:00+01:00

Hooray for O'Reilly, who are giving away ebooks half-price in celebration of the FSF's "Day against DRM"!

When I was first speaking to publishers about this book, I was a bit nervous. I'm very intersed in digital rights and copyright reform, and in fact I'm a reasonably active member of Pirate Party UK (or I was until this little project sucked up all my spare time!)... Could I find a publishing deal that would be compatible with my views? One of my colleagues in the German Pirate Party came in for a lot of criticism when she wrote a book and discovered her publisher sending out DMCA notices to filesharing sites...

I support a manifesto that asks for a 10-year limit on copyright, and for all non-commercial filesharing to be legal. Could I find a publishing deal that was at all consistent with that? The first publisher I spoke to was having none of it...

That's why I was so glad when O'Reilly said they had absolutely no problem with me distributing the book under a CC license... After a bit of haggling, they even agree to a clause in the contract that would say that the book would revert to a CC-0 Public Domain license after 10 years. Even if you think the policy of 10 years is too short for many domains, it's certainly enough for commercial exploitation of a technical book.

So, hooray for my beloved publisher, hooray for Creative Commons, and hooray for the FSF!

"Test Driven Development of Web Applications" -- the book exists, and is available on Early Release

2013-03-26T18:35:00+00:00

I can't quite believe this is actually happening, but I have not only signed a contract with O'Reilly, but i have actually written the first four chapters, or at least a first draft of the first four chapters!

For anyone new, the idea is to provide an introduction to TDD for web app development, for beginners -- whether it's people who've never done web development and Django before, or just people who've never done TDD.

I'm releasing the book as early as possible to try and gather feedback, so please, please do get yourself a copy, and tell me what I'm doing right and wrong! What would you change? Am I pitching it at the right level? Is the pace right or wrong? Should I spend more or less time on explaining the basics of Django, more or less time on the basics of TDD, more or less time with detailed examples?

Here's the book on the O'Reilly Early Release program:

Test-Driven Development with Python

Buying direct from O'Reilly is probably the best option, they have DRM-free versions you can download in all sorts of formats (MOBI/epub/pdf)...

Alternatively, if you have an account on Safari, you can get it there too

Remember, this is an early draft, so expect the occasional rough edges... But I really, really want your feedback folks, so do let me know what you think -- here in the comments, in the comments on either site, or just by email to hjwp2@cantab.net.

Thanks to everyone for your kind words of support, and thanks in advance for your help!

[update] -- in case it's not clear: if you buy this book now, either via O'Reilly or via Safari, you'll get updates from now on, as each new chapter is released, until the final version is released and you get the full copy. Anyone that provides feedback is totes getting a credit in the final book!

[update 2] -- naturally I assume you're all more than keen to pay for this book, but just as a matter of principle, there will be a Creative Commons, "try-before-you-buy", free version of the early drafts, released sometime in the near future. The final version of the book will also have a Creative Commons (NC-ND) digital version released on publication. And after 10 years, it will automatically revert to CC-0 Public Domain. But I (and my publishers) very much appreciate every single early sale!

[update 3] -- follow me on twitter @hjwp or sign up for this blog's updates to find out more about progress with the book, including when the CC version comes out... And, just let me know you're interested!

"TDD / Django tutorial: The Book!"

2013-03-03T00:00:00+00:00

[update April 2012] : The book is now coming out on O'Reilly! more info here:

http://www.tdd-django-tutorial.com/blog/articles/2013/test-driven-development-web-applications-book-exis/

Thanks so much for following on this far! I'm afraid that's all there is for now, but I am about to start on the next stage -- a proper book on TDD for web apps.

If you've found the tutorial useful so far, I wonder whether I can solicit some feeback regarding a book version?

Here's the chapter outline I've got so far. Now, remember, this is a very rough draft, and it's very much bound to change, but I'd love to hear your thoughts... Especially about some quite general questions, like:

Am I broadly covering the right sort of stuff?
Am I right to spend plenty of time talking about JavaScript, as well as Python?
What should I choose as my example app? I've written the outline as if it were a forums app, but I'm starting to see the appeal of one of the other classic examples, the "to-do list" (very simple at base, can be extended easily, opportunities for sharing/social bits, plenty of stuff to do on the client-side too...). What would you choose as an example?

PART 1 - Beginning web app development with TDD

The idea is to dive straight in with a practical example, rather than talking a lot of theory up-front. I had originally thought to start with a toy example (cf these 3 abandoned chapters: http://www.tdd-django-tutorial.com/tutorial/6/ ), but I decided that people prefer real practical stuff to toy examples...

I also want the first few chapters to be very short + bit-sized, so that the reader feels like they're really making progress quickly... (the inspiration comes from Kent Beck's TDD by Example, an awesome book)

Essentially part 1 is a re-hash of this online tutorial, using a differnt example app...

1: Our first functional test with Selenium

Briefly discuss difference between functional testing (AKA acceptance testing, integration testing, whatever) and unit testing
Write first test - Introduce Selenium, setUp, tearDown
Demonstrate we can get it to open a web browser, and navigate to a web page eg - google.com

2: Getting Django set-up and running

Change our test to look for the test server
Switch to Django LiveServerTestCase. Explain
Get the first test running and failing for a sensible reason
Create django project django-admin.py startproject
It worked!

3: A static front page

Look for "Welcome to the Forums", or similar
urls.py, direct_to_template ?

4: Super-users and the Django admin site

Extend FT to try and log in
Explain the admin site
Database setup, settings.py, syncdb, admin.py
runserver to show login code
Explain difference between test database and real database
Fixtures

5: First unit tests and Database model

Distinction between unit tests and functional tests
Extend FT to try and create a new topic
new app
models.py
test/code cycle

6: Testing a view

urls.py again
Test view as a function
assert on string contents

7: Django's template system

Introduce template syntax
Keep testing as a function
The, introduce the Django Test Client

8: Reflections: what to test, what not to test

time for a bit of theory/philosophy
"Don't test constants"
Test logic
Tests for simple stuff should be simple, so not much effort

9: Simple Forms

Manually coded HTML
Refactor test classes

10: User Authentication

Sign up, login/logout
Email?

11: More advanced forms

Use Django Forms classes

12: On Refactoring

Martin Fowler
Tests critical
Methodical process - explain step by step

13: Pagination

Extend various old unit tests and FTs

PART 2: More advanced testing for a more advanced site

14: Notifications

Django Notifications, for post edits

15: Adding style with MarkDown

Using an external library

16: Switching to OAuth: Mocking

"Don't store passwords"
Discuss challenges of external dependencies

17: Getting Dynamic: Testing Javascript part 1

Simple input validation
Choose JS unit testing framework (probably Qunit, or YUI)

18: Testing Javascript part 2 - Ajax

Dynamic previews of post input

19: Getting pretty: Bootstrap

Bring in nicer UI elements

20: Getting pretty: Gravatar

pictures for users

PART 3: Getting seriously sexy

21: Getting serious about the client-side + single-page website?

Introduce one of the client-side js frameworks -- backbone.js / ember.js / angular

22: Switching Databases 1: PostgreSQL

show how Django makes this easy

23: Websockets and Async on the server-side

we want dynamic notifications of when new posts appear on a thread we're looking at
Need to spin up, Tornado/Twisted/Gevent as well as Django LiveServerTestCase
FT opens multiple browser tabs in parallel
Big change!

24: Switching Databases 2: NoSQL and MongoDB

obligatory discussion of NoSQL and MongoDB
descrine installation, particularities of testing

26: Continuous Integration

Need to build 3 server types
Jenkins (or maybe buildbot)
Need to adapt Fts, maybe rely less on LiveServerTestCase

27: Caching for screamingly fast performance

unit testing memcached
Functionally testing performance
Apache ab testing

Well, that's what I have so far. What do you think? Have I missed anything out? Does anything seem superfluous? Most importantly, would you buy it?

The next stage: writing a book

2013-01-08T00:00:00+00:00

I've decided to try and develop this tutorial into an actual book on test-driven development for web apps. Have a look at the chapter outline I've posted as part 6:

http://www.tdd-django-tutorial.com/tutorial/6/

I'd love to hear what you think! Am I completely mad?