Working Incrementally
Now let’s address our real problem, which is that our design only allows for one global list. In this chapter I’ll demonstrate a critical TDD technique: how to adapt existing code using an incremental, step-by-step process which takes you from working state to working state. Testing Goat, not Refactoring Cat!
Small Design When Necessary
Let’s have a think about how we want support for multiple lists to work.
At the moment the only URL for our site is the homepage, and that’s why there’s only one global list. The most obvious way to support multiple lists is to say that each list gets its own URL, so that people can start multiple lists, or so that different people can have different lists. How might that work?
Not Big Design Up Front
TDD is closely associated with the agile movement in software development, which includes a reaction against Big Design Up Front, the traditional software engineering practice whereby, after a lengthy requirements gathering exercise, there is an equally lengthy design stage where the software is planned out on paper. The agile philosophy is that you learn more from solving problems in practice than in theory, especially when you confront your application with real users as soon as possible. Instead of a long up-front design phase, we try to put a minimum viable application out there early, and let the design evolve gradually based on feedback from real-world usage.
But that doesn’t mean that thinking about design is outright banned! In the last big chapter we saw how just blundering ahead without thinking can eventually get us to the right answer, but often a little thinking about design can help us get there faster. So, let’s think about our minimum viable lists app, and what kind of design we’ll need to deliver it:
-
We want each user to be able to store their own list—at least one, for now.
-
A list is made up of several items, whose primary attribute is a bit of descriptive text.
-
We need to save lists from one visit to the next. For now, we can give each user a unique URL for their list. Later on we may want some way of automatically recognising users and showing them their lists.
To deliver the "for now" items, we’re going to have to store lists and their items in a database. Each list will have a unique URL, and each list item will be a bit of descriptive text, associated with a particular list.
YAGNI!
Once you start thinking about design, it can be hard to stop. All sorts of other thoughts are occurring to us—we might want to give each list a name or title, we might want to recognise users using usernames and passwords, we might want to add a longer notes field as well as short descriptions to our list, we might want to store some kind of ordering, and so on. But we should obey another tenet of the agile gospel: "YAGNI" (pronounced yag-knee), which stands for "You ain’t gonna need it!" As software developers, we have fun creating things, and sometimes it’s hard to resist the urge to build things just because an idea occurred to us and we might need it. The trouble is that more often than not, no matter how cool the idea was, you won’t end up using it. Instead you have a load of unused code, adding to the complexity of your application.
YAGNI is the mantra we use to resist our overenthusiastic creative urges.
REST (ish)
We have an idea of the data structure we want—the Model part of
Model-View-Controller (MVC). What about the View and Controller parts?
How should the user interact with Lists and their Items using a web browser?
Representational State Transfer (REST) is an approach to web design that’s usually used to guide the design of web-based APIs. When designing a user-facing site, it’s not possible to stick strictly to the REST rules, but they still provide some useful inspiration (skip ahead to [appendix_rest_api] if you want to see a real REST API).
REST suggests that we have a URL structure that matches our data structure, in this case lists and list items. Each list can have its own URL:
/lists/<list identifier>/
That will fulfill the requirement we’ve specified in our FT. To view a list, we use a GET request (a normal browser visit to the page).
To create a brand new list, we’ll have a special URL that accepts POST requests:
/lists/new
To add a new item to an existing list, we’ll have a separate URL, to which we can send POST requests:
/lists/<list identifier>/add_item
(Again, we’re not trying to perfectly follow the rules of REST, which would use a PUT request here—we’re just using REST for inspiration. Apart from anything else, you can’t use PUT in a standard HTML form.)
In summary, our scratchpad for this chapter looks something like this:
Implementing the New Design Incrementally Using TDD
How do we use TDD to implement the new design? Let’s take another look at the flowchart for the TDD process in The TDD Process With Both Functional and Unit Tests.
At the top level, we’re going to use a combination of adding new functionality (by adding a new FT and writing new application code), and refactoring our application—that is, rewriting some of the existing implementation so that it delivers the same functionality to the user but using aspects of our new design. We’ll be able to use the existing functional test to verify we don’t break what already works, and the new functional test to drive the new features.
At the unit test level, we’ll be adding new tests or modifying existing ones to test for the changes we want, and we’ll be able to similarly use the unit tests we don’t touch to help make sure we don’t break anything in the process.
Ensuring We Have a Regression Test
Let’s translate our scratchpad into a new functional test method, which introduces a second user and checks that their to-do list is separate from Edith’s.
We’ll start out very similarly to the first. Edith adds a first item to create a to-do list, but we introduce our first new assertion—Edith’s list should live at its own, unique URL:
def test_can_start_a_todo_list(self):
# Edith has heard about a cool new online to-do app.
[...]
# Satisfied, she goes back to sleep
def test_multiple_users_can_start_lists_at_different_urls(self):
# Edith starts a new to-do list
self.browser.get(self.live_server_url)
inputbox = self.browser.find_element(By.ID, "id_new_item")
inputbox.send_keys("Buy peacock feathers")
inputbox.send_keys(Keys.ENTER)
self.wait_for_row_in_list_table("1: Buy peacock feathers")
# She notices that her list has a unique URL
edith_list_url = self.browser.current_url
self.assertRegex(edith_list_url, "/lists/.+") (1)| 1 | assertRegex is a helper function from unittest
that checks whether a string matches a regular expression.
We use it to check that our new REST-ish design has been implemented.
Find out more in the unittest documentation.
|
Next we imagine a new user coming along. We want to check that they don’t see any of Edith’s items when they visit the home page, and that they get their own unique URL for their list:
[...]
self.assertRegex(edith_list_url, "/lists/.+")
# Now a new user, Francis, comes along to the site.
## We delete all the browser's cookies
## as a way of simulating a brand new user session (1)
self.browser.delete_all_cookies()
# Francis visits the home page. There is no sign of Edith's
# list
self.browser.get(self.live_server_url)
page_text = self.browser.find_element(By.TAG_NAME, "body").text
self.assertNotIn("Buy peacock feathers", page_text)
self.assertNotIn("make a fly", page_text)
# Francis starts a new list by entering a new item. He
# is less interesting than Edith...
inputbox = self.browser.find_element(By.ID, "id_new_item")
inputbox.send_keys("Buy milk")
inputbox.send_keys(Keys.ENTER)
self.wait_for_row_in_list_table("1: Buy milk")
# Francis gets his own unique URL
francis_list_url = self.browser.current_url
self.assertRegex(francis_list_url, "/lists/.+")
self.assertNotEqual(francis_list_url, edith_list_url)
# Again, there is no trace of Edith's list
page_text = self.browser.find_element(By.TAG_NAME, "body").text
self.assertNotIn("Buy peacock feathers", page_text)
self.assertIn("Buy milk", page_text)
# Satisfied, they both go back to sleep| 1 | I’m using the convention of double-hashes (##)
to indicate "meta-comments"—comments
about how the test is working and why—so that
we can distinguish them from regular comments in FTs
which explain the User Story.
They’re a message to our future selves,
which might otherwise be wondering why we’re
faffing about deleting cookies…
|
Other than that, the new test is fairly self-explanatory. Let’s see how we do when we run our FTs:
$ python manage.py test functional_tests
[...]
.F
======================================================================
FAIL: test_multiple_users_can_start_lists_at_different_urls (functional_tests.t
ests.NewVisitorTest.test_multiple_users_can_start_lists_at_different_urls)
---------------------------------------------------------------------
Traceback (most recent call last):
File "...goat-book/functional_tests/tests.py", line 77, in
test_multiple_users_can_start_lists_at_different_urls
self.assertRegex(edith_list_url, "/lists/.+")
AssertionError: Regex didn't match: '/lists/.+' not found in
'http://localhost:8081/'
---------------------------------------------------------------------
Ran 2 tests in 5.786s
FAILED (failures=1)
Good, our first test still passes, and the second one fails where we might expect. Let’s do a commit, and then go and build some new models and views:
$ git commit -a
Iterating Towards the New Design
Being all excited about our new design, I had an overwhelming urge to dive in at this point and start changing models.py, which would have broken half the unit tests, and then pile in and change almost every single line of code, all in one go. That’s a natural urge, and TDD, as a discipline, is a constant fight against it. Obey the Testing Goat, not Refactoring Cat! We don’t need to implement our new, shiny design in a single big bang. Let’s make small changes that take us from a working state to a working state, with our design guiding us gently at each stage.
There are four items on our to-do list.
The FT, with its Regex didn't match error,
is suggesting to us that the second item—giving lists their own URL
and identifier—is the one we should work on next.
Let’s have a go at fixing that, and only that.
The URL comes from the redirect after POST.
In lists/tests.py, let’s find test_redirects_after_POST,
and change the expected redirect location:
def test_redirects_after_POST(self):
response = self.client.post("/", data={"item_text": "A new list item"})
self.assertRedirects(response, "/lists/the-only-list-in-the-world/")Does that seem slightly strange? Clearly, /lists/the-only-list-in-the-world isn’t a URL that’s going to feature in the final design of our application. But we’re committed to changing one thing at a time. While our application only supports one list, this is the only URL that makes sense. We’re still moving forwards, in that we’ll have a different URL for our list and our home page, which is a step along the way to a more REST-ful design. Later, when we have multiple lists, it will be easy to change.
| Another way of thinking about it is as a problem-solving technique: our new URL design is currently not implemented, so it works for 0 items. Ultimately, we want to solve for n items, but solving for 1 item is a good step along the way. |
Running the unit tests gives us an expected fail:
$ python manage.py test lists [...] AssertionError: '/' != '/lists/the-only-list-in-the-world/' [...]
We can go adjust our home_page view in lists/views.py:
def home_page(request):
if request.method == "POST":
Item.objects.create(text=request.POST["item_text"])
return redirect("/lists/the-only-list-in-the-world/")
items = Item.objects.all()
return render(request, "home.html", {"items": items})Django’s unit test runner picks up on the fact that this is not a real URL yet:
$ python3 manage.py test lists [...] AssertionError: 404 != 200 : Couldn't retrieve redirection page '/lists/the-only-list-in-the-world/': response code was 404 (expected 200)
Taking a First, Self-Contained Step: One New URL
Our singleton list URL doesn’t exist yet. We fix that in superlists/urls.py.
from django.urls import path
from lists import views
urlpatterns = [
path("", views.home_page, name="home"),
path("lists/the-only-list-in-the-world/", views.home_page, name="view_list"), (1)
]| 1 | We’ll just point our new URL at the existing home page view. This is the minimal change. |
| Watch out for trailing slashes in URLs, both here in urls.py and in the tests. They’re a common source of bugs. |
That gets our unit tests passing:
$ python3 manage.py test lists [...] OK
What do the FTs think?
$ python3 manage.py test functional_tests [...] AssertionError: 'Buy peacock feathers' unexpectedly found in 'Your To-Do list\n1: Buy peacock feathers'
Good, they get a little further along, we now confirm that we have a new URL, but the actual page content is still the same, it shows the old list.
Separating out our home page and list view functionality
We now have two URLs, but they’re actually doing the exact same thing. Under the hood, they’re just pointing at the same function. Continuing to work incrementally, we can start to break apart the responsibilities for these two different URLs:
-
the home page only needs to display and react to creating a brand new list based on its first item.
-
the list view page needs to be able to display existing list items and add new items to the list
Let’s split out some tests for our new URL.
Open up lists/tests.py, and add a new test class called ListViewTest.
Then move the method called test_displays_all_list_items
across from HomePageTest into our new class,
changing just the URL that is invoked by self.client.get():
class HomePageTest(TestCase):
def test_uses_home_template(self):
[...]
def test_can_save_a_POST_request(self):
[...]
def test_redirects_after_POST(self):
[...]
class ListViewTest(TestCase):
def test_displays_all_list_items(self):
Item.objects.create(text="itemey 1")
Item.objects.create(text="itemey 2")
response = self.client.get("/lists/the-only-list-in-the-world/")
self.assertContains(response, "itemey 1")
self.assertContains(response, "itemey 2")Let’s try running this test now:
$ python3 manage.py test lists OK
It passes, because the URL is still pointing at the home_page view.
Let’s make it point at a new view:
from django.urls import path
from lists import views
urlpatterns = [
path("", views.home_page, name="home"),
path("lists/the-only-list-in-the-world/", views.view_list, name="view_list"),
]That predictably fails because there is no such view function yet:
$ python3 manage.py test lists
[...]
path("lists/the-only-list-in-the-world/", views.view_list,
name="view_list"),
^^^^^^^^^^^^^^^
AttributeError: module 'lists.views' has no attribute 'view_list'
A New View Function
Fair enough. Let’s create a dummy view function in lists/views.py:
def view_list(request):
passNot quite good enough:
ValueError: The view lists.views.view_list didn't return an HttpResponse object. It returned None instead. [...] FAILED (errors=2)
Looking for the minimal code change, let’s just make the view return our existing home.html template, but with nothing in it:
def view_list(request):
return render(request, "home.html")Now the tests guide us to making sure that our list view shows existing list items:
AssertionError: False is not true : Couldn't find 'itemey 1' in response
So let’s copy the last two lines from home_page more directly:
def view_list(request):
items = Item.objects.all()
return render(request, "home.html", {"items": items})That gets us to passing unit tests!
Ran 6 tests in 0.035s OK
The FTs detect a regression
As always when we get to passing unit tests, we run the functional tests to check how things are doing "in real life":
$ python manage.py test functional_tests
[...]
FF
======================================================================
FAIL: test_can_start_a_todo_list
(functional_tests.tests.NewVisitorTest.test_can_start_a_todo_list)
---------------------------------------------------------------------
Traceback (most recent call last):
File "...goat-book/functional_tests/tests.py", line 63, in
test_can_start_a_todo_list
[...]
AssertionError: '2: Use peacock feathers to make a fly' not found in ['1: Buy
peacock feathers']
======================================================================
FAIL: test_multiple_users_can_start_lists_at_different_urls (functional_tests.t
ests.NewVisitorTest.test_multiple_users_can_start_lists_at_different_urls)
---------------------------------------------------------------------
Traceback (most recent call last):
File "...goat-book/functional_tests/tests.py", line 89, in
test_multiple_users_can_start_lists_at_different_urls
self.assertNotIn("Buy peacock feathers", page_text)
AssertionError: 'Buy peacock feathers' unexpectedly found in 'Your To-Do
list\n1: Buy peacock feathers'
Not only is our new test failing, but the old one is too. That tells us we’ve introduced a regression.
What are they trying to tell us?
Both tests are failing when we try to add the second item. We have to put our debugging hats on here. We know the home page is working, because the test has got all the way down to line 63 in the first FT, so we’ve at least added a first item. And our unit tests are all passing, so we’re pretty sure the URLs and views that we do have are doing what they should. Let’s have a quick look at those unit tests to see what they tell us:
$ grep -E "class|def" lists/tests.py
class HomePageTest(TestCase):
def test_uses_home_template(self):
def test_can_save_a_POST_request(self):
def test_redirects_after_POST(self):
def test_only_saves_items_when_necessary(self):
class ListViewTest(TestCase):
def test_displays_all_list_items(self):
class ItemModelTest(TestCase):
def test_saving_and_retrieving_items(self):
The home page displays the right template, and can handle POST requests, and the /only-list-in-the-world/ view knows how to display all items… but it doesn’t know how to handle POST requests. Ah, that gives us a clue.
A second clue is the rule of thumb that, when all the unit tests are passing but the functional tests aren’t, it’s often pointing at a problem that’s not covered by the unit tests, and in our case, that’s often a template problem.
The answer is that our home.html input form currently doesn’t specify an explicit URL to POST to:
<form method="POST">By default the browser sends the POST data back to the same URL it’s currently on. When we’re on the home page that works fine, but when we’re on our /only-list-in-the-world/ page, it doesn’t.
Getting Back to a Working State as Quickly as Possible
Now we could dive in and add POST request handling to our new view, but that would involve writing a bunch more tests and code, and at this point we’d like to get back to a working state as quickly as possible. Actually the quickest thing we can do to get things fixed is to just use the existing home page view, which already works, for all POST requests:
<form method="POST" action="/">Try that, and we’ll see our FTs get back to a happier place:
FAIL: test_multiple_users_can_start_lists_at_different_urls (functional_tests.t ests.NewVisitorTest.test_multiple_users_can_start_lists_at_different_urls) [...] AssertionError: 'Buy peacock feathers' unexpectedly found in 'Your To-Do list\n1: Buy peacock feathers' Ran 2 tests in 8.541s FAILED (failures=1)
Our regression test passes once again, so we know we’re back to a working state. The new functionality may not be working yet, but at least the old stuff works as well as it used to.
Green? Refactor
Time for a little tidying up.
In the Red/Green/Refactor dance, we’ve arrived at green, so we should see what needs a refactor. We now have two views, one for the home page, and one for an individual list. Both are currently using the same template, and passing it all the list items currently in the database. Post requests are only handled by the home page though.
It feels like the responsibilities of our two views are a little tangled up. Let’s try and disentangle them a little.
Another Small Step: A Separate Template for Viewing Lists
Since the home page and the list view are now quite distinct pages, they should be using different HTML templates; home.html can have the single input box, whereas a new template, list.html, can take care of showing the table of existing items.
Let’s add a new test to check that it’s using a different template:
class ListViewTest(TestCase):
def test_uses_list_template(self):
response = self.client.get("/lists/the-only-list-in-the-world/")
self.assertTemplateUsed(response, "list.html")
def test_displays_all_list_items(self):
[...]Let’s see what it says:
AssertionError: False is not true : Template 'list.html' was not a template used to render the response. Actual template(s) used: home.html
Looks about right, let’s change the view:
def view_list(request):
items = Item.objects.all()
return render(request, "list.html", {"items": items})But, obviously, that template doesn’t exist yet. If we run the unit tests, we get:
django.template.exceptions.TemplateDoesNotExist: list.html
Let’s create a new file at lists/templates/list.html:
$ touch lists/templates/list.html
A blank template, which gives us this error—good to know the tests are there to make sure we fill it in:
AssertionError: False is not true : Couldn't find 'itemey 1' in response
The template for an individual list will reuse quite a lot of the stuff we currently have in home.html, so we can start by just copying that:
$ cp lists/templates/home.html lists/templates/list.html
That gets the tests back to passing (green).
$ python manage.py test lists [...] OK
Now let’s do a little more tidying up (refactoring).
We said the home page doesn’t need to list items,
it only needs the new list input field,
so we can remove some lines from lists/templates/home.html,
and maybe slightly tweak the h1 to say "Start a new To-Do list":
I’ll present the code change as a diff, which hopefully shows a bit more clearly what we need to modify:
<body>
- <h1>Your To-Do list</h1>
+ <h1>Start a new To-Do list</h1>
<form method="POST" action="/">
<input name="item_text" id="id_new_item" placeholder="Enter a to-do item" />
{% csrf_token %}
</form>
- <table id="id_list_table">
- {% for item in items %}
- <tr><td>{{ forloop.counter }}: {{ item.text }}</td></tr>
- {% endfor %}
- </table>
</body>We rerun the unit tests to check that hasn’t broken anything…
OK
Good.
Now there’s actually no need to pass all the items to the home.html template
in our home_page view, so we can simplify that and delete a couple of lines:
if request.method == "POST":
Item.objects.create(text=request.POST["item_text"])
return redirect("/lists/the-only-list-in-the-world/")
-
- items = Item.objects.all()
- return render(request, "home.html", {"items": items})
+ return render(request, "home.html")Rerun the unit tests once more; they still pass:
OK
Time to run the functional tests:
AssertionError: '1: Buy milk' not found in ['1: Buy peacock feathers', '2: Buy milk']
Not bad! Our regression test (the first FT) is passing, and our new test is now getting slightly further forwards—it’s telling us that Francis isn’t getting his own list page (because he still sees some of Edith’s list items).
It may feel like we haven’t made much headway since, functionally, the site still behaves almost exactly like it did when we started the chapter, but this really is progress. We’ve started on the road to our new design, and we’ve implemented a number of stepping stones without making anything worse than it was before. Let’s commit our progress so far:
$ git status # should show 5 changed files and 1 new file, list.html
$ git add lists/templates/list.html
$ git diff # should show we've simplified home.html,
# moved one test to a new class in lists/tests.py added a new view
# in views.py, and simplified home_page and added a line to urls.py
# moved one test to a new class in lists/tests.py,
# added a new view and simplified home_page in views.py,
# and added a line to urls.py.
$ git commit -a # add a message summarising the above, maybe something like
# "new URL, view and template to display lists"
A Third Small Step: A New URL for Adding List Items
Where are we with our own to-do list?
We’ve sort of made progress on the second item, even if there’s still only one list in the world. The first item is a bit scary. Can we do something about items 3 or 4?
Let’s have a new URL for adding new list items at /lists/new: If nothing else, it’ll simplify the home page view.
A Test Class for New List Creation
Open up lists/tests.py,
and move the test_can_save_a_POST_request and test_redirects_after_POST methods
into a new class, then change the URL they POST to:
class NewListTest(TestCase):
def test_can_save_a_POST_request(self):
self.client.post("/lists/new", data={"item_text": "A new list item"})
self.assertEqual(Item.objects.count(), 1)
new_item = Item.objects.get()
self.assertEqual(new_item.text, "A new list item")
def test_redirects_after_POST(self):
response = self.client.post("/lists/new", data={"item_text": "A new list item"})
self.assertRedirects(response, "/lists/the-only-list-in-the-world/")
This is another place to pay attention to trailing slashes, incidentally.
It’s /lists/new, with no trailing slash.
The convention I’m using is that
URLs without a trailing slash are "action" URLs which modify the database.
|
Try running that:
self.assertEqual(Item.objects.count(), 1)
AssertionError: 0 != 1
[...]
self.assertRedirects(response, "/lists/the-only-list-in-the-world/")
[...]
AssertionError: 404 != 302 : Response didn't redirect as expected: Response
code was 404 (expected 302)
The first failure tells us we’re not saving a new item to the database,
and the second says that, instead of returning a 302 redirect,
our view is returning a 404.
That’s because we haven’t built a URL for /lists/new,
so the client.post is just getting a "not found" response.
Do you remember how we split this out into two tests earlier?
If we only had one test that checked both the saving and the redirect,
it would have failed on the 0 != 1 failure,
which would have been much harder to debug.
Ask me how I know this.
|
A URL and View for New List Creation
Let’s build our new URL now:
urlpatterns = [
path("", views.home_page, name="home"),
path("lists/new", views.new_list, name="new_list"),
path("lists/the-only-list-in-the-world/", views.view_list, name="view_list"),
]Next we get a no attribute 'new_list', so let’s fix that, in
lists/views.py:
def new_list(request):
passThen we get "The view lists.views.new_list didn’t return an HttpResponse
object". (This is getting rather familiar!) We could return a raw
HttpResponse, but since we know we’ll need a redirect, let’s borrow a line
from home_page:
def new_list(request):
return redirect("/lists/the-only-list-in-the-world/")That gives:
self.assertEqual(Item.objects.count(), 1) AssertionError: 0 != 1
Seems reasonably straightforward.
We borrow another line from home_page:
def new_list(request):
Item.objects.create(text=request.POST["item_text"])
return redirect("/lists/the-only-list-in-the-world/")And everything now passes:
Ran 7 tests in 0.030s OK
And we can run the FTs to check that we’re still in the same place: our regression test passes, and the new FT gets to the same point.
[...] AssertionError: '1: Buy milk' not found in ['1: Buy peacock feathers', '2: Buy milk'] Ran 2 tests in 8.972s FAILED (failures=1)
Removing Now-Redundant Code and Tests
We’re looking good.
Since our new views are now doing most of the work that home_page used to do,
we should be able to massively simplify it.
Can we remove the whole if request.method == 'POST' section,
for example?
def home_page(request):
return render(request, "home.html")Yep! The unit tests pass:
OK
And while we’re at it, we can remove the now-redundant
test_only_saves_items_when_necessary test too!
Doesn’t that feel good? The view functions are looking much simpler. We rerun the tests to make sure…
Ran 6 tests in 0.016s OK
and the FTs?
A Regression! Pointing Our Forms at the New URL
Oops. When we run the FTs:
ERROR: test_can_start_a_todo_list
[...]
File "...goat-book/functional_tests/tests.py", line 52, in
test_can_start_a_todo_list
[...]
self.wait_for_row_in_list_table("1: Buy peacock feathers")
[...]
table = self.browser.find_element(By.ID, "id_list_table")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[...]
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: [id="id_list_table"]; For documentation [...]
ERROR: test_multiple_users_can_start_lists_at_different_urls (functional_tests.
tests.NewVisitorTest.test_multiple_users_can_start_lists_at_different_urls)
[...]
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate
element: [id="id_list_table"]; For documentation [...]
[...]
Ran 2 tests in 11.592s
FAILED (errors=2)
Once again, the FTs pick up a tricky little bug, something that our unit tests alone would find it hard to catch.
It’s because our forms are still pointing to the old URL. In both home.html and lists.html, let’s change them to:
<form method="POST" action="/lists/new">And that should get us back to working again:
AssertionError: '1: Buy milk' not found in ['1: Buy peacock feathers', '2: Buy milk'] [...] FAILED (failures=1)
That’s another nicely self-contained commit, in that we’ve made a bunch of changes to our URLs, our views.py is looking much neater and tidier, and we’re sure the application is still working as well as it was before. We’re getting good at this working-state-to-working-state malarkey!
$ git status # 5 changed files $ git diff # URLs for forms x2, moved code in views + tests, new URL $ git commit -a
And we can cross out an item on the to-do list:
Biting the Bullet: Adjusting Our Models
Enough housekeeping with our URLs. It’s time to bite the bullet and change our models. Let’s adjust the model unit test. Again, I’ll use a diff to show you the changes:
@@ -1,5 +1,5 @@
from django.test import TestCase
-from lists.models import Item
+from lists.models import Item, List
class HomePageTest(TestCase):
@@ -35,20 +35,30 @@ class ListViewTest(TestCase):
self.assertContains(response, "itemey 2")
-class ItemModelTest(TestCase):
+class ListAndItemModelsTest(TestCase):
def test_saving_and_retrieving_items(self):
+ mylist = List()
+ mylist.save()
+
first_item = Item()
first_item.text = "The first (ever) list item"
+ first_item.list = mylist
first_item.save()
second_item = Item()
second_item.text = "Item the second"
+ second_item.list = mylist
second_item.save()
+ saved_list = List.objects.get()
+ self.assertEqual(saved_list, mylist)
+
saved_items = Item.objects.all()
self.assertEqual(saved_items.count(), 2)
first_saved_item = saved_items[0]
second_saved_item = saved_items[1]
self.assertEqual(first_saved_item.text, "The first (ever) list item")
+ self.assertEqual(first_saved_item.list, mylist)
self.assertEqual(second_saved_item.text, "Item the second")
+ self.assertEqual(second_saved_item.list, mylist)We create a new List object
and then we assign each item to it by setting it as its .list property.
We check that the list is properly saved,
and we check that the two items have also saved their relationship to the list.
You’ll also notice that we can compare list objects with each other directly
(saved_list and mylist)—behind the scenes,
these will compare themselves by checking
that their primary key (the .id attribute) is the same.
Time for another unit-test/code cycle.
For the first couple of iterations, rather than explicitly showing you what code to enter in between every test run, I’m only going to show you the expected error messages from running the tests. I’ll let you figure out what each minimal code change should be, on your own.
Need a hint?
Go back and take a look at the steps we took
to introduce the Item model in the chapter before last.
|
Your first error should be:
ImportError: cannot import name 'List' from 'lists.models'
Fix that, and then you should see:
AttributeError: 'List' object has no attribute 'save'
Next you should see:
django.db.utils.OperationalError: no such table: lists_list
So we run a makemigrations:
$ python manage.py makemigrations
Migrations for 'lists':
lists/migrations/0003_list.py
- Create model List
And then you should see:
self.assertEqual(first_saved_item.list, mylist) AttributeError: 'Item' object has no attribute 'list'
A Foreign Key Relationship
How do we give our Item a list attribute?
Let’s just try naively making it like the text attribute
(and here’s your chance
to see whether your solution so far looks like mine, by the way):
from django.db import models
class List(models.Model):
pass
class Item(models.Model):
text = models.TextField(default="")
list = models.TextField(default="")As usual, the tests tell us we need a migration:
$ python manage.py test lists
[...]
django.db.utils.OperationalError: no such column: lists_item.list
$ python manage.py makemigrations
Migrations for 'lists':
lists/migrations/0004_item_list.py
- Add field list to item
Let’s see what that gives us:
AssertionError: 'List object (1)' != <List: List object (1)>
We’re not quite there. Look closely at each side of the !=.
Do you see the quotes (')?
Django has only saved the string representation of the List object.
To save the relationship to the object itself,
we tell Django about the relationship between the two classes using a ForeignKey:
class Item(models.Model):
text = models.TextField(default="")
list = models.ForeignKey(List, default=None, on_delete=models.CASCADE)That’ll need a migration too. Since the last one was a red herring, let’s delete it and replace it with a new one:
$ rm lists/migrations/0004_item_list.py
$ python manage.py makemigrations
Migrations for 'lists':
lists/migrations/0004_item_list.py
- Add field list to item
| Deleting migrations is dangerous. Now and again it’s nice to do it to keep things tidy, because we don’t always get our models code right on the first go! But if you delete a migration that’s already been applied to a database somewhere, Django will be confused about what state it’s in, and won’t be able to apply future migrations. You should only do it when you’re sure the migration hasn’t been used. A good rule of thumb is that you should never delete or modify a migration that’s already been committed to your VCS. |
Adjusting the Rest of the World to Our New Models
Back in our tests, now what happens?
$ python manage.py test lists [...] ERROR: test_displays_all_list_items django.db.utils.IntegrityError: NOT NULL constraint failed: lists_item.list_id [...] ERROR: test_redirects_after_POST django.db.utils.IntegrityError: NOT NULL constraint failed: lists_item.list_id [...] ERROR: test_can_save_a_POST_request django.db.utils.IntegrityError: NOT NULL constraint failed: lists_item.list_id Ran 6 tests in 0.021s FAILED (errors=3)
Oh dear!
There is some good news. Although it’s hard to see, our model tests are passing. But three of our view tests are failing nastily.
The cause is the new relationship we’ve introduced between Items and Lists,
which requires each item to have a parent list,
and which our old tests and code aren’t prepared for.
Still, this is exactly why we have tests!
Let’s get them working again.
The easiest is the ListViewTest;
we just create a parent list for our two test items:
class ListViewTest(TestCase):
[...]
def test_displays_all_list_items(self):
mylist = List.objects.create()
Item.objects.create(text="itemey 1", list=mylist)
Item.objects.create(text="itemey 2", list=mylist)That gets us down to two failing tests,
both on tests that try to POST to our new_list view.
Decoding the tracebacks using our usual technique,
working back from error to line of test code to,
buried in there somewhere,
the line of our own code that caused the failure:
File "...goat-book/lists/tests.py", line 19, in test_redirects_after_POST
response = self.client.post("/lists/new", data={"item_text": "A new list
item"})
[...]
File "...goat-book/lists/views.py", line 10, in new_list
Item.objects.create(text=request.POST["item_text"])
It’s when we try to create an item without a parent list. So we make a similar change in the view:
from lists.models import Item, List
[...]
def new_list(request):
nulist = List.objects.create()
Item.objects.create(text=request.POST["item_text"], list=nulist)
return redirect("/lists/the-only-list-in-the-world/")And that[1]
gets our tests passing again:
Ran 6 tests in 0.030s OK
Are you cringing internally at this point? Arg! This feels so wrong; we create a new list for every single new item submission, and we’re still just displaying all items as if they belong to the same list! I know, I feel the same. The step-by-step approach, in which you go from working code to working code, is counterintuitive. I always feel like just diving in and trying to fix everything all in one go, instead of going from one weird half-finished state to another. But remember the Testing Goat! When you’re up a mountain, you want to think very carefully about where you put each foot, and take one step at a time, checking at each stage that the place you’ve put it hasn’t caused you to fall off a cliff.
So just to reassure ourselves that things have worked, we rerun the FT:
AssertionError: '1: Buy milk' not found in ['1: Buy peacock feathers', '2: Buy milk'] [...]
Sure enough, it gets all the way through to where we were before. We haven’t broken anything, and we’ve made a big change to the database. That’s something to be pleased with! Let’s commit:
$ git status # 3 changed files, plus 2 migrations $ git add lists $ git diff --staged $ git commit
And we can cross out another item on the to-do list:
Each List Should Have Its Own URL
We can get rid of the silly the-only-list-in-the-world URL,
but what shall we use as the unique identifier for our lists?
Probably the simplest thing, for now,
is just to use the auto-generated id field from the database.
Let’s change ListViewTest so that the two tests point at new URLs.
We’ll also change the old test_displays_all_list_items test
and call it test_displays_only_items_for_that_list instead,
making it check that only the items for a specific list are displayed:
class ListViewTest(TestCase):
def test_uses_list_template(self):
mylist = List.objects.create()
response = self.client.get(f"/lists/{mylist.id}/")
self.assertTemplateUsed(response, "list.html")
def test_displays_only_items_for_that_list(self):
correct_list = List.objects.create()
Item.objects.create(text="itemey 1", list=correct_list)
Item.objects.create(text="itemey 2", list=correct_list)
other_list = List.objects.create()
Item.objects.create(text="other list item", list=other_list)
response = self.client.get(f"/lists/{correct_list.id}/")
self.assertContains(response, "itemey 1")
self.assertContains(response, "itemey 2")
self.assertNotContains(response, "other list item")| Are you wondering about the line spacing in the test? I’m grouping together five lines at the beginning which set up the test, one line in the middle which actually calls the code under test, and the assertions at the end. This isn’t obligatory, but it does help see the structure of the test. Some people refer to this structure as Arrange-Act-Assert, or Given-When-Then: Given the database contains our list with two items, and another list, When I make a GET request for our list, Then I see the items in our list, but not the items in the other list. |
Running the unit tests gives an expected 404, and another related error:
FAIL: test_displays_only_items_for_that_list AssertionError: 404 != 200 : Couldn't retrieve content: Response code was 404 (expected 200) [...] FAIL: test_uses_list_template AssertionError: No templates used to render the response
Capturing Parameters from URLs
It’s time to learn how we can pass parameters from URLs to views:
urlpatterns = [
path("", views.home_page, name="home"),
path("lists/new", views.new_list, name="new_list"),
path("lists/<int:list_id>/", views.view_list, name="view_list"),
]We adjust the regular expression for our URL to include a capture group,
<int:list_id>, which will match any numerical characters, up to the following /,
The captured id will get passed to the view as an argument.
In other words, if we go to the URL /lists/1/, view_list will get a second
argument after the normal request argument, namely the integer 1.
But our view doesn’t expect an argument yet! Sure enough, this causes problems:
ERROR: test_displays_only_items_for_that_list [...] TypeError: view_list() got an unexpected keyword argument 'list_id' [...] ERROR: test_uses_list_template [...] TypeError: view_list() got an unexpected keyword argument 'list_id' [...] FAIL: test_redirects_after_POST [...] AssertionError: 404 != 200 : Couldn't retrieve redirection page '/lists/the-only-list-in-the-world/': response code was 404 (expected 200) [...] FAILED (failures=1, errors=2)
We can fix that easily with a dummy parameter in views.py:
def view_list(request, list_id):
[...]That takes us down to our expected failure, plus an only-list-in-the-world that’s still hanging around somewhere, which I’m sure we can fix later.
FAIL: test_displays_only_items_for_that_list [...] AssertionError: 1 != 0 : Response should not contain 'other list item' [...] FAIL: test_redirects_after_POST AssertionError: 404 != 200 : Couldn't retrieve redirection page '/lists/the-only-list-in-the-world/': response code was 404 (expected 200)
Let’s make our list view discriminate over which items it sends to the template:
def view_list(request, list_id):
our_list = List.objects.get(id=list_id)
items = Item.objects.filter(list=our_list)
return render(request, "list.html", {"items": items})Adjusting new_list to the New World
It’s time to address the only-list-in-the-world failure:
FAIL: test_redirects_after_POST [...] AssertionError: 404 != 200 : Couldn't retrieve redirection page '/lists/the-only-list-in-the-world/': response code was 404 (expected 200)
Let’s have a little look and find the test that’s moaning:
class NewListTest(TestCase):
[...]
def test_redirects_after_POST(self):
response = self.client.post("/lists/new", data={"item_text": "A new list item"})
self.assertRedirects(response, "/lists/the-only-list-in-the-world/")It looks like it hasn’t been adjusted to the new world of Lists and Items.
The test should be saying that this view redirects
to the URL of the specific new list it just created.
def test_redirects_after_POST(self):
response = self.client.post("/lists/new", data={"item_text": "A new list item"})
new_list = List.objects.get()
self.assertRedirects(response, f"/lists/{new_list.id}/")The test still fails, but we can now take a look at the view itself, and change it so it redirects to the right place:
def new_list(request):
nulist = List.objects.create()
Item.objects.create(text=request.POST["item_text"], list=nulist)
return redirect(f"/lists/{nulist.id}/")That gets us back to passing unit tests:
$ python3 manage.py test lists [...] ...... --------------------------------------------------------------------- Ran 6 tests in 0.033s OK
What about the functional tests? We must be almost there?
The Functional Tests Detect Another Regression
Well, almost. When we run the FTs, we get:
F.
======================================================================
FAIL: test_can_start_a_todo_list
(functional_tests.tests.NewVisitorTest.test_can_start_a_todo_list)
---------------------------------------------------------------------
Traceback (most recent call last):
File "...goat-book/functional_tests/tests.py", line 63, in
test_can_start_a_todo_list
self.wait_for_row_in_list_table("2: Use peacock feathers to make a fly")
[...]
AssertionError: '2: Use peacock feathers to make a fly' not found in ['1: Use
peacock feathers to make a fly']
---------------------------------------------------------------------
Ran 2 tests in 8.617s
FAILED (failures=1)
Our new FT is actually passing: different users can get different lists. But the old test is warning us of a regression. It looks like you can’t add a second item to a list any more. It’s because of our quick-and-dirty hack where we create a new list for every single POST submission. This is exactly what we have functional tests for!
And it correlates nicely with the last item on our to-do list:
One More View to Handle Adding Items to an Existing List
We need a URL and view to handle adding a new item to an existing list (/lists/<list_id>/add_item). We’re getting pretty good at these now, so let’s knock one together quickly:
class NewItemTest(TestCase):
def test_can_save_a_POST_request_to_an_existing_list(self):
other_list = List.objects.create()
correct_list = List.objects.create()
self.client.post(
f"/lists/{correct_list.id}/add_item",
data={"item_text": "A new item for an existing list"},
)
self.assertEqual(Item.objects.count(), 1)
new_item = Item.objects.get()
self.assertEqual(new_item.text, "A new item for an existing list")
self.assertEqual(new_item.list, correct_list)
def test_redirects_to_list_view(self):
other_list = List.objects.create()
correct_list = List.objects.create()
response = self.client.post(
f"/lists/{correct_list.id}/add_item",
data={"item_text": "A new item for an existing list"},
)
self.assertRedirects(response, f"/lists/{correct_list.id}/")
Are you wondering about other_list?
A bit like in the tests for viewing a specific list,
it’s important that we add items to a specific list.
Adding this second object to the database prevents me from using a hack
like List.objects.first() in the implementation.
Yes, that would be a stupid thing to do,
and you can go too far down the road of testing
for all the stupid things you must not do
(there are an infinite number of those, after all).
It’s a judgement call, but this one feels worth it.
There’s some more discussion of this in [testing-for-stupidity].
|
So that fails as expected, the list item is not saved, and the new URL currently returns a 404:
AssertionError: 0 != 1 [...] AssertionError: 404 != 302 : Response didn't redirect as expected: Response code was 404 (expected 302)
The Last New URL
Now we’ve got our expected 404, let’s add a new URL for adding new items to existing lists:
urlpatterns = [
path("", views.home_page, name="home"),
path("lists/new", views.new_list, name="new_list"),
path("lists/<int:list_id>/", views.view_list, name="view_list"),
path("lists/<int:list_id>/add_item", views.add_item, name="add_item"),
]Three very similar-looking URLs there. Let’s make a note on our to-do list; they look like good candidates for a refactoring:
Back to the tests, we get the usual missing module view objects:
AttributeError: module 'lists.views' has no attribute 'add_item'
The Last New View
Let’s try:
def add_item(request):
passAha:
TypeError: add_item() got an unexpected keyword argument 'list_id'
def add_item(request, list_id):
passAnd then:
ValueError: The view lists.views.add_item didn't return an HttpResponse object. It returned None instead.
We can copy the redirect() from new_list
and the List.objects.get() from view_list:
def add_item(request, list_id):
our_list = List.objects.get(id=list_id)
return redirect(f"/lists/{our_list.id}/")That takes us to:
self.assertEqual(Item.objects.count(), 1) AssertionError: 0 != 1
Finally we make it save our new list item:
def add_item(request, list_id):
our_list = List.objects.get(id=list_id)
Item.objects.create(text=request.POST["item_text"], list=our_list)
return redirect(f"/lists/{our_list.id}/")And we’re back to passing tests.
Ran 8 tests in 0.050s OK
Testing Template Context Directly
We’ve got our new view and URL for adding items to existing lists; now we just need to actually use it in our list.html template. So we open it up to adjust the form tag…
<form method="POST" action="but what should we put here?">…oh. To get the URL for adding to the current list, the template needs to know what list it’s rendering, as well as what the items are.
We want to be able to do something like this:
<form method="POST" action="/lists/{{ list.id }}/add_item">For that to work, the view will have to pass the list to the template.
Let’s create a new unit test in ListViewTest:
def test_passes_correct_list_to_template(self):
other_list = List.objects.create()
correct_list = List.objects.create()
response = self.client.get(f"/lists/{correct_list.id}/")
self.assertEqual(response.context["list"], correct_list) (1)| 1 | response.context represents the context we’re going to pass into
the render function—the Django Test Client puts it on the response
object for us, to help with testing. |
That gives us:
self.assertEqual(response.context["list"], correct_list)
~~~~~~~~~~~~~~~~^^^^^^^^
[...]
KeyError: 'list'
because we’re not passing list into the template.
It actually gives us an opportunity to simplify a little:
def view_list(request, list_id):
our_list = List.objects.get(id=list_id)
return render(request, "list.html", {"list": our_list})That, of course, introduces a bug,
because the template needed items:
FAIL: test_displays_only_items_for_that_list [...] AssertionError: False is not true : Couldn't find 'itemey 1' in response
But we can fix it in list.html, as well as adjusting the form’s POST action, which is what we were trying to do anyway:
<form method="POST" action="/lists/{{ list.id }}/add_item"> (1)
[...]
{% for item in list.item_set.all %} (2)
<tr><td>{{ forloop.counter }}: {{ item.text }}</td></tr>
{% endfor %}| 1 | There’s our new form action. |
| 2 | .item_set is called a
reverse lookup.
It’s one of Django’s incredibly useful bits of ORM that lets you look up an
object’s related items from a different table.
|
So that gets the unit tests to pass:
Ran 9 tests in 0.040s OK
How about the FTs?
$ python manage.py test functional_tests [...] .. --------------------------------------------------------------------- Ran 2 tests in 9.771s OK
HOORAY! Oh, and a quick check on our to-do list:
Irritatingly, the Testing Goat is a stickler for tying up loose ends too, so we’ve got to do one final thing.
Before we start, we’ll do a commit—always make sure you’ve got a commit of a working state before embarking on a refactor:
$ git diff $ git commit -am "new URL + view for adding to existing lists. FT passes :-)"
A Final Refactor Using URL includes
superlists/urls.py is really meant for URLs that apply to your entire site.
For URLs that only apply to the lists app,
Django encourages us to use a separate lists/urls.py,
to make the app more self-contained.
The simplest way to make one is to use a copy of the existing urls.py:
$ cp superlists/urls.py lists/
Then we replace the three list-specific lines in superlists/urls.py with an include():
from django.urls import include, path
from lists import views as list_views (1)
urlpatterns = [
path("", list_views.home_page, name="home"),
path("lists/", include("lists.urls")), (2)
]| 1 | While we’re at it, we use the import x as y syntax to alias views
This is good practice in your top-level urls.py,
because it will let us import views from multiple apps if we want—and
indeed we will need to later on in the book. |
| 2 | Here’s the include.
Notice that it can take a part of a URL as a prefix,
which will be applied to all the included URLs
(this is the bit where we reduce duplication,
as well as giving our code a better structure). |
Back in lists/urls.py we can trim down to only include the latter part of our three URLs, and none of the other stuff from the parent urls.py:
from django.urls import path
from lists import views
urlpatterns = [
path("new", views.new_list, name="new_list"),
path("<int:list_id>/", views.view_list, name="view_list"),
path("<int:list_id>/add_item", views.add_item, name="add_item"),
]Rerun the unit tests to check that everything worked.
Ran 9 tests in 0.040s OK
When I saw that it passed, I couldn’t quite believe I did it correctly on the first go. It always pays to be skeptical of your own abilities, so I deliberately changed one of the URLs slightly, just to check if it broke a test. It did. We’re covered.
Feel free to try it yourself! Remember to change it back, check that the tests all pass again, and then do a final commit:
$ git status $ git add lists/urls.py $ git add superlists/urls.py $ git diff --staged $ git commit
Phew. This was a marathon chapter. But we covered a number of important topics, starting with some thinking about design. We covered rules of thumb like "YAGNI" and "three strikes then refactor". But, most importantly, we saw how to adapt an existing codebase step by step, going from working state to working state, in order to iterate towards a new design.
I’d say we’re pretty close to being able to ship this site, as the very first beta of the superlists website that’s going to take over the world. Maybe it needs a little prettification first…let’s look at what we need to do to deploy it in the next couple of chapters.
Comments