The Subtleties of Functionally Testing External Dependencies

Appendix D: The Subtleties of Functionally Testing External Dependencies

You might remember from [options-for-testing-real-email] a point at which we wanted to test sending email from the server.

Here were the options we considered:

We could build a "real" end-to-end test, and have our tests log in to an email server, and retrieve the email from there. That’s what I did in the first and second edition.
You can use a service like Mailinator or Mailsac, which give you an email account to send to, and some APIs for checking what mail has been delivered.
We can use an alternative, fake email backend, whereby Django will save the emails to a file on disk for example, and we can inspect them there.
Or we could give up on testing email on the server. If we have a minimal smoke test that the server can send emails, then we don’t need to test that they are actually delivered.

In the end we decided not to bother, but let’s spend a bit of time in this appendix trying out options 1 and 3, just to see some of the fiddliness and trade-offs involved.

How to Test Email End-To-End with POP3

Here’s an example helper function that can retrieve a real email from a real POP3 email server, using the horrifically tortuous Python standard library POP3 client.

To make it work, we’ll need an email address to receive the email. I signed up for a Yahoo account for testing, but you can use any email service you like, as long as it offers POP3 access.

You will need to set the RECEIVER_EMAIL_PASSWORD environment variable in the console that’s running the FT.

$ export RECEIVER_EMAIL_PASSWORD=otheremailpasswordhere

src/functional_tests/test_login.py (ch23l001)

import os
import poplib
import re
impot time
[...]

def retrieve_pop3_email(receiver_email, subject, pop3_server, pop3_password):
    email_id = None
    start = time.time()
    inbox = poplib.POP3_SSL(pop3_server)
    try:
        inbox.user(receiver_email)
        inbox.pass_(pop3_password)
        while time.time() - start < POP3_TIMEOUT:
            # get 10 newest messages
            count, _ = inbox.stat()
            for i in reversed(range(max(1, count - 10), count + 1)):
                print("getting msg", i)
                _, lines, __ = inbox.retr(i)
                lines = [l.decode("utf8") for l in lines]
                print(lines)
                if f"Subject: {subject}" in lines:
                    email_id = i
                    body = "\n".join(lines)
                    return body
            time.sleep(5)
    finally:
        if email_id:
            inbox.dele(email_id)
        inbox.quit()

If you’re curious, I’d encourage you to try this out in your FTs. It definitely can work. But, having tried it in the first couple of editions of the book. I have to say it’s fiddly to get right, and often flaky, which is a highly undesirable property for a testing tool. So let’s leave that there for now.

Using a Fake Email Backend For Django

Next let’s investigate using a filesystem-based email backend. As we’ll see, although it definitely has the advantage that everything stays local on our own machine (there are no calls over the internet), there are quite a few things to watch out for.

Let’s say that, if we detect an environment variable EMAIL_FILE_PATH, we switch to Django’s file-based backend:

src/superlists/settings.py (ch23l002)

EMAIL_HOST = "smtp.gmail.com"
EMAIL_HOST_USER = "[email protected]"
EMAIL_HOST_PASSWORD = os.environ.get("EMAIL_PASSWORD")
EMAIL_PORT = 587
EMAIL_USE_TLS = True
# Use fake file-based backend if EMAIL_FILE_PATH is set
if "EMAIL_FILE_PATH" in os.environ:
    EMAIL_BACKEND = "django.core.mail.backends.filebased.EmailBackend"
    EMAIL_FILE_PATH = os.environ["EMAIL_FILE_PATH"]

Here’s how we can adapt our tests to conditionally use the email file, instead of Django’s mail.outbox, if the env var is set when running our tests:

src/functional_tests/test_login.py (ch23l003)

class LoginTest(FunctionalTest):
    def retrieve_email_from_file(self, sent_to, subject, emails_dir):  (1)
        latest_emails_file = sorted(Path(emails_dir).iterdir())[-1]  (2)
        latest_email = latest_emails_file.read_text().split("-" * 80)[-1]  (3)
        self.assertIn(subject, latest_email)
        self.assertIn(sent_to, latest_email)
        return latest_email

    def retrieve_email_from_django_outbox(self, sent_to, subject):  (4)
        email = mail.outbox.pop()
        self.assertIn(sent_to, email.to)
        self.assertEqual(email.subject, subject)
        return email.body

    def wait_for_email(self, sent_to, subject):  (5)
        """
        Retrieve email body,
        from a file if the right env var is set,
        or get it from django.mail.outbox by default
        """
        if email_file_path := os.environ.get("EMAIL_FILE_PATH"):  (6)
            return self.wait_for(  (7)
                lambda: self.retrieve_email_from_file(sent_to, subject, email_file_path)
            )
        else:
            return self.retrieve_email_from_django_outbox(sent_to, subject)

    def test_login_using_magic_link(self):
        [...]

1	Here’s our helper method for getting email contents from a file. It takes the configured email directory as an argument, as well as the sent-to address and expected subject.
2	Django saves a new file with emails every time you restart the server. The filename has a timestamp in it, so we can get the latest one by sorting the files in our test directory. Check out the Pathlib docs if you haven’t used it before, it’s a nice, relatively new way of working with files in Python.
3	The emails in the file are separated by a line of 80 hyphens.
4	This is the matching helper for getting the email from `mail.outbox`.
5	Here’s where we dispatch to the right helper based on whether the env var is set.
6	Checking whether an environment variable is set, and using its value if so, is one of the (relatively few) places where it’s nice to use the walrus operator.
7	I’m using a `wait_for()` here because anything involving reading and writing from files, especially across the filesystem mounts inside and outside of Docker, has a potential race condition.

We’ll need a couple more minor changes to the FT, to use the helper:

src/functional_tests/test_login.py (ch23l004)

@@ -59,15 +59,12 @@ class LoginTest(FunctionalTest):
         )

         # She checks her email and finds a message
-        email = mail.outbox.pop()
-        self.assertIn(TEST_EMAIL, email.to)
-        self.assertEqual(email.subject, SUBJECT)
+        email_body = self.wait_for_email(TEST_EMAIL, SUBJECT)

         # It has a URL link in it
-        self.assertIn("Use this link to log in", email.body)
-        url_search = re.search(r"http://.+/.+$", email.body)
-        if not url_search:
-            self.fail(f"Could not find url in email body:\n{email.body}")
+        self.assertIn("Use this link to log in", email_body)
+        if not (url_search := re.search(r"http://.+/.+$", email_body, re.MULTILINE)):
+            self.fail(f"Could not find url in email body:\n{email_body}")
         url = url_search.group(0)
         self.assertIn(self.live_server_url, url)

Now let’s set that file path, and mount it inside our docker container, so that it’s available both inside and outside the container:

# set a local env var for our path to the emails file
$ export EMAIL_FILE_PATH=/tmp/superlists-emails
# make sure the file exists
$ mkdir -p $EMAIL_FILE_PATH
# re-run our container, with the EMAIL_FILE_PATH as an env var, and mounted.
$ docker build -t superlists . && docker run \
    -p 8888:8888 \
    --mount type=bind,source=./src/db.sqlite3,target=/src/db.sqlite3 \
    --mount type=bind,source=$EMAIL_FILE_PATH,target=$EMAIL_FILE_PATH \  (1)
    -e DJANGO_SECRET_KEY=sekrit \
    -e DJANGO_ALLOWED_HOST=localhost \
    -e EMAIL_PASSWORD \
    -e EMAIL_FILE_PATH \  (2)
    -it superlists

1	Here’s where we mount the emails file so we can see it both inside and outside the container
2	And here’s where we pass the path as an env var, once again re-exporting the variable from the current shell.

And we can re-run our FT, first without using Docker or the EMAIL_FILE_PATH, just to check we didn’t break anything:

$ ./src/manage.py test functional_tests.test_login
[...]
OK

And now with Docker and the EMAIL_FILE_PATH:

$ TEST_SERVER=localhost:8888 EMAIL_FILE_PATH=/tmp/superlists-emails \
  python src/manage.py test functional_tests
[...]
OK

It works! Hooray.

Double-Checking our Test and Our Fix

As always, we should be suspicious of any test that we’ve only ever seen pass! Let’s see if we can make this test fail.

Before we do—we’ve been in the detail for a bit, it’s worth reminding ourselves of what the actual bug was, and how we’re fixing it! The bug was, the server was crashing when it tried to send an email. The reason was, we hadn’t set the EMAIL_PASSWORD environment variable. We managed to repro the bug in Docker. The actual fix is to set that env var, both in Docker and eventually on the server. Now we want to have a test that our fix works, and we looked in to a few different options, settling on using the filebased.EmailBackend" `EMAIL_BACKEND setting using the EMAIL_FILE_PATH environment variable.

Now, I say we haven’t seen the test fail, but actually we have, when we repro’d the bug. If we unset the EMAIL_PASSWORD env var, it will fail again. I’m more worried about the new parts of our tests, the bits where we go and read from the file at EMAIL_FILE_PATH. How can we make that part fail?

Well, how about if we deliberately break our email-sending code?

src/accounts/views.py (ch23l005)

def send_login_email(request):
    email = request.POST["email"]
    token = Token.objects.create(email=email)
    url = request.build_absolute_uri(
        reverse("login") + "?token=" + str(token.uid),
    )
    message_body = f"Use this link to log in:\n\n{url}"
    # send_mail(  (1)
    #     "Your login link for Superlists",
    #     message_body,
    #     "noreply@superlists",
    #     [email],
    # )
    messages.success(
        request,
        "Check your email, we've sent you a link you can use to log in.",
    )
    return redirect("/")

1	We just comment out the entire send_email block.

We rebuild our docker image:

# check our env var is set
$ echo $EMAIL_FILE_PATH
/tmp/superlists-emails
$ docker build -t superlists . && docker run \
    -p 8888:8888 \
    --mount type=bind,source=./src/db.sqlite3,target=/src/db.sqlite3 \
    --mount type=bind,source=$EMAIL_FILE_PATH,target=$EMAIL_FILE_PATH \
    -e DJANGO_SECRET_KEY=sekrit \
    -e DJANGO_ALLOWED_HOST=localhost \
    -e EMAIL_PASSWORD \
    -e EMAIL_FILE_PATH \
    -it superlists

And we re-run our test:

$ *TEST_SERVER=localhost:8888 EMAIL_FILE_PATH=/tmp/superlists-emails \
  ./src/manage.py test functional_tests.test_login
[...]
Ran 1 test in 2.513s

OK

Eh? How did that pass?

Testing side-effects is fiddly!

We’ve run into an example of the kinds of problems you often encounter when our tests involve side-effects.

Let’s have a look in our test emails directory:

$ ls $EMAIL_FILE_PATH
20241120-153150-262004991022080.log
20241120-153154-262004990980688.log
20241120-153301-272143941669888.log

Every time we restart the server, it opens a new file, but only when it first tries to send an email. Because we’ve commented out the whole email-sending block, our test instead picks up on an old email, which still has a valid url in it, because the token is still in the database.

You’ll run into a similar issue if you test with "real" emails in POP3. How do you make sure you’re not picking up an email from a previous test run?

Let’s clear out the db:

$ rm src/db.sqlite3 && ./src/manage.py migrate
Operations to perform:
  Apply all migrations: accounts, auth, contenttypes, lists, sessions
Running migrations:
  Applying accounts.0001_initial... OK
  Applying accounts.0002_token... OK
  Applying contenttypes.0001_initial... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0001_initial... OK

And…

cmdgg

$ TEST_SERVER=localhost:8888 ./src/manage.py test functional_tests.test_login
[...]
ERROR: test_login_using_magic_link (functional_tests.test_login.LoginTest.test_login_using_magic_link)
    self.wait_to_be_logged_in(email=TEST_EMAIL)
    _~_~_~_~_~_~_~_~~^{^}^{^}^{^}^{^}^{^}^{^}
[...]
selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: #id_logout; [...]

OK sure enough, the wait_to_be_logged_in() helper is failing, because now, although we have found an email, its token is invalid.

Here’s another way to make the tests fail:

$ rm $EMAIL_FILE_PATH/*

Now when we run the FT:

$ TEST_SERVER=localhost:8888 ./src/manage.py test functional_tests.test_login
ERROR: test_login_using_magic_link
(functional_tests.test_login.LoginTest.test_login_using_magic_link)
[...]
    email_body = self.wait_for_email(TEST_EMAIL, SUBJECT)
[...]
    return self.wait_for(
           _~_~_~_~~^
        lambda: self.retrieve_email_from_file(sent_to, subject, email_file_path)
        ^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}^{^}
[...]
    latest_emails_file = sorted(Path(emails_dir).iterdir())[-1]
                         _~_~_~_~_~_~_~_~_~_~_~~^{^}^
IndexError: list index out of range

We see there are no email files, because we’re not sending one.

In this configuration of Docker + filebase.EmailBackend, we now have to manage side effects in two locations: the database at src/db.sqlite3, and the email files in /tmp. What Django used to do for us thanks to LiveServerTestCase is now all our responsibility, and as you can see, it’s hard to get right. This is a tradeoff to be aware of when writing tests against "real" systems.

Still, this isn’t quite satisfactory. Let’s try a different way to make our tests fail, where we will send an email, but we’ll give it the wrong contents:

src/accounts/views.py (ch23l006)

def send_login_email(request):
    email = request.POST["email"]
    token = Token.objects.create(email=email)
    url = request.build_absolute_uri(
        reverse("login") + "?token=" + str(token.uid),
    )
    message_body = f"Use this link to log in:\n\n{url}"
    send_mail(
        "Your login link for Superlists",
        "HAHA NO LOGIN URL FOR U",  (1)
        "noreply@superlists",
        [email],
    )
    messages.success(
        request,
        "Check your email, we've sent you a link you can use to log in.",
    )
    return redirect("/")

1	We do send an email, but it won’t contain a login URL.

Let’s rebuild again:

# check our env var is set
$ echo $EMAIL_FILE_PATH
/tmp/superlists-emails
$ docker build -t superlists . && docker run \
    -p 8888:8888 \
    --mount type=bind,source=./src/db.sqlite3,target=/src/db.sqlite3 \
    --mount type=bind,source=$EMAIL_FILE_PATH,target=$EMAIL_FILE_PATH \
    -e DJANGO_SECRET_KEY=sekrit \
    -e DJANGO_ALLOWED_HOST=localhost \
    -e EMAIL_PASSWORD \
    -e EMAIL_FILE_PATH \
    -it superlists

Now how do our tests look?

$ TEST_SERVER=localhost:8888 python src/manage.py test functional_tests
FAIL: test_login_using_magic_link
(functional_tests.test_login.LoginTest.test_login_using_magic_link)
[...]
    email_body = self.wait_for_email(TEST_EMAIL, SUBJECT)
[...]
    self.assertIn("Use this link to log in", email_body)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'Use this link to log in' not found in 'Content-Type:
text/plain; charset="utf-8"\nMIME-Version: 1.0\nContent-Transfer-Encoding:
7bit\nSubject: Your login link for Superlists\nFrom: noreply@superlists\nTo:
[email protected]\nDate: Wed, 13 Nov 2024 18:00:55 -0000\nMessage-ID:
[...]\n\nHAHA NO LOGIN URL FOR
U\n-------------------------------------------------------------------------------\n'

OK good, that’s the error we wanted! I think we can be fairly confident that this testing setup can genuinely test that emails are sent properly. Let’s revert our temporarily-broken views.py, rebuild, and make sure the tests pass once again.

$ git stash
$ docker build [...]
# separate terminal
$ *TEST_SERVER=localhost:8888 EMAIL_FILE_PATH=/tmp/superlists-emails [...]
[...]
OK

It may seem like I’ve gone through a lot of back-and-forth, but I wanted to give you a flavour of the fiddliness involved in these kinds of tests that involve a lot of side-effects.

Decision Time: Which Test Strategy Will We Keep

Let’s recap our three options:

Table 1. Testing Strategy Tradeoffs
Strategy	Pros	Cons
End-to-end with POP3	Maximally realistic, tests the whole system	Slow, fiddly, unreliable
File-based fake email backend	Faster, more reliable, no network calls, tests end-to-end (albeit with fake components)	Still Fiddly, requires managing db & filesystem side-effects
Give up on testing email on the server/Docker	Fast, simple	Less confidence that things work "for real"

This is a common problem in testing integration with external systems, how far should we go? How realistic should we make our tests?

In the book in the end, I suggested we go for the last option, ie give up. Email itself is a well-understood protocol (reader, it’s been around since before I was born, and that’s a whiles ago now) and Django has supported sending email for more than a decade, so I think we can afford to say, in this case, that the costs of building testing tools for email outweigh the benefits.

But not all external dependencies are as well-understood as email. If you’re working with a new API, or a new service, you may well decide it’s worth putting in the effort to get a "real" end-to-end functional test to work.

TODO: recap