Testing Deployment Using a Staging Site
Is all fun and game until you are need of put it in production.
|2018-01-09 — I’ve just made some major changes to this and the next three chapters. If you’re going through it now, can I ask you to please send feedback — what was easy, what was hard, what mistakes did you make, did you get stuck, how could the instructions or explanations be improved? [email protected]|
It’s time to deploy the first version of our site and make it public. They say that if you wait until you feel ready to ship, then you’ve waited too long.
Is our site usable? Is it better than nothing? Can we make lists on it? Yes, yes, yes.
No, you can’t log in yet. No, you can’t mark tasks as completed. But do we really need any of that stuff? Not really—and you can never be sure what your users are actually going to do with your site once they get their hands on it. We think our users want to use the site for to-do lists, but maybe they actually want to use it to make "top 10 best fly-fishing spots" lists, for which you don’t need any kind of “mark completed” function. We won’t know until we put it out there.
In this chapter we’re going to go through and actually deploy our site to a real, live web server.
You might be tempted to skip this chapter—there’s lots of daunting stuff in it, and maybe you think this isn’t what you signed up for. But I strongly urge you to give it a go. This is one of the sections of the book I’m most pleased with, and it’s one that people often write to me saying they were really glad they stuck through it.
If you’ve never done a server deployment before, it will demystify a whole world for you, and there’s nothing like the feeling of seeing your site live on the actual internet. Give it a buzzword name like "DevOps" if that’s what it takes to convince you it’s worth it.
|Why not ping me a note once your site is live on the web, and send me the URL? It always gives me a warm and fuzzy feeling… [email protected].|
TDD and the Danger Areas of Deployment
Deploying a site to a live web server can be a tricky topic. Oft-heard is the forlorn cry "but it works on my machine!"
Some of the danger areas of deployment include:
Once we’re off our own machine, networking issues come in: making sure the DNS service is routing our domain to the correct IP address for our server, making sure our server is configured to listen to traffic coming in from the world, making sure it’s using the right ports, and making sure any firewalls n the way are configured to let traffic through.
We need to make sure that the packages our software relies on (Python, Django, and so on) are installed on the server, and have the correct versions.
- The database
There can be permissions and path issues, and we need to be careful about preserving data between deploys.
Web servers usually need special configuration for serving these.
But there are solutions to all of these. In order:
Using a staging site, on the same infrastructure as the production site, can help us test out our deployments and get things right before we go to the "real" site.
We can also run our functional tests against the staging site. That will reassure us that we have the right code and packages on the server, and since we now have a "smoke test" for our site layout, we’ll know that the CSS is loaded correctly.
Just like on our own PC, a virtualenv is useful on the server for managing packages and dependencies when you might be running more than one Python application.
And finally, automation, automation, automation. By using an automated script to deploy new versions, and by using the same script to deploy to staging and production, we can reassure ourselves that staging is as much like live as possible.
Over the next few pages I’m going to go through a deployment procedure. It isn’t meant to be the perfect deployment procedure, so please don’t take it as being best practice, or a recommendation—it’s meant to be an illustration, to show the kinds of issues involved in deployment and where testing fits in.
As Always, Start with a Test
adapt our functional tests slightly so that it can be run against
a staging site, instead of the local dev server. We’ll do it by checking for an
environment variable called
import os [...] class NewVisitorTest(StaticLiveServerTestCase): def setUp(self): self.browser = webdriver.Firefox() staging_server = os.environ.get('STAGING_SERVER') (1) if staging_server: self.live_server_url = 'http://' + staging_server (2)
Do you remember I said that
LiveServerTestCase had certain limitations?
Well, one is that it always assumes you want to use its own test server, which
it makes available at
self.live_server_url. I still want to be able to do
that sometimes, but I also want to be able to selectively tell it not to
bother, and to use a real server instead.
|1||The way I decided to do it is using an environment variable called
|2||Here’s the hack: we replace
We test that said hack hasn’t broken anything by running the functional tests "normally":
$ python manage.py test functional_tests [...] Ran 3 tests in 8.544s OK
And now we can try them against our staging server URL. I’m planning to host my staging server at superlists-staging.ottg.eu:
|A clarification: in this chapter, we run tests against our staging server, not on our staging server. So we still run the tests from our own laptop, but they target the site that’s running on the server.|
$ STAGING_SERVER=superlists-staging.ottg.eu python manage.py test functional_tests EEE ====================================================================== ERROR: test_can_start_a_list_for_one_user (functional_tests.tests.NewVisitorTest) --------------------------------------------------------------------- Traceback (most recent call last): File "...python-tdd-book/functional_tests/tests.py", line 41, in test_can_start_a_list_for_one_user self.browser.get(self.live_server_url) [...] selenium.common.exceptions.WebDriverException: Message: Reached error page: abo ut:neterror?e=connectionFailure&u=http%3A//superlists-staging.ottg.eu/&c=UTF-8& f=regular&d=Firefox%20can%27t%20establish%20a%20connection%20to%20the%20server% 20at%20superlists-staging.ottg.eu. ====================================================================== ERROR: test_layout_and_styling (functional_tests.tests.NewVisitorTest) --------------------------------------------------------------------- Traceback (most recent call last): File "...python-tdd-book/functional_tests/tests.py", line 126, in test_layout_and_styling [...] selenium.common.exceptions.WebDriverException: Message: Reached error page: abo [...] ====================================================================== ERROR: test_multiple_users_can_start_lists_at_different_urls (functional_tests.tests.NewVisitorTest) --------------------------------------------------------------------- Traceback (most recent call last): File "...python-tdd-book/functional_tests/tests.py", line 80, in test_multiple_users_can_start_lists_at_different_urls [...] selenium.common.exceptions.WebDriverException: Message: Reached error page: abo [...] Ran 3 tests in 10.518s FAILED (errors=3)
|If, on Windows, you see an error saying something like "STAGING_SERVER is not recognized as a command", it’s probably because you’re not using Git-Bash. Take another look at the “[pre-requisites]” section.|
You can see that all the tests are failing, as expected, since I haven’t actually set up my domain yet. Selenium reports that Firefox is seeing an error and "cannot establish connection to the server" (depending on your registrar, you might see content from its default landing page instead).
The FT seems to be testing the right things though, so let’s commit:
$ git diff # should show changes to functional_tests.py $ git commit -am "Hack FT runner to be able to test staging"
Getting a Domain Name
We’re going to need a couple of domain names at this point in the book—they can both be subdomains of a single domain. I’m going to use superlists.ottg.eu and superlists-staging.ottg.eu. If you don’t already own a domain, this is the time to register one! Again, this is something I really want you to actually do. If you’ve never registered a domain before, just pick any old registrar and buy a cheap one—it should only cost you $5 or so, and you can even find free ones. I promise seeing your site on a "real" website will be a thrill.
Manually Provisioning a Server to Host Our Site
We can separate out "deployment" into two tasks:
Provisioning a new server to be able to host the code
Deploying a new version of the code to an existing server
Some people like to use a brand new server for every deployment—it’s what we do at PythonAnywhere. That’s only necessary for larger, more complex sites though, or major changes to an existing site. For a simple site like ours, it makes sense to separate the two tasks. And, although we eventually want both to be completely automated, we can probably live with a manual provisioning system for now.
As you go through this chapter, you should be aware that provisioning is something that varies a lot, and that as a result there are few universal best practices for deployment. So, rather than trying to remember the specifics of what I’m doing here, you should be trying to understand the rationale, so that you can apply the same kind of thinking in the specific future circumstances you encounter.
Choosing Where to Host Our Site
There are loads of different solutions out there these days, but they broadly fall into two camps:
Running your own (possibly virtual) server
Using a Platform-As-A-Service (PaaS) offering like Heroku, OpenShift, or PythonAnywhere
Particularly for small sites, a PaaS offers a lot of advantages, and I would definitely recommend looking into them. We’re not going to use a PaaS in this book however, for several reasons. Firstly, I have a conflict of interest, in that I think PythonAnywhere is the best, but then again I would say that because I work there. Secondly, all the PaaS offerings are quite different, and the procedures to deploy to each vary a lot—learning about one doesn’t necessarily tell you about the others. Any one of them might change their process or business model radically by the time you get to read this book.
Instead, we’ll learn just a tiny bit of good old-fashioned server admin, including SSH and web server config. They’re unlikely to ever go away, and knowing a bit about them will get you some respect from all the grizzled dinosaurs out there.
What I have done is to try to set up a server in such a way that’s a bit like the environment you get from a PaaS, so you should be able to apply the lessons we learn in the deployment section, no matter what provisioning solution you choose.
Spinning Up a Server
I’m not going to dictate how you do this—whether you choose Amazon AWS, Rackspace, Digital Ocean, your own server in your own data centre or a Raspberry Pi in a cupboard under the stairs, any solution should be fine, as long as:
Your server is running Ubuntu 16.04 (aka "Xenial/LTS").
You have root access to it.
It’s on the public internet.
You can SSH into it.
I’m recommending Ubuntu as a distro because it’s easy to get Python 3.6 on it and it has some specific ways of configuring Nginx, which I’m going to make use of next. If you know what you’re doing, you can probably get away with using something else, but you’re on your own.
If you’ve never started a Linux server before and you have absolutely no idea where to start, I wrote a very brief guide on GitHub.
|Some people get to this chapter, and are tempted to skip the domain bit, and the "getting a real server" bit, and just use a VM on their own PC. Don’t do this. It’s not the same, and you’ll have more difficulty following the instructions, which are complicated enough as it is. If you’re worried about cost, have a look at the link above for free options.|
User Accounts, SSH, and Privileges
In these instructions, I’m assuming that you have a nonroot user account set up that has "sudo" privileges, so whenever we need to do something that requires root access, we use sudo, and I’m explicit about that in the various instructions that follow.
My user is called "elspeth", but you can call yours whatever you like!
Installing Python 3.6
Python 3.6 wasn’t available in the standard repositories on Ubuntu at the time of writing, but the user-contributed "Deadsnakes PPA" has it. Here’s how we install it:
Look out for that
And while we’re at it, we’ll just make sure Git is installed too.
[email protected]:$ sudo apt install git
Configuring Domains for Staging and Live
We don’t want to be messing about with IP addresses all the time, so we should point our staging and live domains to the server. At my registrar, the control screens looked a bit like Domain setup.
In the DNS system, pointing a domain at a specific IP address is called an "A-Record". All registrars are slightly different, but a bit of clicking around should get you to the right screen in yours. You’ll need two A-records, one for the staging address and one for the live one. No need to worry about any other type of record.
DNS records take some time to "propagate" around the world (it’s controlled by a setting called "TTL", Time To Live), so once you’ve set up your A-record, you can check its progress on a "propagation checking" service like this one:
Deploying Our Code Manually
The next step is to get a basic copy of the staging site up and running As we do so, we’re starting to move into doing "deployment" rather than provisioning, so we should be thinking about how we can automate the process as we go.
|One rule of thumb for distinguishing provisioning from deployment is that you tend to need root permissions for the former, but you don’t for the latter.|
We need a directory for the source to live in. We’ll put it somewhere in the home folder of our nonroot user; in my case it would be at /home/elspeth (this is likely to be the setup on any shared hosting system, but you should always run your web apps as a nonroot user, in any case). I’m going to set up my sites like this:
/home/elspeth ├── sites │ ├── www.live.my-website.com │ │ ├── db.sqlite3 │ │ ├── manage.py │ │ ├── [etc...] │ │ ├── static │ │ │ ├── base.css │ │ │ ├── [etc...] │ │ └── virtualenv │ │ ├── lib │ │ ├── [etc...] │ │ │ ├── www.staging.my-website.com │ │ ├── db.sqlite3 │ │ ├── [etc...]
Each site (staging, live, or any other website) has its own folder, which will contain a checkout of the source code (managed by git), along with the database, static files and virtualenv (managed separately).
To get our code onto the server, we’ll use Git and go via one of the code-sharing sites. If you haven’t already, push your code up to GitHub, BitBucket, GitLab, or similar. They all have excellent instructions for beginners on how to do that.
Here are some Bash commands that will set this all up.
exportcommand sets up a "local variable" in Bash; a bit like the inline environment variable we used earlier, but it’s available to all subsequent commands in that same shell.
git clonetakes your repo URL as its first argument, and an (optional) destination as its second argument. That will create the target folder for us and get our code into the right place in one go.
A Bash variable defined using
Now we’ve got the code, let’s just try running the dev server, and see how far we get:
[email protected]:$ $ cd ~/sites/$SITENAME $ python3.6 manage.py runserver Traceback (most recent call last): File "manage.py", line 8, in <module> from django.core.management import execute_from_command_line ImportError: No module named django [...] ImportError: Couldn't import Django. Are you sure it's installed and available on your PYTHONPATH environment variable? Did you forget to activate a virtual environment?
Ah. Django isn’t installed on the server.
Creating a Virtualenv on the server Using requirements.txt
Just like on our own machine, a virtualenv is useful on the server to make sure we have full control over the packages installed for a particular project. It can also let us run different projects with different (or conflicting) dependencies on the same server.
To reproduce our local virtualenv, we can "save" the list of packages we’re using by creating a requirements.txt file. Back on our own machine:
$ echo "django==1.11" > requirements.txt $ git add requirements.txt $ git commit -m "Add requirements.txt for virtualenv"
|You may be wondering why we didn’t add our other dependency, Selenium, to our requirements. The reason is that Selenium is only a dependency for the tests, not the application code (we’re never going to run the tests on the server itself). Some people like to also create a file called test-requirements.txt.|
Now we do a
git push to send our updates up to our code-sharing site:
$ git push
And we can pull those changes down to the server:
[email protected]:$ git pull # may ask you to do some git config first
We create our virtualenv just like we did on our own machine:
If we wanted to activate the virtualenv, we could do so with
source ./virtualenv/bin/activate just like we do locally, but on the
server we don’t need that. We can actually do everything we want to by directly
calling the versions of Python, pip, and the other executables in the
virtualenv’s bin directory, as we’ll soon see.
For example, to install our requirements into the virtualenv, we use the virtualenv pip:
[email protected]:$ ./virtualenv/bin/pip install -r requirements.txt Collecting django==1.11 (from -r requirements.txt (line 1)) [...] Successfully installed django-1.11 pytz-2017.3
And to run Python in the virtualenv, we use the virtualenv
[email protected]:$ ./virtualenv/bin/python manage.py runserver Performing system checks... System check identified no issues (0 silenced). [...] You have 15 unapplied migration(s). Your project may not work [...] [...] Starting development server at http://127.0.0.1:8000/
If we ignore the ominous message about migrations for now, Django certainly looks a lot happier.
Progress! We’ve got a system for getting code to and from the server
git push and
git pull), we’ve got a virtualenv set up to match our local
one, and a single file, requirements.txt, to keep them in sync.
Using the FT to check our deployment works
Let’s see what our FTs think about this version of our site running on
the server. I’ll use the
--failfast option to exit as soon as a single test
$ STAGING_SERVER=superlists-staging.ottg.eu ./manage.py test functional_tests --failfast [...] selenium.common.exceptions.WebDriverException: Message: Reached error page: [...]
Nope! What’s going on here? Time for a little debugging.
Debugging a deployment that doesn’t seem to work at all
You may remember that Django’s runserver usually chooses to run on port 8000. But a "normal" web server should run on port 80, and that’s where our FTs are currently looking, on superlists-staging.ottg.eu.
But we can actually use our
STAGING_SERVER variable to point the tests at
port 8000. Let’s try that:
$ STAGING_SERVER=superlists-staging.ottg.eu:8000 ./manage.py test functional_tests --failfast selenium.common.exceptions.WebDriverException: Message: Reached error page: [...]
Nope, that didn’t work earlier. Let’s try an even lower-level smoke test, the traditional Unix utility "curl" — it’s a command-line tool for making web requests. Try it on your own computer first:
$ curl superlists-staging.ottg.eu curl: (7) Failed to connect to superlists-staging.ottg.eu port 80: Connection refused
And maybe just to be sure, we could even open up our web browser and type in http://superlists-staging.ottg.eu:8000, and confirm using a familiar tool that things aren’t working. Nope.
We’re pretty sure the server is running and listening on port 8000, but we
can’t get to it from the outside. What about from the inside? Try
curl on the server itself:
(you’ll need a second SSH shell onto your server, so as not to interrupt the
[email protected]:$ curl localhost:8000 <!DOCTYPE html> <html lang="en"> <head> [...] <title>To-Do lists</title> [...] </body> </html>
Ah-ha! That looks like the HTML for our site. So we can reach it from the server itself, just not from the outside. What could be going on?
Actually there’s clue in the output that Django printed out earlier when
Starting development server at http://127.0.0.1:8000/
Django’s development server is configured to listen on 127.0.0.1, aka the "localhost" IP address. But we’re trying to reach it from the outside, via the server’s "real" public address.
But Django isn’t listening on that address by default.
Here’s how we tell it to listen on all addresses. Use Ctrl-C to
runserver process, and restart it like this:
[email protected]:$ ./virtualenv/bin/python manage.py runserver 0.0.0.0:8000 [...] Starting development server at http://0.0.0.0:8000/
And in a second SSH shell, we can confirm it works from the server:
[email protected]:$ curl localhost:8000 <!DOCTYPE html> [...] </html>
What about from our own laptop?
$ curl superlists-staging.ottg.eu:8000 <!DOCTYPE html> <html lang="en"> [...] </body> </html>
Looks good at first glance! Let’s try our FTs again:
$ STAGING_SERVER=superlists-staging.ottg.eu:8000 ./manage.py test functional_tests --failfast ====================================================================== FAIL: test_can_start_a_list_for_one_user (functional_tests.tests.NewVisitorTest) --------------------------------------------------------------------- Traceback (most recent call last): File "...python-tdd-book/functional_tests/tests.py", line 44, in test_can_start_a_list_for_one_user self.assertIn('To-Do', self.browser.title) AssertionError: 'To-Do' not found in 'DisallowedHost at /' --------------------------------------------------------------------- Ran 1 test in 4.010s FAILED (failures=1) [...]
|At this point, if your FTs still can’t talk to the server, something else must be in the way. Check your provider’s firewall settings, and make sure ports 80 and 8000 are open to the world. On AWS, for example, you may need to configure the "security group" for your server.|
Oops, spoke too soon! Another error. We didn’t look closely enough at
Hacking ALLOWED_HOSTS in settings.py
Don’t be disheartened! We may have just fixed one problem only to run straight into another, but this problem is definitely a much easier one. At least we can talk to the server! And it’s giving us a helpful pointer. Try opening the site manually Another hitch along the way:
ALLOWED_HOSTS is a security setting designed to reject requests that are
likely to be forged, broken or malicious because they don’t appear to be
asking for your site (HTTP request contain the address they were intended for
in a header called "Host").
By default, when DEBUG=True,
ALLOWED_HOSTS effectively allows localhost,
our own machine, so that’s why it was working OK in dev, and from the server
itself (where we ask for localhost), but not from our own machine (where we
ask for superlists-staging.ottg.eu)
There’s more information in the Django docs.
The upshot is that we need to adjust
ALLOWED_HOSTS in settings.py. Since
we’re just hacking for now, let’s set it to the totally insecure allow-everyone
# SECURITY WARNING: don't run with debug turned on in production! DEBUG = True ALLOWED_HOSTS = ['*'] [...]
We commit that locally, then push it up to github…
$ git commit -am "hack ALLOWED_HOSTS to be *" $ git push
And pull it down on the server, and restart our
A quick visual inspection confirms—the site is up (The staging site is up!)!
Let’s see what our functional tests say:
$ STAGING_SERVER=superlists-staging.ottg.eu:8000 ./manage.py test functional_tests --failfast [...] selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: [id="id_list_table"]
The tests are failing as soon as they try to submit a new item, because we haven’t set up the database. You’ll probably have spotted the yellow Django debug page (But the database isn’t) telling us as much as the tests went through, or if you tried it manually.
|The tests saved us from potential embarrassment there. The site looked fine when we loaded its front page. If we’d been a little hasty and only testing manually, we might have thought we were done, and it would have been the first users that discovered that nasty Django DEBUG page. Okay, slight exaggeration for effect, maybe we would have checked, but what happens as the site gets bigger and more complex? You can’t check everything. The tests can.|
Creating the Database with migrate
migrate using the
--noinput argument to suppress the two little "are
you sure" prompts:
[email protected]:$ ./virtualenv/bin/python manage.py migrate --noinput Operations to perform: Apply all migrations: auth, contenttypes, lists, sessions Running migrations: Applying contenttypes.0001_initial... OK [...] Applying lists.0004_item_list... OK Applying sessions.0001_initial... OK
That looks good. We restart the server:
[email protected]:$ ./virtualenv/bin/python manage.py runserver 0.0.0.0:8000
And try the FTs again:
$ STAGING_SERVER=superlists-staging.ottg.eu:8000 ./manage.py test functional_tests [...] ... --------------------------------------------------------------------- Ran 3 tests in 10.718s OK
Hooray, that’s a working deploy!
Time for a well-earned tea break I think, and perhaps a Chocolate biscuit.
Success! Our Hack Deployment Works
Phew. Well, it took a bit of hacking about, but now we can be reassured that the basic piping works. Notice that the FT was able to guide us incrementally towards a working site.
But we really can’t be using the Django dev server in production, or running on port 8000 forever. In the next chapter, we’ll make our hacky deployment more production-ready.