Much of the focus of Docker is on the process of packaging and running your application in an isolated container. There are countless tutorials that explain how to run your application in a Docker container, but very few that discuss how properly stop your containerized app. That may seem like a silly topic — who cares how you stop a container, right?
Well, depending on your application, the process by which you stop your app could be very important. If your application is serving HTTP requests you may want to complete any outstanding requests before you shutdown your container. If your application writes to a file, you probably want to ensure that the data is properly flushed and the file is closed before your container exits.
Things would be easy if you simply started a container and it ran forever, but there’s a good chance that your application will need to be stopped and restarted at some point to facilitate an upgrade or a migration to another host. For those times when you need to stop a running container, it would be preferable if the process could shutdown smoothly instead of abruptly disconnecting users and corrupting files.
So, let’s look at some of the things you can do to gracefully stop your Docker containers.
Sending Signals
There are a number of different Docker commands you can use to stop a running container.
docker stop
When you issue a docker stop command Docker will first ask nicely for the process to stop and if it doesn’t comply within 10 seconds it will forcibly kill it. If you’ve ever issued a docker stop and had to wait 10 seconds for the command to return you’ve seen this in action
The docker stop command attempts to stop a running container first by sending a SIGTERM signal to the root process (PID 1) in the container. If the process hasn’t exited within the timeout period a SIGKILL signal will be sent.
Whereas a process can choose to ignore a SIGTERM, a SIGKILL goes straight to the kernel which will terminate the process. The process never even gets to see the signal.
When using docker stop the only thing you can control is the number of seconds that the Docker daemon will wait before sending the SIGKILL:
docker stop ----time=30 foo
docker kill
By default, the docker kill command doesn’t give the container process an opportunity to exit gracefully — it simply issues a SIGKILL to terminate the container. However, it does accept a --signal flag which will let you send something other than a SIGKILL to the container process.
For example, if you wanted to send a SIGINT (the equivalent of a Ctrl-C on the terminal) to the container «foo» you could use the following:
docker kill ----signal=SIGINT foo
Unlike the docker stop command, kill doesn’t have any sort of timeout period. It issues just a single signal (either the default SIGKILL or whatever you specify with the --signal flag).
Note that the default behavior of the docker kill command is different than the standard Linux kill command it is modeled after. If no other arguments are specified, the Linux kill command will send a SIGTERM (much like docker stop). On the other hand, using docker kill is more like doing a Linux kill -9 or kill -SIGKILL.
docker rm -f
The final option for stopping a running container is to use the --force or -f flag in conjunction with the docker rm command. Typically, docker rm is used to remove an already stopped container, but the use of the -f flag will cause it to first issue a SIGKILL.
docker rm ----force foo
If your goal is to erase all traces of a running container, then docker rm -f is the quickest way to achieve that. However, if you want to allow the container to shutdown gracefully you should avoid this option.
Handling Signals
While the operating system defines a set list of signals, the way in which a process responds to a particular signal is application-specific.
For example, if you want to initiate a graceful shutdown of an nginx server, you should send a SIGQUIT. None of the Docker commands issue a SIGQUIT by default so you’d need use the docker kill command as follows:
docker kill ----signal=SIGQUIT nginx
The nginx log output upon receiving the SIGQUIT would look something like this:
2015/05/11 20:30:20 [notice] 1#0: signal 3 (SIGQUIT) received, shutting down
2015/05/11 20:30:20 [notice] 9#0: gracefully shutting down
2015/05/11 20:30:20 [notice] 9#0: exiting
2015/05/11 20:30:20 [notice] 9#0: exit
2015/05/11 20:30:20 [notice] 1#0: signal 17 (SIGCHLD) received
2015/05/11 20:30:20 [notice] 1#0: worker process 9 exited with code 0
2015/05/11 20:30:20 [notice] 1#0: exit
In contrast, Apache uses SIGWINCH to trigger a graceful shutdown:
docker kill ----signal=SIGWINCH apache
According to the Apache documentation a SIGTERM will cause the server to immediately exit and terminate any in-progress requests, so you may not want to use docker stop on an Apache container.
If you’re running a third-party application in a container you may want to review the app’s documentation to understand how it responds to different signals. Simply running a docker stop may not give you the result you want.
When running your own application in a container, you must decide how the different signals will be interpreted by your app. You will need to make sure you are trapping the relevant signals in your application code and taking the necessary actions to cleanly shutdown the process.
If you know that you’re going to package your application in a Docker image you might consider using SIGTERM as your graceful shutdown signal since this is what the docker stop command sends.
No matter which language you’re using, there is a good chance that it supports some form of signal handling. I’ve collected links to the relevant package/module/library for a handful of languages in the list below:
- Bash
- Ruby
- Python
- Node
- Go
If you’re using Go for your application, take a look at the tylerb/graceful package which automatically enables the graceful shutdown of http.Handler servers in response to SIGINT or SIGTERM signals.
Receiving Signals
Coding your application to gracefully shutdown in response to a particular signal is a good first step, but you also need to ensure that your application is packaged in such a way that it has a chance to receive the signals sent by the Docker commands. If you’re not careful in how you launch your application it may never receive any of the signals sent by docker stop or docker kill.
To demonstrate, let’s create a simple application that we’ll run inside a Docker container:
#!/usr/bin/env bash
trap 'exit 0' SIGTERM
while true; do :; done
This trivial bash script simply goes into an infinite loop, but will exit with a 0 status if it receives a SIGTERM.
We’ll package this into a Docker image with the following Dockerfile:
FROM ubuntu:trusty
COPY loop.sh /
CMD /loop.sh
This will simply copy our loop.sh bash script into an Ubuntu-based image and set it as the default command for the running container.
Now, let’s build this image, start a container, and then immediately stop it.
$ docker build -t loop .
Sending build context to Docker daemon 3.072 kB
Sending build context to Docker daemon
Step 0 : FROM ubuntu:trusty
---> 07f8e8c5e660
Step 1 : COPY loop.sh /
---> 161f583a7028
Removing intermediate container e0988f66358a
Step 2 : CMD /loop.sh
---> Running in 6d6664be02da
---> 18b3feccee90
Removing intermediate container 6d6664be02da
Successfully built 18b3feccee90
$ docker run -d loop
64d39c3b49147f847722dbfd0c7976315533a729d9453c34cb6cbdaa11d46c21
$ docker stop 64d39c3b
If you’re following along, you may have noticed that the docker stop command above took about 10 seconds to complete — this is typically a sign that your container didn’t respond to the SIGTERM and had to be forcibly terminated with a SIGKILL.
We can validate this by looking at the container’s exit status.
$ docker inspect -f '{{.State.ExitCode}}' 64d39c3b
137
Based on the handler we setup in our application, had our container received a SIGTERM we should have seen a 0 exit status, not 137. In fact, an exit status greater than 128 is typically a sign that the process was terminated as a result of an unhandled signal. 137 = 128 + 9 — meaning that the process was terminated as a result of signal number 9 (SIGKILL).
So, what happened here? Our application is coded to trap SIGTERM and exit gracefully. We know that docker stop sends a SIGTERM to the container process. Yet it appears that the signal never made it to our app.
To understand what happened here, let’s start another container and take a peek at the running processes.
$ docker run -d loop
512c36b5b517b3a43246b519bc5cdb756cdbc4d2ff1e0a3984e83b094f3db136
$ docker exec 512c36b5 ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 16:03 ? 00:00:00 /bin/sh -c /loop.sh
root 13 1 61 16:03 ? 00:00:10 bash /loop.sh
root 14 0 0 16:03 ? 00:00:00 ps -ef
The important thing to note in the output above is that our loop.sh script is NOT running as PID 1 inside the container. The script is actually running as a child of the /bin/sh process running at PID 1.
When you use docker stop or docker kill to signal a container, that signal is sent only to the container process running as PID 1.
Since /bin/sh doesn’t forward signals to any child processes, the SIGTERM we sent never reached our script. Clearly, if we want our app to be able to receive signals from the host we need to find a way to run it as PID 1.
To do this we need to go back to our Dockerfile and look at the CMD instruction used to launch our script. There are actually a few different forms the CMD instruction can take. In our Dockerfile above we used the shell form which looks like this:
CMD command param1 param2
When using the shell form, the specified command is executed with a /bin/sh -c shell. If you look back at the process list for our container you will see the process at PID 1 shows a command string of «/bin/sh -c /loop.sh». So the /bin/sh runs as PID 1 and then forks/execs our script.
Luckily, Docker also supports an exec form of the CMD instruction which looks like this:
CMD ["executable","param1","param2"]
Note that the content appearing after the CMD instruction in this case is formatted as a JSON array.
When the exec form of the CMD instruction is used the command will be executed without a shell.
Let’s change our Dockerfile to see this in action:
FROM ubuntu:trusty
COPY loop.sh /
CMD ["/loop.sh"]
Rebuild the image and look at the processes running in the container:
$ docker build -t loop .
[truncated]
$ docker run -d loop
4dda905ee902c91d1f56082d1092d6d72ef54b3d4582fe6b453cba90777554e2
$ docker exec 4dda905e ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 30 16:42 ? 00:00:04 bash /loop.sh
root 13 0 0 16:42 ? 00:00:00 ps -ef
Now, our script is running as PID 1. Let’s send a SIGTERM to the container and look at the exit status:
$ docker stop 4dda905e
$ docker inspect -f '{{.State.ExitCode}}' 4dda905e
This is exactly the result we were expecting! Our script received the SIGTERM sent by the docker stop command and exited cleanly with a 0 status.
The bottom line is that you should audit the processes inside your container to make sure they’re in a position to receive the signals you intend to send. Using the exec form of the CMD (or ENTRYPOINT) instruction in your Dockerfile is a good start.
Conclusion
It’s pretty easy to terminate a Docker container with a docker kill command, but if you actually want to wind-down your applications in an orderly fashion there is a little more work involved. You should now understand how to send signals to your containers, how to handle those signals in your custom applications and how to ensure that your apps can even receive those signals in the first place.