Best Practices for Running Docker in Production

Docker makes it far easier to deploy an application, but using Docker in production comes with security and operational risks that the developer has to deal with. You can certainly build a Docker image that works on your desktop, but the production environment is a whole lot more complicated and you need a more structured approach. When you’re in the situation of operations teams managing hundreds of containers, it’s very easy to lose the thread of what’s going on with the processes, the data, and the configuration. If one container fails, it can have a knock-on effect on other services and therefore your uptime. To address that, follow comprehensive Docker best practices.

1. Use Small, Secure Images

Image size has an impact on both performance and security. Bigger Docker images consume more resources, result in slower response times and are less efficient. They can also introduce security vulnerabilities which are more difficult to spot and remediate in production.

Use small base images that are appropriate for your application’s programming language. Many multi-language apps will use a large all-in-one image but you can keep things very lean with multi-stage Docker builds. For a Node.js app, for instance, you’d install all your dependencies during the build phase and then just copy over the files and code you need into a much smaller production image. In your Dockerfile, you’ll of course want to use a .dockerignore file to exclude all the other stuff you don’t need (all the dependencies, the version control folders, the test data, etc.).

Developers and ops teams should be working to reduce the attack surface of their production images. Strip out any third-party libraries and apps that aren’t absolutely necessary, and you not only reduce your risk but make maintenance a whole lot easier.

2. Run Applications without Root Privileges

A container’s process should be running as a normal user unless there’s a very good reason not to. An app with broad write permissions is a security risk regardless of how tightly you’ve locked down the Docker process itself.

Docker best practices demand that one should run the process as an unprivileged user with restricted access. Developers should use dedicated users and set up the appropriate permissions to give access to files and data and to grant process rights.

Ops engineers can use Linux privileges and restrictions to further secure their applications. Don’t use privileged containers and limit the Linux capabilities of the Docker daemon. The Docker engine and the underlying OS should both be hardened to reduce the attack surface. This will go some way to mitigating the risk of a network-based exploit compromising a container.

3. Limit CPU and Memory Utilization

Ops teams need to ensure that a Docker container can’t crash other processes or consume all the system memory. Set memory limits to prevent the app from spawning runaway processes that will consume all the CPU and bring the whole system to its knees.

The Dockerfile can specify memory and CPU requirements but the ops team should then cap the container’s resource usage well below the server’s total capacity. Don’t go too low with the limits, though, as you need to leave headroom for traffic spikes so the app remains available.

Regularly profile your app’s performance to determine how much CPU and memory to allocate to the Docker process. You’ll need to adjust those limits as your production workload changes as the amount of resources required isn’t a fixed value.

4. Design Services with Health Checks and Restart Policies

Sometimes the process will fail to get the app started properly or will hit a production issue. It may hang if it can’t get to an external service during startup for instance. The developer can deal with this by having health checks for the external service at regular intervals.

A health check within the container can be of use in that it can check that any external services are available prior to starting the app. Here’s a simple example assuming you have curl installed and the app has a health URL. You can then have different actions depending on the exit status of the command.

HEALTHCHECK –interval=30s –timeout=5s –start-period=20s –retries=3 \

CMD curl -fsS http://localhost:8080/health || exit 1

Processes will often crash or run into memory errors and therefore will just go down in production. Microservices apps can also go down if one of the services fails.

Ops teams should have restart policies in place for their Docker processes and applications so that they will restart after a crash, an OOM (out of memory) or a planned exit. The policy should also cover daemon and OS updates. Always restart is the policy you’d use for services that need to be up and running. Unless stopped is the policy you’d use for services that you’d just leave running unless you specifically stopped them. Again, restart policies are not a silver bullet, and you will still need to use crash logs, app monitoring, and alerts to identify and resolve the root cause of any outages.

5. Store Information in Volumes

A Docker container only retains the data that it’s configured to save. If it isn’t, you’re very much at risk of losing that data when the process terminates. The solution is to store your app data in a volume (a folder on the host OS) which will persist even if the container itself is deleted.

Ensure the volume name is available on the host so that the process can use it to store the data. Bind mounts are an alternative but can be quite unstable due to OS level permission or location issues.

Backing up the volumes is just as important. Take regular snapshots, define how long you want to keep them and ensure that backup copies are in place and ready to go should the main data become corrupted or tampered with.

Only back up the volumes that are critical to your operations. Disk problems on the host can still occur, so you need to be ready to recover from an incomplete volume purge (particularly for key applications). As volumes contain the key data, you need to have a recovery strategy in place, and you need to regularly test your ability to restore the data.

6. Secure Secrets and Certificates

Apps need secrets of various kinds to run in production (database passwords, API tokens etc). These are typically not set in the Dockerfile and are instead set via environment variables. This can be a problem as any process that can read the environment will be able to read those secrets.

Best practice is to use different tokens/certificates for dev, test, and production and to restrict the app’s access to those credentials. Don’t put your secrets in the Dockerfile (use a secrets management system that’s appropriate to your deployment environment).

Docker Swarm has its own built-in secret management for apps running across multiple hosts. If you’re using a cloud service or some other tool to manage your Docker then use the secret management capabilities of that service to protect your data. Only give your apps the credentials they need and always send them over an encrypted channel. That way you don’t end up with secrets being shared in the wrong way.

7. Leverage Image Scans to Address Vulnerabilities

Images that contain potentially vulnerable Python packages or libraries should be scanned for vulnerabilities prior to deployment to production. Host OS issues are not relevant in this context as this is purely an OS level issue and is separate from Docker.

Teams can use tools to find and fix issues prior to deploying images into production. These tools can be incorporated into your CI/CD pipeline. You should only scan and deploy images when you’re ready for production, not just the next available tag. Remember that image tags can change so you need to keep your production images consistent.

Developers and ops teams should be installing images from source code in a version-controlled repository, scanning them, testing them, and then deploying to production. They should be using a consistent tagging strategy when moving images into production to avoid confusion and to keep the version control clean. In some cases, you might want to tag an image with its digest to ensure that the same software is running in production. You should be aware of the risks of using the latest tag and use safer update mechanisms. Any significant changes should be tested and you should always have a rollback plan in place in the event that an update goes wrong.

8. Consolidate Logging and Monitoring for Applications

Debugging a container can be difficult, particularly when you have multiple processes going on inside it. One app may crash and come back up again, which makes things even more difficult to track down. You need to be monitoring continuously and have alerts in place so that when something goes wrong you’re aware of it and can go and look at it. The processes themselves need to provide sufficient information to be able to check memory and CPU usage. The ops team should be archiving the Docker container stats into their monitoring systems. The logs are absolutely vital when you’re trying to debug an app or service that has just shut down.

Docker supports a number of logging drivers and external monitoring tools can pull the data from the logs and from Docker itself. Log rotation and log retention are obviously important if you’re going to be running low on storage on the server.

Keep logs for as long as your recovery plan requires. The engineers can then analyze the log data to derive useful metrics and insights. If your app is logging excessively, then you should trim down the amount of detail. More data can be useful for debugging but too much can also expose sensitive information such as passwords so you’ll want to strip out the excess once you’ve finished debugging.

9. Manage Docker Host Networking to Avoid Misconfiguration

A production app will make use of a whole host of services (databases, caches, front ends, APIs etc). Some of these services will be public, some will be private. One service may be public, another may be entirely internal. Containers on the same host can make use of a private network which is only visible to those services.

Docker Compose allows you to define services, networks, volumes, etc., which is very useful in the development environment. In production, you’ll typically need something more sophisticated to route the requests and do load balancing across your containers.

This approach means you’re exposing far fewer ports on the Docker host. You wouldn’t open up a database port to the public unless you absolutely had to in order to allow another service to access it. For large-scale applications, you’d probably want to look at Docker Swarm or some other form of orchestration.

Some organizations employ a web hosting control panel for the deployment and management of their website and server applications. This strategy helps to keep website management separated from server and container management, streamlining the operations process for an organization. Orchestration tools specify relationships between containers, applications, databases, and other resources. This allows a web host control panel to deal with web server maintenance while Docker manages the application and database containers.

When your apps and services are all on the same Docker host and network, then connect to them by name rather than IP. IPs are going to change when the containers restart, so names are the thing to use. Don’t give network addresses open access, particularly not to the critical bits of your app. Use network segmentation and authentication to mitigate the risk.

10. Automate Reproducible Builds and Recovery Processes

If you’re a very small organization with just a few projects, you can quite happily run your containers manually. As you start to manage more containers, though, it becomes increasingly difficult. You need to store your deployment files in version control so that your deployments and maintenance are repeatable at scale. You can use Docker Compose to build containers on a single host. For larger applications that are running across multiple servers, you’ll want to use Docker Swarm, Kubernetes, or some other form of orchestration.

Developers should have their Docker configs under version control and use the appropriate infrastructure and automation tools to establish a robust process. You should have a deployment pipeline in place so that when a change is pushed, it gets tested on staging. The ops team needs to be prepared to restore the system if a change goes wrong. That means having a rollback process in place to either roll back to a previous version of the app or to a previous version of the database as appropriate.

Recovering an app after an outage can be a nightmare so it’s worth automating the recovery process (or at least documenting it) so the ops team can get back to normal as quickly as possible. Have a rollback plan in place with versioned Docker configs so you can roll back to a known good state very quickly if an update goes wrong. Automation will help with the sheer volume of stuff you’ll have to do as the app grows. It may not save you time in all cases but it will make the process more consistent and faster.

Conclusion

As your app grows and you start to run more containers across more servers the security and performance issues that Docker can bring become more and more apparent. The best practice is to use small images, run as non-root users, set memory and CPU limits, have health checks, use volumes appropriately. You need to secure your secrets, scan your images for vulnerabilities and monitor performance. Your networking and your orchestration tools need to be set up correctly and you need to have automated deployments in place so you can recover quickly if something does go wrong. All of this will make it far easier to maintain, manage and update your app as it grows. Not every app or every organization will need a fully automated multi-server setup. You need to pick the right tools and the right practices for your app and your infrastructure. Get these best practices in place early on as it will make the whole process of managing your app a lot easier.