One lesson I have learned over the years is not to take any unnecessary risks during production deployments. I hope that by sharing this wisdom I can help others, but I have the sad feeling that this one of those lessons that people need to learn the hard way. Unfortunately it is too tempting to “just go for it” sometimes. If it works, consider yourself lucky. When it fails (and sooner or later it will) you will regret your cowboy ways. I hope you will read on, there are some great ways to mitigate the risk of deployments.
Most of the problems that occur during deployments are caused by simple human error. Nothing serious, just plain old forgetfulness! In my experience, deployment issues usually arise because someone just forgets to do something. The thing that gets forgotten could be anything from a database script that gets missed, a forgotten view, an overlooked setting, or a .config file omission. And I don’t mean that this happens because someone is a bad developer. It’s just human nature. We are not perfect and can’t possibly remember everything.
There is no foolproof solution for perfect deployments. However, by following these tips you can greatly increase your likelihood of success. Of course, any of these practices will help but for best results, combine them all.
Deploy as often as possible
The best way to make sure nothing gets missed is to deploy often. Time is your enemy, the longer you wait, the more likely you are to forget something. During the development process we do so many things to make our applications work properly in our development environment. Sometimes we use a lot of trial and error too. As time goes on we can’t possibly remember all of the steps we have taken along the way. Following an agile process with short sprints is a great way to achieve short deployment cycles. Agile or not, try to deploy when features are complete and at major milestones. You don’t need to actually deploy to production. Use the same deployment process for all your environments. If you can’t deploy to production (there are lots of reasons for this), create a staging environment and treat it like production.
Even if you deploy often, don’t rely simply on your memory. Take a lot of notes along the way during development and during your deployments to other environments such as test, staging, etc. Develop a good system so that your notes are there when you need them, not buried and forgotten in a notebook somewhere. I like OneNote for stuff like this.
Have a solid plan
Don’t wing it. Have a good, standard plan and stick to it. The more you follow the plan, the more likely you are to succeed.
Use a good code base
Of course, you only deploy code that you know works correctly, right? Once you know it works, don’t leave anything to chance. One of the best ways to do this is to deploy the same code to all your environments. Avoid going back to your source repository and recompiling. So many things can go wrong once you do. After you deploy to your test environment and everything has been verified, you should use the exact same files (.dlls, script files, whatever) and deploy them to prod (or your other environments). Of course you will be probably need to make configuration changes, that is often unavoidable.
Watch your timing
Find a time that works good for you and your team. Different businesses will have varying constraints on when they can do production deployments so you will of course need to work within those parameters. But I definitely have my preferences for when I would choose to deploy. I like to deploy first thing in the morning when I am fresh and thinking clearly. If something goes wrong, I have a lot of time to fix it. Also, I’ll have access to other people/resources for help if needed. If I deploy at the end of the day and something goes wrong I’ll be stuck working at night to fix it. That is bad because a) I’ll be getting tired and might not think clearly, b) I won’t have people around to help and c) I won’t get to have dinner with my kids. I also like to deploy midweek. Monday mornings are bad because over the weekend I have forgotten things already! Monday is a great day to put final preparations on a big deployment. Fridays are bad too. Because if things go wrong, I could be stuck trying to fix things over the weekend!
Automate your process
I don’t trust myself or my memory much. The best way I can protect myself from me is to automate my deployments. Once things are scripted, automated and tested, I have greatly increased my chances for success. Most of the other tips I have listed are fairly easy to implement, but putting together an automated deployment can be a big project all in itself! Depending upon your project, automated deployments can be quite complicated and there is a lot to learn in this department. There are lots of great build and automation tools out there to help such as MSBuild, NAnt, TeamCity, Hudson, CruiseControl, etc. I’ve used a bunch of different tools over the years and now I am using Team Foundation Server 2012. I’ve found it to be the easiest and fastest of all to set up for my web project deployments.
Practice your deployments
Practice makes perfect, right? Unfortunately with deployments, just when you get your process working smoothly there will be some big change that will require you to modify the process anyway. But practice will definitely increase your chance for success. I even practice my automated deployments, that is the reason to have a staging/faux-prod environment. And practicing really comes in handy when it comes to database changes. It’s great to have a staging environment set up just for the purpose of practicing deployments. To practice, get your staging environment set up to match your production environment including .dlls, html and script files, databases and other resources. Then run through your deployment process on staging. Once complete, test your application to make sure everything works. If something went wrong in the deployment, don’t patch it. The best thing to do is roll back the entire staging system, tweak your process and run it again. You do this as many times as needed until you get everything right. Then you are ready to repeat your steps in your production environment. Of course, the more you automate, the less likely errors will be. But don’t assume that just because you have an automated system you don’t need to practice too.
Lastly, do not wait to implement this process. Many developers wait, thinking that they’ve got a long development cycle and won’t be deploying to prod for a long time, maybe months down the road. When I start a new project I begin working on the build/deployment automation right away. I also make sure there are tasks created for this process. Making this an official deliverable buys me time for this work which can definitely be time consuming. And if you don’t get this work done early, you might not get to it at all. When a project is almost complete and the delivery date approaches, you will never find time to get your process in place. You will then be stuck with an error prone deployment that not only keeps you worried but will likely cause you to work on weekends fixing deployment issues.
A solid deployment process takes a lot of planning and work. Of course, if you have ever been on the wrong side of a bad deployment you know that all the planning is worth the extra effort! Good luck.