What Does Your DR Look Like? Or "Holy #$*%! Everything is down!"
This is a topic near and dear to me these days. Having suffered a recent outage at my job with over 9 hours of downtime, this is now a major issue for me to work through. Everyone always gives disaster recovery (DR) lip service. They come up with ways to backup data, provide alternative networking access as they can afford, and try to create plans. My feeling, much like what I have dealt with at prior positions, is that no one really invests into DR. I hope to provide a few cautionary tales to help you convince your management to make the investment.
Disaster recovery is insurance. All the investment that is made in DR is insurance against a downtime. At the same time, everyone keeps saying "it will never happen to me." I can provide references that it does happen and the outcomes can be brutal for a business. Downtime can lead to loss of business opportunity, change in customer perception reducing their business with you, loss of customers entirely, or complete collapse and closure of the business. To offset these outcomes, businesses invest in disaster recovery to mitigate the impact of downtimes.
When looking at DR, first thing that people need to determine is what are those critical systems; what systems do you have that if you lost them would impact the business most. For a manufacturing company, it could be their control systems for their machinery. For a datacenter, it could be the power and networking systems to keep the hosted systems online. For a healthcare company, it could be all the systems involved with patient care. The IT team needs to sit with the business and management teams to determine which systems are those critical systems and all of the infrastructure that supports it.
Now that the critical systems are identified and their infrastructure is determined, a full risk assessment of those systems and infrastructure needs to be completed. Are there devices that have single points of failure? Can servers be connected to the network in diverse paths, also known as teaming? Can the software be setup in clustering technologies to allow more than one server to be setup and kept in sync? What equipment is the oldest and have a higher possibility of failure? Working through the risk assessment with knowledgeable team members in both the IT and business teams will help find the answers quickly.
Now that the risks are identified, the professionals need to step in and make some plans to mitigate those risks. That planning can include duplicate systems, cluster creation, backup and recovery techniques, additional networking equipment and lines, and warm/cold spare hardware to name a few. Each of these plans need to be fully thought out including the costs of creation and ongoing maintenance.
Part of the maintenance of backup systems is using them, a largely overlooked step of DR planning. Both business and IT teams need to role-play disasters to ensure these policies, procedures, and systems will work. These sorts of tests interrupt normal business operations but should be done on a regular basis to ensure all systems are go for a real disaster. After each test, the affected teams should get together and review the test event to improve policies, procedures, or systems in the future.
I know that what I have said so far is something that everyone else has said to their management to push for better DR planning and testing. I have said it myself at times. Having gone through a large outage that affected my company's business has brought it to the forefront for me and gotten the attention of my company, a company that runs 24x7 for our business. We lost our primary datacenter, the hosting location for primary servers and the hub of our network, for approximately 9 hours on a Thursday night, which is our busiest times of the week. While we had some basic processes and procedures in place, it was thanks to the hard working teams at my company that we made it through the outage.
During the outage, the primary datacenter lost its primary power at the Automatic Transfer Switch (ATS) that allowed them to select either the utility company or their generators as the power source. Not only did they lose the power there, the ATS literally blew up blowing out part of the wall behind it. In trying to get the datacenter power back online, they also found that a fuse in the transformer was bad, possibly causing the whole problem. To correct the transformer fuse, they would have to fail their second power source from the utility to generator to allow the utility to pull a fuse from that second transformer as the utility crew did not have a spare on hand instead of waiting up to 2.5 hours for them to go get one at their warehouse and returning.
While seeming a simple fix, this would have impacted part of the datacenter that was still operational and hosting one of their biggest customers. That customer did not want any more change introduced into their hosting systems. As a customer impacted by the continued outage, I pushed on the datacenter to start the change with haste. This put the datacenter in the middle between customers.
Eventually, this was resolved and the generator added to the second circuit, allowing the utility to repair the primary circuit. This is where good process and planning helped out my team because we knew which systems had to be started first and what order to effectively restart our business. Once we got our systems up, the business teams started in cleaning up their issues from the outage.
After the outage, an emphasis was placed on all parts of my company to determine ways to improve our business resilience to outages. This includes alternative network connectivity for outages, secondary datacenters, hardened systems, and improved policies and procedures to reduce the impact on our customers if we have another outage.
I will admit that I wrote this blog entry a while ago but could not finish it off until now. It was difficult to read what I wrote because it would make me go back and remember all that happened; reading my blog entry brought back all of those memories and feelings as if they were happening again. Major service interruptions are difficult for any group. What made this worse for me was that there was nothing I could do but wait for our hosting provider to fix their facility and services. Since this occurred, they also have taken some steps to improve their offering to ensure clients like my company do not suffer through something like this again. Improvement can happen for you directly or for your providers and partners.
The key takeaway is that outages will occur. The better your systems and networks are designed and the more time is invested in both business and IT policies and procedures, downtime impact can be reduced and customers can be kept happy during those outages. The best outcome that IT and business teams can hope for is no impacts for their customers at all while systems are offline or unavailable. No single system can stay 100% available forever but well-designed systems and networks can offer the "Five 9's of availability" (99.999%) or no more than just over 5 minutes per year of downtime.
What are you doing for your disaster recovery? Is it even a thought for you or your company?
Moved my Site and Blog to Azure … How Easy!
Well, I finally made the switch. As many of you can see in the URL, my blog has moved from my personal servers onto the Azure fabric. It is something I wanted to do for a while and never got quite around to finishing until now. It is not totally done but I am happy with the interim results.
For those that don't know, Azure can offer easy web hosting up in their cloud with CMS systems like WordPress and DotNetNuke. I personally do use DotNetNuke and have for several years. Installation was looking to be interesting thanks to a few projects around like the DotNetNuke Azure Accelerator. Other blog entries and wikis are out there talking about how to get this accomplished.
A few weeks ago, I tried to use these "recipes" and failed miserably. In the same evening, I also screwed up local installs of some test servers and thought if I could just strike out at a bar, the evening would be complete. The process seemed to be fraught with missing settings, steps that did not work as advertised and some complications, later found out to be caused by Azure issues.
The next day, I sat back down and looked at the Azure offering and within the "creation" workflow for a website is a "From Gallery" option. For grins, I clicked on it and the world got so easy for me! Within this option, Azure offers a multitude of predefined systems for installation from their Azure Store. This includes many CMS systems like DotNetNuke, Drupal, Joomla!, WordPress, MediaWiki, Orchard and many more. It also includes E-commerce engines, forum systems, galleries, and wikis. Right at the top of the list is DotNetNuke Community Edition, my choice for CMS. They make the Professional version available as well but that is a paid product and I know what I am doing with the CMS engine.I started down their wizard path to creating my new Azure website using the Gallery image of DotNetNuke Community edition 7.0.3. Clicking the "next" arrow brought me to their initial configuration screen where I put in my Azure URL, told it to create me a new DB for the project, and allowed me to choose the region for hosting, West US in my case. One more screen for the DB setup on a new server, the DB username and password, and which region for the DB hosting, West US again for me, and we are off to the races. The next steps are very DNN specific so I will not bore the majority of my readers with those details.
Once all was setup, I could browse to the Azure Base Site URL and look at my new DNN installation. Within 10 minutes, I had my beloved DotNetNuke 7.0.3 running in the Azure cloud without any major work on my part. I was able to install my favorite blogging module, Live Blog from Mandeeps, and thanks to my SQL knowledge, port over this blog from my personal server to the Azure site. A quick set of redirections and here you are with my new version of my blog. Now, I just need to get more content up here…
Have you started using Azure for hosting of your sites? If not, why not give it a try with a 90-day trial? Sign up at http://www.windowsazure.com/en-us/pricing/free-trial/
Tips to Cut the Cord With
"Cutting the Cord" is a catch phrase that is thrown around in this modern age. The main meaning is ability for many people to remove services that they used to pay for that seems redundant in these changing days, primary being television and telephone services. For me, I am fully cord cut when it comes to telephone and nearly cut with television. I will explain what I have done, how I came about my decisions and what it took to execute my cord cutting. In the end, there is no way to cut all cords unless you want to disconnect from the world and entertainment. Instead, the goal for most cord cutters is to run all of their needs across their data service lines. What you need to do is find your goals for cord cutting and then find what will help you achieve those goals.
The easiest service for me to cut was the telephone. I live on my cell phone; I know many others that do as well. When I used to have a phone line, I was paying nearly $50 for something I rarely used. My concerns for dropping landline service keyed around people getting a hold of me and emergency services. Since everyone had my cell phone that I wanted to contact me, that first concern was null. As for emergency services, the concern is proper location services for emergency crews to arrive at. Cell phones are now required to have E911 (Enhanced 911) location services but this is not a guarantee. Instead, I use a little known fact.
My condo already was wired for phones and that service is attached to the local phone carrier. I can plug a phone into that line and call 911 without costing me anything. This is perfect for emergencies and the 911 operator will have the location information for the line that was established by the phone company. For everything else, I use my cell phone.
My solution for phone service was easy for me, but it won't be easy for everyone. There are some good alternatives out there including Vonage and Ring Central that provide VoIP solutions over your broadband data connection to Skype and Google Voice that provide some call management and VoIP features as well. Think through what you need for yourself and your family. Then, find the service that provides what you are looking for.
Now that I have started cutting the cords, I reviewed my television entertainment needs. This will vary from person to person, not just as a whole family unit. Those needs can change over the years which will mean that flexibility is key. Let's use myself as a test case now. In 2004, I used to be a background TV person where I left the TV on all the time not really noticing what was on. Over the years, I have changed my consumption habit to enjoy specific programs. These changes were both caused and caused by my cord cutting choice.
I find that most of the shows I like to watch are available on the main networks, or specific shows on cable channels. Since I cannot purchase an a la carte cable package and do not want to pay up to $75/mo. for the small number of channels I want, I worked through the shows legally online. I start with channels that my local cable company offers via their "Basic" package (Comcast offers at $15/mo). This includes the local network stations (ABC, NBC, CBS, PBS, Fox, WB, CBC, Ion) in HD, Discovery Channel in SD, and several local off-band stations. It also includes stations that I do not care for (religious, 24 shopping, government access, non-english) thus making it a normal cable subscription. Depending on your state regulations, not every cable provider offers this basic subscription. You should be able to get these channels with a TV tuner (in a TV or in a computer) that can get digital cable in the clear (Clear QAM). However, some companies still require one of their set-top boxes to get even this "basic" package. I will be cutting this service if that occurs with Comcast.
Add to the basic cable subscription, I utilize several online services to watch episodes to fill in on channels I do not have. With my desired list of shows, most of them are available online via services like Netflix ($8/mo.) and Hulu/Hulu Plus (Free on web/$8/mo.). When I add up all the entertainment I get that way, about 90% of the TV shows I want to watch are available. To fill that last gap, I utilize Xbox Video (formerly Zune Video) and Amazon Instant Streaming. One or both of these services has the rest of the shows I want to see available to purchase with pricing based on the length of the season and the quality. There are other sources like iTunes but I choose not to use them as my devices are not well matched for it. With smaller cable subscriptions and online sources, you can find most of your content that you want to watch without paying the high rates of cable. Want even better news? You now have more options to watch entertainment thanks to the internet.
Remember that one of the changes I have undergone is having the TV on in the background to now watching specific shows and paying attention to them? This change alone reduced the amount that I was watching and helped to filter what I watched to a very specific set of shows. It also added a brand new source of content that most forget is available, the internet. Content creators have started to understand they do not need to work through "traditional" media publishing channels. They can create a website and an RSS feed to launch a "video netcast or podcast". Some large media people have jumped over to this new medium such as Leo Laporte with his TWiT network, Adam Carolla with his Adam Carolla Entertainment network, and former MTV VJ Adam Curry with his Mevio network. Others have been on the internet from inception like Audible for audiobooks and special interest sites such as Technet on Microsoft for Microsoft IT professionals. In my experiences, I find this content better than the content coming from the networks and cable, making me miss the deluge of cable channels filled with programs I never watched.
Now that you have done some homework on what you want for content, you need to think about how you are going to consume that content. Since I started with the notion of replacing television, I will focus on the use of a television in either a living or bed room. The easiest devices are some sort of set-top box that has the content available through apps. I have a Roku device in my bedroom and installed a Windows Media Center PC in my living room. I am not the norm though here in that I built and managed a computer that was a DVR/set-top box. It was the most flexible and offered all the content but not all in a 10-foot UI. Some of the content I had to use a web browser with a wireless mouse and keyboard to access. I was willing to go through that while others are not.
To make things very simple for the average user, you should really look at the Roku devices to plug into online services with their applications. They offer applications that connect to content services like Netflix, Amazon, Crackle and Hulu Plus. In addition to the major content services, Roku devices can connect to many new media companies with apps like TWiT, Revision3, The Onion News, and CNET. This gives the Roku devices a big advantage in the fight for a single box to add these services. Even some traditional television channels have applications of their own on Roku devices like CNBC, Sail TV, Fox News and NBC News that offer live feeds from their cable channels online. HBO even has their HBO-to-go service, targeted at mobile devices like tablets and phones, available on Roku devices. For this to work, you need to have a cable subscription with HBO added to it to access it. The Roku devices could be easy, cheaper set-top boxes for additional TV's in a house to reduce the need for cable/satellite set-top boxes.
Recently, I changed my living room to use a Xbox 360 as the primary device. With apps for Hulu Plus, Netflix, Amazon, YouTube and other online services, connectivity to my Media Center for recorded and live content, native support for Xbox Video, and it's DVD drive, it is a single device that I can use. As my media collection does not include Blu-rays, the HD content I get is from online sources so I am not hurt by the missing Blu-ray drive of the Xbox 360. If I paid for a full cable subscription, I could use my Xbox 360 as a set-top box for their services on Xfinity and FiOS. Through the Xbox devices, I see Microsoft trying to make a play for the living room via easy to use devices and I get that now. Rumors are with the next release of Xboxes coming in 2013, we might see a specialized media only device along with a new gaming unit.
As you have seen, I have done a lot of research on what is best for me given my consumption of entertainment. What works for me may not work for everyone. One key demographic I can see is families with children. The story here is improving with a "children's focused" Netflix integration and view along with specific apps on Roku for kid's programming. Parents need to research what is best as there is so much content available on the internet. While it can be overwhelming, it is the same thing I would expect most parents to do with other forms of entertainment. Most of the better systems to allow for parental controls to manage what kids watch but it does not beat being there and watching with them to know what they watch.
Cord cutting is possible today, even though we are in the early days of it. You see large media companies trying to slow or stop it as much as they can to keep their current revenue models flowing the dollars to them. Accepting that some content will not be available for a long time or ever is one thing a "cord cutter" using legal sources has to accept. Just one example of this for me is Game of Thrones from HBO. Without an HBO subscription, requiring a much higher cable package than I had nor wanted, I accepted that I would not get it until it was released on Xbox Video or Amazon, nearly 1-2 years later. Spend the time figuring out what you can live with and without, where can you source it, and what device can show it on your preferred screen.
The last thing I will mention is most of this requires a broadband style network connection and can push usage caps if they are attached . Since I am an IT Professional doing a lot of work online and knew I would be using media services, I purchased business class internet that provides to me 25 Mb download guaranteed with no caps. This is not cheap internet at $110/mo. and if added to everything else, might push someone back to regular cable service. I use the bandwidth for more than entertainment so I feel the monthly costs for that inexpensive and would rather put my money for that over a television subscription. Add it all up for yourself and figure out what works best for you.
What sort of cord cutting have you done? What are your goals with cord cutting? Let us all know through the comments below and help others get out the wire cutters.
The Good, The Bad, The Ugly … The Wonderful World of Compliance in IT
"Compliance" is often perceived as such a dirty word to IT professionals that it might as well be censored. The mere mention of "compliance" brings about visions of additional paperwork and processes that slow down everyday tasks and project schedules for many IT pros. With newer regulations, be them federal laws such as Sarbanes-Oxley Act of 2002 (SOX or SOX404)[1] or Health Information Portability and Accountability Act of 1996 (HIPAA) [2]; association or vendor regulations like Payment Card Industry Data Security Standard (PCI DSS) [3]; or internal standards created by management, IT teams in both engineering and operations have to work to meet these regulations and standards as a part of their project and daily work. This was a great discussion topic for Denny Cherry and me on his People Talking Tech Podcast. [4]
Want to make an IT team squirm? Create a meeting about "compliance" or introduce an auditor or consultant. Let’s be honest; we’re technical people, and we want to make things work as quickly and efficiently as possible. It’s uncomfortable to have someone looking over your shoulder to verify that you are “doing things right," either through the operations team installing and configuring, or developers writing code that is deployed to users for saving data into the server or the cloud. But get this: it is actually in our best interest to see it from a different angle. Compliance, standards and regulations are the IT professional's friend. Much like all other aspects of IT, with proper planning and execution, complying with standards and regulations ensures that you have "air cover" for everything you do.
Proper planning for compliance is just like any other IT project: the earlier it can be integrated in plans, the easier the execution can occur. This is true with engineered projects by developers or with integrated projects by operation teams. Compliance can range from smaller tasks such as documenting what is built or installed, all the way to deep logging and intricate permissions management systems. In most situations, there is no "silver bullet" solution to comply with regulations or standards, and any vendor offering this to you or your company should be reviewed carefully. These sorts of vendors typically try to engage with non-technical management to sell them on solutions that the IT team has to later "figure out how to integrate them." (If you have good horror stories around this exact situations, feel free to share in the comments below.) To get ahead of those "snake oil salesmen", be ready to show your management how you are currently or will meet your standards and regulations.
Identify the Requirements
First step in creating a good compliance plan is to understand what you need to comply with. This can be fairly straightforward for some, such HIPAA for healthcare, while being much more complicated for others, combining SOX with PCI DSS or US state and federal regulations with other countries. This step is very critical and will require IT professionals to reach out to the business users they support and possibly consultants like lawyers or compliance officers. It may sound simple, but this can be the most difficult step as many regulations are not black-and-white. Each person can read the same words and interpret it differently. Documenting this interpretation as it is being reviewed will only help you in the future if you have to defend that interpretation from regulators or re-interpretation with new staff or consultants.
Create and Socialize the Plan
Second step is creating internal processes, procedures, and standards that meet all of the items you found in the discovery. Many companies have unwritten ways to do things, rules that everyone follows without question, and systems they use to track what is done and how they do it. While some companies do a great job in documenting their processes, procedures and standards, most do not , and getting teams to change this practice can mean a cultural shift.
Where you often see this issue is when smaller companies grow larger. Small companies consider their IT teams agile because they are all generalists, and any member can and does fill all possible roles. As companies and their IT teams grow, specializations occur. The company's business users yearn for those old days when they could call one of the IT team members to get help and the IT team just got it done. The fight to follow processes and procedures while "getting things done" is a constant struggle. Selling the benefits of following processes and procedures internally is one of the toughest things for IT management to do in situations like this.
Document and Organize
The third step is to take all of the documentation and organize it so that it can be reviewed, executed and tracked. This includes processes, procedures and standards, along with how the team is executing against those standards. In many cases, regulators can show up without notice and audit the company's compliance. Without having the information in an easy to access system or storage, it is useless to both the IT team and the auditors. Again, there is no "sliver bullet" solution for this. Every team needs to find their own solution that works for them. For some teams, it could simply be a file share with Word and Excel documents; for others, it might take a self-developed or commercially available software package.
All of this seems easy on paper, but there is no quick solution or answers. IT teams, management and business users need to take their time to understand what needs to be done, and when they need to meet regulations and standards. Sometimes, one group may ask for more than what is required in the regulations. This needs to be tempered with timelines and costs. Lastly, remember that it takes time for people to adopt changes, and anticipate that when creating the project execution plan. By working together with realistic timelines and good communication, a proper compliance plan can be executed.
Plan for Continuous Improvement
Once the compliance plan is pulled together and the users and IT team are following the plan, the best thing to show most regulators and auditors is a continual improvement processes. Regulators, auditors and compliance officers love to see improvement. In some ways, it is better to create your initial plan and then slowly improve that plan over time rather than trying to create the "perfect plan." Improvement around compliance is seen as a good sign of compliance in an organization. This makes the improvement process just as important as the initial compliance.
IT compliance does not need to be a dirty word. Everyone has his or her stories around the good, the bad and the ugly of compliance; the stories of well executed plans, stories of bad or no plans, stories of regulators imposing large fines and sanctions. Take your time to prepare and execute the best compliance plan you can with the resources available. Once that plan is in place, create an environment that makes ongoing improvements as painless as possible, so that compliance is something everyone understands and wants, and not seen as an impediment to their work.
How do you feel about IT compliance? Do you have stories to share, whether good or bad? If so, put them into the comments below.
This was cross-posted by Veronica Wei Sopher on the Born to Learn Blog at MS Learning. You can check this one out specificially at http://borntolearn.mslearn.net/btl/b/weblog/archive/2013/01/28/the-wonderful-world-of-compliance-in-it.aspx and other great posts over at http://borntolearn.mslearn.net/. Special thanks to Veronica for helping with this posting.
Notes:
1 - More information about Sarbanes-Oxley Act of 2002 (SOX or SOX404)
2 - More information about Health Information Portability and Accountability Act of 1996
- http://www.hhs.gov/ocr/privacy/hipaa/understanding/index.html
- http://www.cms.gov/Regulations-and-Guidance/HIPAA-Administrative-Simplification/HIPAAGenInfo/index.html
3 - More information about Payment Card Industry Data Security Standard
4 - Direct link to my appearance on People Talking Tech, January 22nd, 2013
Where the Heck Have You Been??
I wanted to post something on my blog as I haven't written on here for a while. The holiday season always creates havoc for people in the IT world as this is a time when a lot of work can be done because people are gone due to the holidays. On top of that, my day job has a lot of projects in flight due to regulations and compliance.
One fun thing I got to do was record with Mr. Denny on his podcast, People Talking Tech, for an upcoming show. We spoke about compliance with my former background with Sarbanes-Oxley Act of 2002 and SEC compliance at my prior position in Microsoft and HIPAA and HI-TECH compliance at my current position. I will tweet and post out when it get's posted but check out the other great episodes on the People Talking Tech website.
I haven't forgot about this blog as I have a few topics I have been working on such as:
- Cutting the Cable Cord
- Expansion of My Media Market after Cutting the Cord
- Gadget Reviews:
- LG HBS-700 Stereo Bluetooth Headset
- 11,000 mAh USB External Battery Packs
- Surface RT … 3 Months Later
- Nokia Lumia 920
- Windows 8/RT Apps of Note
- Windows Phone Apps of Note
- Am I going to get a Surface Pro? (Answer will surprise you …)
- My Home Network/Lab Experiment
- My Experiment Moving My DNN Installation to Azure
- Moving from WHS/Drobo to WHS/Storage Spaces
- IT Compliance Thoughts
If you have some topics you would like me to cover, leave a comment below.