Downtime, outages and interruptions: understand your real costs
- April 11, 2019
- Written by: gad cohen
One
This content is brought to you by Evolven. Evolven Change Analytics is a unique AIOps solution that tracks and analyzes all actual changes made to the enterprise cloud environment. Evolven helps leading companies reduce the number of incidents, improve problem resolution time, and eliminate unauthorized changes.Learn more
When it comes to mission-critical applications or data center performance quality, companies are willing to invest heavily. Unfortunately, these investments are not always fully delivered.
deal with system downtime
Despite the efforts that have been invested in infrastructure resiliency, many IT organizations continue to deal with database, hardware, and software downtimes that last from a few minutes to several days, crippling the business completely and causes huge losses.
expected downtime
The world of IT outages can seem strange at times.
Despite the variety of advanced solutions and the growing amount of data collected by major enterprise software vendors and IT departments—from ERP to CRM and more—outages remain a valid and serious threat to the industry.
On the other hand, IT outages have somehow become an inherently accepted, even expected, part of business life.
That's counterintuitive...
Revised IT downtime
While IT professionals experience downtime from time to time and then focus their efforts to overcome it, the business organization as a whole suffers "financial pain" with an impact that is often very significant.
In the past, we've delved into the various ways IT downtime can impact back-end operations (you can read more about it here:Cost and extent of unplanned outages). In doing so, we consider different aspects, from direct sales losses and reputational damage to indirect effects such as reduced productivity.
Now, I'd like to revisit the topic and examine how organizations should address and assess threats to their IT operations, including systems, applications, and data, by looking at solid (and established) benchmarks that measure costs. potential downtime and disruption they represent. .
System Outages:
Measuring the failures of big brands
System Outages:When should the industry start measuring the financial impact of major brand failures like the recent one?Facebook, oone that reached hundreds of thousands of Lloyds Bank customers, or theJetstar OutageWhat caused hundreds of flight delays?
In other words, at what point is an outage "significant enough" that a cost analysis becomes valuable for the industry to learn and predict the impact of future outage incidents?
Well, at some point failure will have publicity effects that cannot be ignored. This is the point of no return followed by estimates of the financial impact.
Downtime costs vary significantly by industry. The size of the company affected is, of course, a critical factor, but not the only important one. The role of IT systems in the business is also crucial.
Defining a numerical value behind an IT outage means pre-defining its impact on various organizational and business aspects so that the entire industry can learn and optimize accordingly.
A failure of a critical application can lead to two different types of losses:
- Application service interruption: The impact of downtime varies by application and organization;
- Data Loss – The potential loss of data due to system failure can have significant legal and financial implications.
Well, I'm sure you'll agree that today's data centers should never go down; Applications must remain available 24/7, and internal (not to mention external) end users around the world must be able to rely on data centers to be available at all times ( for the availability of critical data and applications).
Well, reality bites. This is not the case in the back office (ie, within the data center). No organization enjoys 100% uptime. Should you reach 100%? Clear. But you also need to develop a deep understanding of the impact of downtime and ways to minimize it.
The worst nightmare of failure of all time? What probably happened to you...
Some previous outages have turned into public relations disasters, like the mythical Virgin Blue disaster in 2010 or the recent one that hit Facebook.
Because? The massive impact probably had something to do with it.
As a reminder, the Virgin Blue outage prevented passengers from boarding flights for 11 days (!!), resulting in negative press, damaged reputations and millions in losses.
More specifically, Navitaire, Virgin Blue's reserve management company, ultimately compensated Virgin Blue for more than $20 million (Navitaire's booking error gives Virgin $20 million in compo).
There are many other incidents that still attract media attention. Here's a recent oneUSA Today article on the Wells Fargo blackoutwho prevented customers from accessing their accounts for many hours.
It's safe to say that anyone in IT would agree that outages or interruptions are VERY bad for business. They are undesirable, very economically damaging and must be fought with all available means.
Configuration errors are critical
The IT Process Institute's Visible Operations Handbook has historically reported that "80% of unplanned outages are due to poorly planned changes made by administrators ("operations") or developers" (visible operations).
The Enterprise Management Association reported that 60% of availability and performance failures are due to incorrect configurations.
How much does it cost?
Downtime can cost organizations $5,600 per minute and up to $300,000 per hour in web application downtime (for a2014 Gartner Analysis).
Average cost per hour of downtime for enterprise servers, worldwide, 2017-2018:
Those:Politically
Application maintenance costs are increasing by 20% per year. But that can't solve all your problems. Previous industry studies have found that at least a quarter of surveyed downtime is due to configuration errors. (How much will you spend on app downtime this year?).
How common are downtime or breakdowns?
Granted, downtime can be a financial nightmare. That part is clear. But if you want to properly assess the potential risk of business interruption, the immediate question should be, "How likely is that?"
Those:data center knowledge
Granted, interruptions are too common to think "I probably won't have a big interruption." Now the question arises how to calculate your specific risk for your company.
Clarified production costs and app downtime
Unplanned outages are resolved by IT. However, as I mentioned earlier, these outages eventually affect the entire organization.
An important part of a complete downtime risk assessment process is estimating how much money you will lose per hour (or minute, or whatever time interval you choose) due to downtime.
For businesses that rely solely on the ability of data centers to provide IT and network services to customers, such as telecommunications providers or e-commerce companies, downtime can be particularly costly, with the highest cost for a single $1 million event (more than $11,000 per minute) according to expert estimates.
In a USA Today survey of 200 data center managers, more than 80% said their downtime costs exceeded $50,000 per hour. More than 25% reported downtime costs of more than $500,000 per hour (!!).
According to another survey, one in ten companies said their availability should be greater than 99.999%, even though companies cannot achieve zero downtime.
Those:Searchcio Techtarget
To get a solid understanding of the impact of production and launch downtime, let's take a look at how the consequences of downtime manifest.
Downtime Costs: Per Year or Per Incident?
AStudy 2017found that 46% of 400 IT decision makers experienced more than four hours of IT-related downtime in 12 months; 23% said they incur costs between $12,000 and more than $1 million per hour.
More than 35% admitted that they are unsure of the cost of a business interruption.
If you ask Delta Airlines, which had to cancel 280 flights due to disruption in 2017, the losses from a single disruption incidentcould reach more than 150 million dollars.
A few years ago, Dun & Bradstreet reported that 59% of Fortune 500 companies experience at least 1.6 hours of downtime per week.
If you take the average Fortune 500 company (or any company with at least 10,000 employees) and assume that it pays IT staff an average of $56 per hour, then (assuming all IT spends is to fix downtime) inactivity), only a part of the work. Downtime for a company this size would amount to $896,000 per week, which equates to over $46 million per year (Assessing the financial impact of downtime).
Of course, reality is more complicated, since many parameters must be taken into account, such as: B. the time of the event (weekdays or weekends? day or night?) and much more. However, understanding the cost of downtime will go a long way in assessing your risk exposure and the return on investment of tools that can help minimize the impact of downtime.
Could the industry learn from the past and minimize collateral damage during an outage?
How have things changed since the past?
So we already know that blackouts and blackouts still happen today and the industry still hasn't managed to get rid of them. But how have costs changed over time? Are these incidents less harmful today?
from 2010,a survey by Coleman Parkesfound that IT downtime costs companies a total of more than 127 million hours per year (an average of 545 hours per company) in employee productivity.
In 2009, the average cost of downtime was reported to vary significantly by industry, from about $90,000 per hour in the media industry to about $6.48 million per hour for large online brokers (How to quantify downtime).
According to a survey of IT managers conducted over the years, companies are increasingly aware of the direct financial cost of IT failures. Research has found that one in five companies are losing $12,000 per hour due to system downtime (How to quantify downtime).
As mentioned above, a subsequent analysis by Gartner in 2014 found average costs of $5,600 per minute and more than $300,000 per hour.
As far back as 2004, a conservative Gartner estimate put the cost of computer network downtime at $42,000 per hour. As a result, a company with less than an average of 175 hours of downtime per year can lose more than $7 million per year. However, the cost of each interruption affects each business differently, so it is important to know how to calculate the exact financial impact (How to quantify downtime).
It makes sense to think that the cost of interruption only increases over time (as we are more reliant on data systems today). Thus, you can understand why data from the past can be multiplied by a significant number to reflect current reality...
every minute counts
More than a decade ago, the average cost of data center failure across all industries was estimated to be approximately $5,600 per minute (Unplanned IT outages cost more than $5,000 per minute), appreciate that, secondgardener, stayed the same until 2014. The earlier study by the Ponemon Institute mentioned above calculated the minimum, average, average, and maximum cost per minute of unplanned outages, based on inputs from 41 data centers. The highest cost of an unplanned outage was over $11,000 per minute.
On average, the cost of an unplanned outage is likely to exceed $5,000 per minute.
It just becomes more meaningful.
AStudy 2013saw an increase of more than 41% over the previous averages described above and an average cost of more than $7,900 per minute.
As2015 ITIC-Umfrageclearly showed that the cost per hour (compared to 2008 data) increased by 25-30%.
Impact of downtime per year
A previous Gartner analysis estimated that downtime can average 87 hours per year. Obviously, this is the sum of many interruptions, from a few minutes to several hours (The average large business experiences 87 hours of network downtime per year).
How things have changed?
latersurvey 2011revealed that while the industry has successfully addressed the downtime epidemic and reduced its frequency, we are still seeing significant downtime and huge revenue losses (Source:resulted in more than 3 million (apparently Whatsapp users) switching to Telegram)
The impact on reputation and loyalty
How much is your company's reputation worth? This can be extremely difficult to assess, as can the long-term impact of a damaged reputation and its impact on sales and profitability.
In this case, the cost of downtime includes lost customers (both short-term and long-term) and other tangible elements that reflect the cost of reputation degradation, such as profiling.
What parameters should influence its calculation?
When trying to estimate the cost of downtime, there are obvious direct costs (such as lost business during downtime). However, there are also many indirect costs to consider, such as personnel costs or the reputation issues mentioned above.
Staff costs are derived from the cost of burning out "war room" tasks meant to get IT systems back up and running, the cost of delays in all other scheduled tasks, the cost of additional staff (if any), and further. Add to this the value of data loss, emergency maintenance fees (especially if the outage occurs outside of business hours), and additional repair costs that can linger long after service is restored.
It goes without saying that you should calculate these costs when assessing the impact of downtime, as they are often very significant; But even a rough estimate can be extremely helpful in understanding the risks and deciding what level of technology to rely on to combat them.
There is also the impact of lost sales. To get an accurate estimate of total lost sales, the hit percentage must be increased to reflect the true lifetime value of customers permanently switching to a competitor. For example, the aforementioned Facebook (and Whatsapp) outage.Unconscious costs: denying the true cost of network downtime. What is the revenue loss due to these users experiencing fewer billable ad impressions?
Shares fell 25%
Although it is difficult to quantify so many parameters, they are meaningful and meaningful. For example, when Amazon.com was offline for several hours in the first few days, its inventory dropped by 25% in a single day (Unconscious costs: denying the true cost of network downtime)!
darinAmazon cloud outageFor example, the company continued to fight to bring its cloud services back online. As a result, many customers questioned the reliability of their cloud and Amazon's communications around the outage. Other customers felt that they should be compensated for downtime as part of their SLA.
I know you're curious: SLA-wise, despite the nearly four-day outage, Amazon's EC2 SLA was not breached (Seven lessons from Amazon's failure).
Downtime costs: calculate yourself
How much will you lose due to unexpected server or business application downtime?
According to various sources, the easiest way to calculate the potential loss of revenue during an outage is to use this equation:
| LOSS OF SALES | = | (GR/TH) x I x H |
| GRAMS | = | annual gross income |
| º | = | total annual working time |
| UE | = | percentage impact |
| H | = | Number of hours of interruption |
How can you minimize the risk of interruptions and downtime?
Downtime and failures are catastrophic, but they don't have to be that serious. By using solutions that focus on getting to the root of the problem, failures can be prevented even before they happen.
Change Analysis EvolvedHe developed a unique AIOps solution that focuses on change - the root cause of performance incidents. Evolven helps enterprise IT and cloud operations teams prevent and remediate incidents before problems arise.
Contact Usto see how we are helping leading companies reduce incidents and MTTR.
FAQs
How much does downtime really cost? ›
The average cost of downtime across all industries has historically been about $5,600 per minute, but recent studies have shown this cost has grown to about $9,000 per minute.
What is downtime What are the costs associated with downtime? ›Downtime cost is defined as any profit that a company loses when its equipment or network stops functioning. The cost of downtime implies not only direct financial loss but can have an impact on your company in at least the other 4 ways.
What is the difference between downtime and outage? ›Downtime occurs when a system can't complete its primary function. It can be broken up into two types: IT outages and brownouts. IT brownouts occur when a system is slowed or partially available. This might mean customers can access your site, but pages load slowly or dynamic features like "add to cart" don't function.
What is the meaning of outage cost in business? ›Related Definitions
Outage Costs means the actual increased costs of replacement energy incurred by Transmission Owner during an Outage calculated in accordance with this section and does not include costs that would have been incurred notwithstanding the Generating Facility interconnection.
| Availability % | Downtime per year | Downtime per month |
|---|---|---|
| 99.9% ("three nines") | 8.77 hours | 43.83 minutes |
| 99.95% ("three and a half nines") | 4.38 hours | 21.92 minutes |
| 99.99% ("four nines") | 52.60 minutes | 4.38 minutes |
| 99.995% ("four and a half nines") | 26.30 minutes | 2.19 minutes |
Common categories of downtime include excessive tool changeover, excessive job changeover, lack of operator, and unplanned machine maintenance.
What are some examples of downtime? ›Downtime has many causes, including shutdowns for maintenance (known as scheduled downtime), human errors, software or hardware malfunctions, and environmental disasters such as power outages, fires, flooding or major temperature changes.
How do you calculate downtime cost per hour? ›The cost per hour of downtime is calculated by adding labor costs per hour to the revenue lost per hour.
What is the number one cause of downtime? ›Human Error: Regardless of whether accidental or due to negligence, human error is one of the most common causes of unplanned downtime. An employee unintentionally deleting data or accidentally unplugging a cable or not following standard protocols can lead to costly downtime.
What are the two types of downtime? ›Downtime falls into two categories: planned and unplanned. Planned downtime is notable because it offers advanced warning and gives users a chance to prepare. Planned downtime is usually done for upgrades or maintenance to the network infrastructure.
How do you calculate downtime? ›
To get a quick estimate of your company's probable downtime costs, use the following formula, based on the size of your business and the number of minutes your most recent incident lasted: Downtime cost = minutes of downtime x cost-per-minute.
How do you define an outage? ›an interruption or failure in the supply of power, especially electricity. the period during which power is lost: a two-hour outage on the East Coast.
How do companies keep their costs down? ›Cost cutting measures may include laying off employees, reducing employee pay, closing facilities, streamlining the supply chain, downsizing to a smaller office, or moving to a less expensive building or area, reducing or eliminating outside professional services, such as advertising agencies and contractors, etc.
What are the financial impacts of downtime? ›The cost of downtime = downtime duration x per-minute cost.
You can use around $400 as a cost-per-minute figure for small enterprises. In the case of large and medium businesses, use $10,000. Many people only associate downtime costs with lost revenue.
Maintenance Outage means a time period during which Seller plans to reduce the Power Output of the Power Product, in full or in part, in order to facilitate maintenance work on the Generating Facility, other than a Major Overhaul.
How do you calculate monthly downtime? ›1. Divide your total revenue by the planned operating time to get your daily revenue. 2. Assess by how much your daily revenue goes down if the chosen piece of equipment stops working for 1 hour.
What is 5 nines downtime? ›Availability is normally expressed in 9's. For example, “5 nines uptime” means that a system is fully operational 99.999% of the time — an average of less than 6 minutes downtime per year.
What is average downtime? ›Average downtime is usually built into the price of goods produced to recover its costs through the sales revenue. Opposite of "uptime." Also called "waiting time."
What is the industry standard for downtime? ›World Class Standards For Downtime
Aim for unscheduled downtime to be 10% or less.
What Are the 8 Wastes? The Lean Construction Institute has identified eight different kinds of waste that occur during projects: Over/Under Production, Waiting, Unnecessary Transportation, Over/Under Processing, Excess Inventory, Unnecessary Motion, Defects, and Unused Creativity of Team Members.
What is a major outage? ›
More Definitions of Major Outage
Major Outage means any Power Outage that lasts for at least ten (10) consecutive minutes and/or any Temperature Irregularity, in each case causing inoperability of Customer's Equipment.
It's basically what it sounds like—a list of things you can do in your downtime. I consider it a low-pressure cousin to the to-do list. It's not anything fancy, just a bank of activities and tasks that I either enjoy doing or need to get around to doing at some point.
How do you manage downtime? ›- Know the best windows of time for planned downtime based on your company's production cycle. ...
- Prioritize all your assets and know which should be handled first. ...
- Implement clear guidelines and well-defined standard operating procedures (SOPs) for each repeated operation.
Calculating Downtime Cost
The duration of the downtime and the cost incurred per minute you're offline are the two variables that most affect the financial impact of an outage.
A good rule of thumb is to allow yourself at least 3.5 hours a day. It's time used to engage in activities that promote rest and relaxation. The benefits of scheduling downtime are plentiful. Not only will it help reduce stress and fatigue, but it can also lead to improved concentration, creativity, and productivity.
What are the consequences of downtime? ›Consequences of unplanned downtime
Lost productivity and revenue: Every minute of downtime can result in lost productivity and revenue, affecting a business's bottom line. Decreased customer satisfaction: Unplanned downtime can lead to delayed deliveries, canceled orders, and frustrated customers.
The downtime policy ensures that systems be taken offline to maintain and improve system performance, safeguard data, or to respond to emergency situations.
What are downtime metrics? ›The most well-known downtime metric is Mean Time to Repair (MTTR). The MTTR metric reflects the average time it takes to troubleshoot and repair a failed piece of equipment.
What is another word for outages? ›On this page you'll find 11 synonyms, antonyms, and words related to outage, such as: blackout, brownout, disruption, interruption, dimout, and disconnection.
What is an unplanned outage called? ›An unplanned outage (also called an unscheduled outage) is typically caused by a failure.
What is the opposite of outage? ›
Noun. Opposite of suspension of operation. uptime. operating time. operable time.
What are three techniques used to reduce cost in a business? ›Combination: Bundle goods and services across an organization to reduce costs. Elimination: Remove unnecessary products, processes, benefits, and workflows. Optimization: Streamlining processes and workflows to reduce bottlenecks and redundancies. Substitution: Using cheaper products or services.
What is the strategy to reduce the cost? ›Cost reduction strategies are practices and principles designed to optimize operational efficiency. They cover all aspects of running a business, from hiring employees to booking flights. Successful implementation works by streamlining processes, allocating resources effectively, and eliminating waste.
How do you control costs? ›- Planning the budget properly. One method of cost control that most businesses use when starting a new project is budget management. ...
- Monitoring all expenses using checkpoints. ...
- Using change control systems. ...
- Having time management. ...
- Tracking earned value.
Network downtime means that your customers can't access your online services. They can't find or buy your services and products. If your potential customers can't access your website, then it will affect your revenue. Also, your existing customers can't access your products and services.
How do you measure downtime impact? ›The first way to measure your equipment downtime is in actual time. For a given asset (or set of assets), record the amount of time during each month that the asset is broken down. Keeping a running tally and comparing it to past months will help you know when an asset is having more issues than normal.
What is the purpose of downtime? ›Planned downtime can reduce the total time equipment is offline, which reduces costs. Planned downtime can extend the lifespan of equipment and machinery. Following a regular schedule for maintenance also improves asset performance.
What is the difference between downtime and maintenance? ›In manufacturing, “downtime” occurs when an unplanned event halts production for a period of time. This event can be a malfunction, repair, or changeover of tools or equipment. Maintenance downtime in particular is when a machine is not operating or being productive due to required maintenance work.
How can service outages be prevented? ›- How Do You Avoid Service Downtime? The best way to avoid service downtime is to: ...
- Use Enterprise-Level Network Infrastructure. ...
- Always Have a Backup Plan. ...
- Keep Things Simple. ...
- Monitor Frequently. ...
- Test and Retest. ...
- Deploy Network Redundancy. ...
- Regularly Update Systems.
An outage management system (OMS) is a utility network management software application that models network topology for safe, efficient field operations related to outage restoration.
How much does downtime cost in the auto industry? ›
For example, in the auto industry, downtime can cost up to $50,000 per minute. That's $3 million per hour. 400 The true downtime cost includes a variety of wasted business support costs and lost business opportunity costs because resources were needed to resolve a downtime incident that probably didn't need to happen.
How much does downtime cost in healthcare? ›The cost of information technology (IT) downtime for the health care industry is similar to other enterprises, with most recent studies citing ranges between $5,300 and $9,000 per minute.
Is database downtime costly? ›Database outages can have a significant impact on top line revenue. In fact, according to a survey conducted by ITIC, 98% of organizations say a single hour of downtime costs over $100,000, while 81% report that it costs over $300,000. And that's just for a single hour!
How much does unscheduled downtime cost? ›Unplanned Downtime Has Plenty of Obvious Costs
That downtime comes at a cost, and it isn't cheap. For example, the average automotive manufacturer loses $22,000 per minute when the production line stops. That quickly adds up. Overall, unplanned downtime costs industrial manufacturers as much as $50 billion a year.
The chip shortage will cost the global auto industry in 2021 $210 billion in revenues and lost production of 7.7 million vehicles, consultant AlixPartners estimated in September.
How bad is the chip shortage for the auto industry? ›The lack of sufficient semiconductors caused widespread production cuts around the globe. Worldwide, carmakers are estimated to cut a total of 19.6 million vehicles out of their production schedules between 2021 and 2023.
How much did the chip shortage cost the auto industry? ›In 2021, hamstrung by the global microchip shortage, the automotive industry lost more than $200 billion. Eleven million fewer vehicles were produced; manufacturing plants idled.
Is downtime a direct or indirect cost? ›The direct costs of downtime are the expenses you can easily quantify and attribute to a specific downtime event. They include the: Cost of lost employee productivity: This expense captures how much money was lost because employees could not work during the downtime event.
What is the average application downtime per year? ›US companies suffer an average of 12 incidents of unplanned application downtime per year.
What are 3 disadvantages of database? ›- Increased Cost. High cost is one of the main disadvantages of DBMS, the cost can be of many types like hardware or software costs, data storage costs, etc. ...
- Complexity.
- Database Failure. ...
- Performance. ...
- Frequent Updates/Upgrades. ...
- Huge Size.