Climate change is here and it is real. Extreme weather conditions are being felt all over the world. Drought has become a significant challenge. Mountain and forest areas have been experiencing record high temperatures for the past decade. Data centers located in these high-risk areas are in danger of being overwhelmed by high temperatures and low humidity, which increase the risk of fires. Although there are safety measures in the facilities to prevent fires, data managers cannot anticipate wildfires beyond their control.
Dried-up trees and plant life become fuel for forest fires. Whether caused by natural or deliberate intent, forest fires will hinder the operations of the data centers.
Let’s take look at an actual forest fire that took down a supercomputer in California. While California’s forest fires rages, power at the NERSC began to shutdown. For the second time in many weeks, the world’s most powerful supercomputers were carefully switching off.
As with people trying to evacuate California’s suburbs, the National Energy Research Scientific Computing Center (NERSC), which operates the supercomputer system called Cori, was truly powerless to stop the effects of anthropogenic climate change.
Wildfires not only can seriously damage equipment directly, but they also can cause indirect damage such as downtime, with server outages and cleanups from the fire. These fires are disruptive, as they can shut down internet access.
Wildfires in data centers not only create problems in the vicinity of the operating companies and its customers, but also put the people inside the facilities in danger. They also can have an adverse effect on other regions or continents.
Case in point, California on fire
In October 2019, after an unusually long dry spell, remarkably high winds, and arid conditions, Silicon Valley, home to a number of technology firms, faced the danger of being swept away by the wildfires.
As climate change exacerbates the causes of wildfires, it will make maintaining and serving data across the globe increasingly difficult. There needs to be a way to protect data center facilities from being destroyed by environmental forces.
Natural disasters are the bane of data storage and disaster recovery. Managers have to ensure that digital reservoirs are safe and unaffected, no matter what the situation is outside the facility. One data center operations leader recently experienced the necessity of effectively thinking through these scenarios during the deadly wildfires that raged through Sonoma County and other parts of California in October 2017, which ended up destroying more than 8,500 homes.
“My entire team was scattered all over Northern California,” said Stuart Lewallen, data center operations team lead for the County of Sonoma. “The fires had gotten within three blocks of our data center, so the county center had been evacuated, and they wouldn’t let us back. Everybody was working remotely. And that mostly worked okay, but again, we had a lot of learning points from the after-action. We learned a lot from what worked well and what didn’t.”
Offsite data and simplicity for disaster recovery
One thing that Lewallen’s team learned from the wildfire situation was that their data needed to be moved offsite. They used cloud solutions for their data recovery.
“Instead of having a disk array somewhere else, why not just put it up in the cloud?” said Lewallen. “If you don’t need to do anything with … your offsite back-up, it’s a fantastic use case.”
While the cloud is expensive, in the long run, the cost-effectiveness during a crisis can be justified by saving the data stored in their servers.
Another thing learned by the team was that simplicity is everything in a crisis. “Failover had to be simple,” Lewallen stated. “You have no idea who you’re going to have on your staff that’s able to pull the trigger on this. It can’t be some complex thing that only two people in your organization can do and it takes three days just to get it kicked off. It’s got to be a push-button.”
To have consistent and clean electricity, many data centers are located near hydroelectric power plants. This means they are often located close to mountains and forests— areas that are prone to wildfire in an era of climate change.
Road closures and forced evacuation of towns in the projected way of fires could pose immediate risks to staffing personnel: workers might not be able to get to the workplace, or to look after their families first.
A facility’s contingency plans should be kept up to date, and should include announcements for the personnel as to what actions to take in case of emergencies. Personnel should be aware of construction, road repairs, or other problems that might impact alternate routes. Create and test contingency measures in advance.
Wildfires can also burn transmission lines, causing power shutdowns at the facility. After the wildfire, burnt soil erosion can cause landslides, that destroy cables. With climate change, every scenario should be thought out and plans made accordingly.
Wind
High winds often make wildfires go out of control. They feed the flames the oxygen needed to make them bigger. With that in mind, power outages during wildfires are inevitable, as power companies shut down their equipment, to prevent accidents due to burned power lines. These planned outages may continue for a long time, as the utility company upgrades and adapts its systems to deal with the existing and future threats of more extreme weather conditions.
Datacenter facility supervisors also need to consider not only the direct line of fire, but also windblown embers. These embers can light up dry grasses around fence areas, collect near fuel tanks, and even burn employee vehicles in the parking lot. Anticipate any possible risks and plan mitigation strategies.
On-site power generators are one option supervisors might consider as a buffer against unreliable power supplies. There will always be, however, a chance that there will be power outages no matter the amount of preparation. As a result, these utility power outages, whether planned or unplanned, will aggravate the failure of delivery for data center operators.
Smoke and ash
Datacenter facilities may be sited far from an active wildfire, but the resulting smoke and ashes that follow can linger around the area for weeks at a time. This means that the resulting debris from the wildfires could as easily affect the facilities as the fire itself.
All data centers need a continuous flow of fresh air supply to cool their servers. This flow is necessary to remove the stale air and bring in fresh air. Filter systems are installed to stop any particulates in the air from entering the facility. Also, ash can be transported into the facility via the workers’ clothes, maintenance tool kits, and equipment. Even slight movement can agitate the particles to become airborne. Data center wildfires are in danger from smoke and ash infiltration and should take action to prevent contamination.
Engine backup generators, which supply emergency power, may also pose issues. Most have filters on the combustion air intake, but there are rarely any filters on the intake for engine cooling. Talk with manufacturers to improve filter control measures so they do not impact performance or affect warranties.
Fuel storage tanks and the fuel itself can also be a problem. Almost all diesel tanks have vents. These vents allow air into the tanks when fuel is withdrawn and act as a vent for air to escape when fuel is added. Once the generators are used, the vents will be pulling in the air — which means particulates can get into the tanks. Eventually, they will settle to the bottom of the tank and will be picked up by the fuel pumps and clog fuel filters.
The cooling system of all the facilities will be affected the same way. Condenser units for air conditioners, air-cooled chillers, or cooling towers all take in air from the outside and, in effect, suck in particulates from the wildfires. The particulates will contaminate the water in cooling towers or clog condenser coils, resulting in less cooling capacity. Backup units may be similarly affected. Facilities with direct air-cooled data centers need to check the air quality and adjust their filter maintenance schedules whenever possible.
Fires Inside Facilities
Generally, most anticipated fires will come from within a facility, not from the outside. Managers have little or no influence on what goes on outside their area, but with a little foresight, they can create plans for dealing with the effects of wildfires. Alarm and suppression systems are essential for a rapid response to protect your data center, as well as plans for personnel protection. Being prepared for dealing with fires inside the facilities also require planning.
Disaster in data center room server and storage on fire burning
Common fire hazards found in data centers:
- Wires and cables
With any data center, wires and cables are part of the equipment. Cables may not ignite by themselves, but they should be considered as fuel for the fire. The risk of fire comes from power cables, which, if substandard or faulty, could spark, as a source of ignition, and set other components or infrastructure on fire.
It’s imperative to have power cables checked regularly. It’s important to ensure that all the wiring and cables are kept in a neat and tidy fashion for clear inspection – the more they are abused or crushed, the more likely the wires will wear, become exposed, and start a spark.
Datacenter managers who have experienced internal fires should document conditions and actions taken during the crisis, then share the plans with others. The next time the data center management will be more prepared.
- HVAC equipment
The heating, ventilation, and air conditioning equipment (HVAC), works to keep data centers cool and well-ventilated, but also poses dangers because of the motors and electrical components – as well as gaseous refrigerants.
This equipment needs to be well maintained and regularly checked to prevent being a fire hazard. Where break downs occur, they must be repaired immediately. A log of all equipment checks must be maintained.
- Aeration spaces
Raised floors and suspended ceilings in data centers pose a danger also. Within these gaps of the floor or ceiling, any fire breaking out in these spaces might not be detected immediately. To decrease the risk, install fire alarms and smoke detectors.
- Combustible materials
The most common source of fire is combustible materials such as cardboard boxes, paperwork, and other items typically found lying around or in supply rooms. While they don’t present much of immediate danger, they may not have been considered as a source of fuel for a fire. Clear them away when not in use, and store them in safe, cool areas with no equipment nearby.
This is important, as flames can come from within or outside the facilities. The larger the fire, the more heat it creates, and the faster it grows.
Conclusions:
Climate change has changed how data centers cope with fires, not just inside the buildings, but also from outside. Their locations frequently are near hydroelectric power generators. As a result, the risk of wildfires is high due to the mountains and wooded areas. Drought is a very big concern, especially for forests, as drought turns the trees into the fuel for wildfires.
Datacenter managers must create contingency plans, not only for internal fires but also for outside fires. Of course, the primary concern is the workers operating the facility. Their safety and that of their families must come first, before recovering data.
The best method of saving and retrieving data is to take advantage of storing data in the cloud, where anyone with access can retrieve it, from anywhere in the world.
Lessons learned from surviving wildfires by the data center managers should be shared to assess possible measures for coping with future wildfires.
