Stuart Sheehan, Alexander Rakow
IMAGE LICENSED BY INGRAM PUBLISHING
For decades, data center operators have been drawn to the eastern stretches of Oregon and Washington State, USA. In contrast to the coastal climate that most people conjure when thinking of the Pacific Northwest, the land east of the Cascade Mountains is relatively flat, dry in the summer, and snowy in the winter. Affordable prices for land, tax breaks, and cheap hydroelectric power have made the region one of the main hubs of data center development in the United States, with Amazon, Apple, Google, Meta, and Microsoft all investing in multiple campuses.
Despite the relatively remote setting of these facilities, public opposition to data center development has grown in the region, with residents citing the energy and water demands of the facilities. In response, both states have introduced legislation that would apply new clean energy requirements to the data center industry. In Oregon, HB2816 would require data center operators to reduce their greenhouse gas (GHG) emissions per megawatt hour under a standard baseline in set increments, leading to 100% reduction in 2040. A similar measure has been introduced in Washington State, which would apply the same decarbonization schedule to data centers as those already applied to the power sector. If these laws pass, data center operators will need to respond to both public pressure and regulation when designing new facilities and make plans for how they procure and use power.
Similar stories are playing out across the country and around the world. The proliferation of data centers over the past decade has brought these formerly obscure facilities into the public consciousness, and has raised concerns over climate impacts, resource depletion, and competition for water and grid capacity. Data center operators are under pressure to demonstrate that their facilities will be sustainable enough to satisfy the local community, self-sufficient enough not to burden local grids, and as always, resilient enough to provide the reliability their customers require. These pressures are increasingly reflected in every aspect of design, and for many companies, backup power systems are next on the list.
Resilience and reliability are cornerstones of data center requirements, driving choices in all facets of site design, from physical security and site location to electrical design. Site uptime and availability have long been principal to service level agreements, with “five nines” or 99.999x% availability permeating architecture discussions. Traditionally, data centers have backed up their utility-sourced power with diesel generators to ensure continuity of service when there is an upstream outage. On-site generators are used daily in some locations with poor grid availability, but in most cases are rarely used. In these low-use locations, despite having expensive on-site generation with low utilization, grid parallel or demand response participation has been historically limited by sustainability commitments, air quality permits, and the high operating costs of diesel generators.
While standby diesel generator sets, or gensets, deployed solely for emergency backup continue to be the default today, the door is opening to a more diverse and dynamic future.
The data center industry is facing both “pushes” and “pulls” to change backup strategies over the coming decade. Among these are the push of regulatory pressure, the pull of improved sustainability performance, and the opportunity to replace diesel generators with solutions that are more efficient, capable, and resilient for dynamic use. Although all of these pressures are growing, they do not spell the imminent death of diesel, and are not likely to precipitate an overnight change. However, they are conspiring to kickoff the early stages of what is likely to become a widespread transition to more sustainable and dynamic microgrids as a solution for data center resilience.
Diesel generator emissions impact both air quality and climate change, making them a target for regulation. Although data center operators typically expect their generators to run only a few hours or days in a year (given the reliability of the utility grid), the sheer number and scale of the backup generators necessary to cover the load of the data center can trigger a matrix of permitting requirements. In the United States, a data center project may require permits from federal, state, and even local regulators before it can break ground. Any hitch in this process, whether clerical or technical, can cause delays in the data center development process, often at terrific cost to the data center owner.
With growing frequency, utilities and grid operators are calling on data centers to run their generators preemptively to balance loads and prevent grid-outages. These requests can add days or weeks to the annual runtime of the generators, significantly increasing both their emissions and cost to operate. In Virginia, USA, the state Department of Environmental Quality (DEQ) has proposed suspending certain requirements to allow data center operators in the state to run their generators for extended periods of time and avert disruptions from overtaxed transmission infrastructure. In a public notice soliciting comment on the plan, the agency wrote: “Data center operation relies on the use of large amounts of electricity from the grid. DEQ is concerned that the Counties of Fairfax, Loudoun, and Prince William is an area in which there may not be a sufficient amount of electricity for data centers due to severe, localized constraints in electricity transmission.”
Although this plan may grant data center operators a reprieve from the regulations and permit requirements that would have otherwise prohibited the extended run of their generators, the proposal has already raised the hackles of community members and activists who are not eager for diesel to become the answer to the region’s energy woes. In response to pressure from those concerned, legislators in the state have introduced a law that would require the Virginia Department of Energy and DEQ to study the effects of data center development on the state’s environment, energy resources, and sustainability goals. As data center development is increasingly constrained by the ability of existing energy infrastructure to support the additional load, data center operators will be under growing pressure to demonstrate energy self-sufficiency and sustainability to win the support of both regulators and surrounding communities.
Of course, in addition to restricting emissions from diesel generators, governments can encourage the proliferation of cleaner alternatives through public investment. The United States and the European Union have both recently created new incentives for renewables, energy storage, and microgrid projects that may improve the business case for replacing diesel generators.
The sustainability plans of hyperscale data center operators have helped to establish the data center industry as one of the most ambitious in the private sector. Data center operators have set goals to rapidly drive down the waste generated in data center construction, the water used to cool them, and the energy used to operate them. Most major players have committed to achieving some form of carbon neutrality or net-zero carbon emissions in the next few decades, and those that have not are under pressure from customers, investors, and even employees to catch up.
New data centers continue to be built as quickly as global supply chains will allow, and their cumulative demand for energy, materials, and water is leaving a growing environmental footprint. This makes the ambition of the industry’s sustainability goals all the more daunting, and diesel generators have not escaped the attention of design engineers seeking to control emissions.
In addition to contributing a (typically very small) portion of a data center’s GHG emissions, diesel emissions include more than 40 toxic air pollutants, many of which are carcinogenic. According to a California Air Resources Board estimate, an uncontrolled 1-MW diesel engine operating for only 250 h per year would increase the cancer risk to residents within one city block by as much as 50%.
In 2020, Microsoft announced a goal to eliminate diesel fuel from their data centers by 2030 as part of an overall goal to become carbon-negative by that same year. Microsoft noted that “diesel fuel accounts for less than 1% of Microsoft’s overall emissions but finding solutions to reduce reliance on traditional diesel fuels represents a substantial contribution to the technology pathways necessary for deep decarbonization. The most immediate opportunity to reduce the use of traditional diesel fuels is to validate and implement alternative, lower-carbon fuel sources for our generators.”
The rest of the industry is increasingly buying into the vision that the replacement of diesel generators with cleaner, more sophisticated alternatives could have a sustainability impact far greater than the modest climate benefit of eliminating a few hours of diesel burn each year. Given sufficient development, successor technologies, like battery energy storage and hydrogen fuel cells, could be deployed not just during outages, but on-demand to support a variety of use cases that support a cleaner, smarter, and more resilient energy grid.
In ideal circumstances, the significant investment a data center operator makes in diesel generators sits idle throughout the year (when not being run for maintenance). Aside from backup during grid outages, diesel generators are not expected to provide any value. The limitations of air permits and the ambition of sustainability plans prevent them from being deployed for peak shaving, demand response, or any other use case. When diesel is replaced by clean generation (potentially including cleaner generator fuel alternatives, like hydrotreated vegetable oil), these constraints are lifted, and data center design engineers are free to imagine ways that backup power systems could yield both economic and sustainability value.
To take full advantage of these opportunities, on-site generation solutions must feature not just a cleaner emissions profile, but the intelligent control necessary to manage on-site generation, storage, and crucially, interaction with the utility grid. Advanced software controls are capable of managing all of these elements as a system, and can be tailored to optimize performance for cost, emissions, and resilience. It is this central control that turns a data center into a true microgrid. A system like this could be programmed to orchestrate when an on-site generation resource like a hydrogen fuel cell powers the data center, charges on-site battery storage, or exports electricity to the grid. In doing so, a data center could maximize opportunities for financial benefit through grid interaction, including taking advantage of time-of-use rates, participating in demand response programs, providing grid ancillary services, and getting compensated via wholesale energy markets. The data center operator could also use the system to pull electricity from the grid when the utility grid mix is the cleanest or orchestrate the deployment of on-site resources with off-site renewables in pursuit of 24/7 carbon-free energy.
In 2022, Microsoft announced that it would be building a diesel-free data center in San Jose, CA USA featuring a natural gas generator, and completed a working pilot of a 3-MW hydrogen fuel cell system capable of covering the entire load of a typical data center. These types of projects will help to demonstrate the viability of new technologies, and spur the supply chains necessary to support them. This is especially true with hydrogen, where production and distribution is not far removed from infancy. The data center industry has the potential to create the demand necessary to support the build out of significant green hydrogen infrastructure, yielding benefit not just for data centers for but for society at large.
We have already seen the pace of technical innovation increase for energy resources like solar, wind, battery storage, and electrolyzers. Innovation has brought precipitous falls in prices for all of these solutions and has also made them more flexible. This has improved the business case for data center microgrids and expanded the definition of what they can look like. While current grid-connected battery storage and fuel cell systems emulate diesel backup (<10-s transfer and ac coupling), there is an opportunity to reevaluate how these systems can look and operate once the diesel paradigm is dropped. DC coupling to provide immediate power support to critical loads is of great value to data center applications.
As microgrids are deployed in a variety of industries today and the preceding catalysts will make microgrids more and more appealing for data centers, solutions must be developed and evolved to meet the specific needs of data center applications. The following four focus points will be critical in successfully driving adoption of microgrids in the data center industry:
The data center industry is continuing to rapidly expand at a global scale. There are multiple gigawatts of hyperscale data center capacity under development, with individual providers building tens of data centers and hundreds of megawatts annually. To execute at this scale and pace, design standardization is key to simplifying procurement and minimizing construction phase uncertainty. Most data center builders leverage modular or standardized electrical powertrain blocks, which can be repeated throughout their facilities to achieve the desired power and redundancy level (Figure 1). The size of these building blocks and many electrical assets revolves around low-voltage (LV) equipment breakpoints. For example, 3,000- or 4,000-amp LV switchboards serve as optimized backbones for 1- to 3-MW powertrains with LV integrated generators. When designs utilize centralized generator plants at medium voltage (MV), individual generators are still usually LV equipment of similar sizes with step-up transformers, providing a balance of scale and redundancy. The latter is key, as many hyperscalers require multiple equipment sources to limit supply chain risk.
Figure 1. Example layout of modularized hyperscale data center (a) built of repeatable power trains (b). Modular designs increase speed to deploy, allow site expansion, and improve consistency of experience across a fleet. HV: high voltage.
As data centers look to adopt more microgrid functionalities and new distributed energy resources (DERs), solutions must support both deployment speed and scale or they, simply, will not be adopted. This is an evolution from today’s approach, which may offer scale or speed, but seldom both. Large microgrid projects, such as airports, ports, or military campuses, are usually single projects with long design and development cycles and little repeatability, processes that are incompatible with the data center segment. For most applications, new DERs must be standardized into consistent solution chunks that are compatible with the highly optimized building blocks deployed by the industry. This will require well-defined interfaces for both power and controls so that products from different vendors can consistently work together. Systems must provide required certifications and listings to facilitate rapid authority having jurisdiction acceptance and utility interconnection approvals. In some instances, zero export designs will be used to avoid lengthening grid interconnection.
Despite decades of impressive energy efficiency innovation, data center applications are inherently power and energy intensive. While the performance of IT equipment has expanded significantly, so has energy and power consumption. Rack densities greater than 10 kW (15 kW/ft2) are often targeted for standard air-cooled equipment with higher densities becoming more common for high-performance computing. At the data center and campus boundary, power levels have grown to the tens and hundreds of megawatts. Facility design optimization and similar increases in the power density of supporting equipment (power converters, battery systems, cooling systems, and power distribution) have enabled facilities to accommodate this IT equipment with limited footprint. Buildings often exceed 350 W/ft2 and centralized data center campuses can be above 0.5 MW per acre, all in, including substations, stormwater, parking, networking, and administrative buildings.
These power densities become an obvious challenge for the deployment of on-site renewable generation to provide prime power or to significantly cover operations. Solar is widely deployed in many smaller microgrids, but typically provides in the order of 10 W/ft2 (Figure 2). Further, rooftop photovoltaic (PV) has to compete with rooftop space for other critical systems. A study of one data center found roughly one-third of the roof area suitable for PV deployment. Finally, data centers operate 24/7 at a high utilization, compared with solar capacity factors (20–30%), and this makes the energy gap even wider than the power gap. The same study of a data center in Texas, USA mentioned previously found that rooftop solar would only provide 0.2% of the site’s energy needs. While further optimization could increase this percentage, it is unlikely to play a large role at large centralized data centers. Overall, renewables will play an important role, but primarily for smaller edge sites or sited off or near-site where there is sufficient land and they are the most productive.
Figure 2. Approximate footprint comparison between DERs and a data center building. DERs are oversized for redundancy. Only 1-MW nameplate of photovoltaic (PV) is shown. BESS: battery energy storage system; LiB: lithium-ion battery; PEM FC: proton-exchange membrane fuel cell.
As data centers transition to new distributed energy technologies, energy and power density will be strong input to solutions. Battery energy storage systems based on current technologies can provide instant power and competitive power densities, however their energy density can be poor at longer runtimes (Figure 2). As a result, data center battery deployments based on lithium-ion will likely focus on sub-4-h solutions and not provide an exact replacement for long-term backup. Fuel cells separate energy and power components, providing more competitive density for longer runtime and prime energy applications, but emphasize effective fuel storage and availability. This is especially critical with the potential emergence of hydrogen, which must introduce footprint effective solutions for on-site storage and/or delivery (Figure 2). For large-scale campuses, this may lead to pipeline delivery and/or liquid hydrogen storage.
Overall, the technology mix for data center microgrid solutions will differ from less power-dense facilities. Solar will rarely play a significant on-site role for hyperscale data centers and on-site storage and generation must be able to achieve large scales and be deployed with high density.
Sustainability is a strong catalyst to move beyond static diesel backup. However, transitioning to new DERs and microgrid functionalities should be constructive to these sustainability objectives when viewed from rigorous life cycle analysis. Where new fuels are used in engine generators and fuel cells, these fuels should be produced sustainability and not just eliminate tailpipe emissions. While hydrogen has zero local carbon emissions, most hydrogen today is produced by stream methane reforming, creating upstream carbon emissions. A robust supply of green hydrogen (created from electrolysis with clean electricity inputs) is required for tech adoption in data centers. Furthermore, the embodied carbon of new solutions must also be considered. Lithium-ion battery storage has zero local emissions and high round-trip efficiencies but can have significant embodied carbon for long-duration deployments.
The evolution of data center energy systems should also support operators’ increasingly rigorous goals to decarbonize regional energy systems. Traditional carbon emission tracking involves simple monthly and annual net-based calculations. However, maximizing regional decarbonization requires moving into time-domain tracking; clean supply must match demand at all times to sustainably keep the lights on (Figure 3). One may reach “net-zero” but still not have sufficient clean generation for a significant portion of their operating hours. As a result, energy sustainability goals of market leaders have pushed into the time domain. Microsoft and Google have set ambitions to run 24 h, 7 days a week, and 365 days a year on carbon-free electricity (matched hourly) by 2030. While progress on these goals will initially be driven by an off-site energy procurement strategy, on-site microgrid operations are poised to play a role as well. Therefore, systems should be able to track local system sustainability and regional energy system [grid and power purchase agreement] telemetry in real time.
Figure 3. California data center: percent of operating energy that is low carbon (hour by hour) during 2021. Grid only, no power purchase agreement shown.
Real-time grid telemetry is important to rigorously track normal operations, but it becomes essential in accurately tracking the sustainability of grid-interactive microgrid actions. High grid emission rates are correlated with elevated costs and demand response periods [Figure 4(a)], making annual average emission factors inaccurate in these times of action. Local microgrid activity during these times can displace the dirtiest and costliest peak generation engaged in times of grid scarcity [Figure 4(b)], a benefit that is not accounted for when using traditional annual average grid emissions factors. Furthermore, real-time grid marginal emissions rates better reflect the impact of operator actions and can be used to calculate emissions savings for the regional energy system. Marginal emissions rates are also likely to be elevated during times of grid-interactive operation.
Figure 4. Grid (SPP South balancing area) emissions factor versus locational marginal price (LMP) during 2021. (a) A zoomed-in view is shown illustrating the positive correlation during the majority of operating periods. (b) The zoomed-out view is displayed, showing the increased emissions during times of largest scarcity and extreme LMPs. A horizontal black line is drawn to indicate the annual average.
While energy security and resiliency have long been a driver for microgrid deployment, data centers are specialized critical facilities with a particular emphasis on availability. First, data centers are almost always designed with electrical redundancy. To ensure availability, dual corded loads, redundant power paths, and redundant (2N or N + 1) generation equipment are standard at most sites. For LV integration, DERs will be distributed within the LV architecture, matching the same redundancy schemes of the other LV equipment. This can simplify architectures and controls while limiting the “blast radius” of a source failure to that particular IT room. Loads will remain powered using redundant capacity; however, further fault tolerance is limited. Hyperscale data centers are large enough to have a significant MV footprint and some operators will utilize MV DER integration. MV DER integration creates larger pools of DER assets to reduce stranded capacity and can minimize oversizing of the generation fleet while maintaining N + X redundancy. However, large centralized genset plants increase the potential “blast radius” of failures and increase the criticality of MV design on system uptime. Therefore, complicated designs are required to eliminate single points of failure, increase fault tolerance, and enable concurrent maintenance (Figure 5). Where relied on for resiliency, microgrid DERs will likely be deployed in similar optimized redundancy schemes as existing diesel gensets to meet uptime requirements and provide similar fault tolerance. The DERs themselves will also face scrutiny in their own reliability, as this can be a large driver of site availability.
Figure 5. Example genset architectures. MV architectures (a) can accommodate N + 1 generating assets and achieve fault tolerance and concurrent maintainability (close loop). LV architectures (b) in 2N and distributed redundant four makes three.
Control systems will face similar design rigor to ensure reliability and limit risk, applying to digital system architecture as well as use case implementation. Redundant local controllers may be required and systems should be designed to minimize risk of controller failures or external outages. Cloud operations can bring increased data access and compute power to optimization algorithms; however, many data center providers will initially prefer energy management system software to run on-premise rather than in the very same public clouds hosted in their data centers. This can avoid exposure to potential uncontrolled external factors. The cybersecurity of all connected devices and vendor practices will be scrutinized to comply with the high standards of data centers. Data center operations staff are advanced, dedicated users who demand a high level of visibility to site operations and microgrid asset integration into already advanced power management systems. As stated, data center designs are comprised of multiple, redundant energy assets, so supervisory systems must be capable of managing and monitoring multiple powertrains in a single site in a logical manner. As their primary directive is to ensure system uptime, operators must be given full visibility of DER system dynamics to provide them the confidence to move beyond their traditionally conservative approaches. Controls interfaces and upstream design tools must simplify the decision-making process, while keeping the reigns in the operators hands to make decisions. This includes likely activation of individual or a subset of DER capacity to maintain redundancy requirements and meet service level agreements.
How do data center operators currently weigh these catalysts for change against the necessary qualities of present solutions? Schneider Electric is working with customers from across the data center industry to define the business case for diesel generator replacement, vet successor technologies, and design the next generation of data center microgrids. Although each data center operator is at a different point in their journey, these partnerships have yielded several insights about the state of the industry.
Emergency-only backup power systems, anchored by diesel generators, are the trusty steed of data center electrical architectures, enabling site autonomy and uptime. To date, these designs have met the vast majority of requirements of data center operators: available at scale while optimizing low first cost and high energy density. However, this historically uniform set of requirements is becoming more varied, as dynamic energy systems and regulatory environments play an increasing role. This is happening against the backup of rapidly scaling new technologies increasing their viability. The data center industry is digesting these shifting ingredients and exploring the application of microgrid concepts. Architectures with multiple energy sources and more active use cases present the potential to unlock capacity in restricted electricity markets, improve sustainability, and lower total cost of ownership. To realize this promise, solutions must emerge to meet the specific requirements of data centers. Solution standardization will help to increase the speed and scale of deployment, while chemical energy storage and shorter runtime battery storage solutions will meet power and energy density pressures. Full lifecycle sustainability analysis and coordination with telemetry from the greater regional electricity grid will further enable operators to meet their ambitious sustainability objectives.
Progress does not come without effort, or overnight, and the data center industry is currently working to transition from narrow technology pilots to microgrid deployments at production data centers. No single stakeholder can achieve this; change will require collaboration among data center operators, customers, DER solution providers (fuel cells, batteries, etc.), and critical power and controls solution providers, such as Schneider Electric. Microgrids will evolve side-by-side with continued diesel generator deployment for the foreseeable future, but expect a new generation of solutions to emerge, giving the data center industry the leverage to tackle the energy challenges of the future.
B. Acun et al., “Carbon explorer: A holistic framework for designing carbon aware datacenters,” in Proc. ACM Int. Conf. Archit. Support Program. Lang. Oper. Syst. (ACM ASPLOS), 2022. [Online] . Available: https://research.facebook.com/publications/carbon-explorer-a-holistic-framework-for-designing-carbon-aware-datacenters/
A. Wiryawan and C. Baumann, “The reality of replacing diesel generators with natural gas, energy storage, fuel cells and other options,” Schneider Electric, Rueil-Malmaison, France, 2022. [Online] . Available: https://www.se.com/ww/en/download/document/SPD_WP14_EN/
N. Walsh. “Progress on our commitment to sustainable backup power in datacenters by 2030.” Microsoft. Accessed: May 28, 2023. [Online] . Available: https://azure.microsoft.com/en-us/blog/progress-on-our-commitment-to-sustainable-backup-power-in-datacenters-by-2030/
V. Boutin et al., “How microgrids contribute to the energy transition,” Schneider Electric, Rueil-Malmaison, France, White Paper, 2016. [Online] . Available: https://perspectives.se.com/drive-sustainable-growth/how-microgrids-contribute-to-the-energy-transition
Stuart Sheehan (Stuart.Sheehan@SE.com) is with Schneider Electric, Narragansett, RI 02882 USA.
Alexander Rakow (Alex.Rakow@SE.com) is with Schneider Electric, Ithaca, NY 14850 USA.
Digital Object Identifier 10.1109/MELE.2023.3291193
2325-5897/23©2023IEEE