SBEED Validation Against ASHRAE Standard 140-2014

June 2017

Murray Milne
UCLA Department of Architecture and Urban Design
Professor Murray Milne

For the California Energy Commission Contract PIR-12-032:

Tools and Materials for Zero Net Energy California Buildings,
SBEED: Small Building Energy Efficient Design

INTRODUCTION

SBEED (Small Building Energy Efficient Design) is an easy-to-use day-one design tool that
helps owners, builders, and architects create a more energy efficient non-residential building.
ASHRAE Standard 140-2014 is a method for evaluating a building energy analysis computer
program by running 35 variations of a small building design then comparing the results of using
eight different energy performance programs. The current release of SBEED 1.0 (Build 4) could
run 27 of these cases and the heating and cooling loads were reported and compared for each.

BACKGROUND

SBEED uses the Solar-5 computation engine, developed at UCLA beginning in 1978 for its
thermal analysis kernel. Solar-5 calculates an hourly heat balance similar to the method used in
EnergyPlus. It finds the heat gain or heat loss for each of the 8760 hours in a year using standard
ASHRAE algorithms, the Mackey and Wright time lag and decrement factor method of
accounting for heat flow through external mass walls, the Admittance Factor Method to account
for internal thermal mass, and the California Energy Commission’s ACM method to calculate the
performance of basements. To find the hourly heat balance it uses a successive approximation
method to calculate the indoor air temperature. Thus it can integrate loads and energy
calculations at each hourly time step, which means that the HVAC system only adds heating or
cooling energy if the indoor air temperature has floated beyond the upper or lower comfort
limits.

ANSI/ASHRAE Standard 140: Standard Method of Test for the Evaluation of Building Energy
Analysis Computer Programs was developed by the American National Standards Institute
(ANSI) and the American Society of Heating, Refrigerating and Air-Conditioning Engineers
(ASHRAE). This standard is updated periodically, and the version from 2014 has been used to
test the recently released version of SBEED 1.0 Build 4.

Test Procedures:
Standard 140 specifies test procedures to evaluate the results produced by software designed to
calculate the thermal performance of a building and its environmental control systems. The tests
are based on the principle of comparing the performance of one program against the performance
of other programs, and while the tests are not intended to evaluate all aspects of the software,
they are designed to indicate any serious flaws or limitations.
Standard 140 uses a small reference building of 48 square meters (26.25 by 19.69 feet) that has
35 variations of envelope, windows, internal loads, and infiltration. For each case its
performance was reported for eight simulation programs: ESP, BLAST, DOE2, SRES/SUN,
SERIRES, S3PAS, TRANSYS, and TASE. For each case the overall minimum and maximum
value is reported for heating load and for cooling load among these eight programs. The annual
heating and cooling load was also calculated for each case using SBEED, and whether it falls
within the minimum and maximum acceptance range of all the other eight simulation programs.

SBEED, like most U.S. energy simulation programs, uses Inch-Pound units, while Standard 140
is reported in SI (metric) units of MWh per year, which in this report is converted to MBTU per
year. The annual performance of SBEED is reported in MBTU/year on the Building Energy
Performance (BEPS) screen and is reported here in Table 1 and 2. These results are also plotted
graphically in Figures 1 and 2.

The 35 cases in Standard 140 range from quite realistic to extremely abstract. Eight cases are not
included here because using SBEED some test variables can not be changed: interior infrared
emittance, interior shortwave absorption, exterior combined radiative and convective surface
coefficients, cavity albedo, solid opaque windows, and adding a sunspace. The results of the
remaining 27 cases are plotted graphically in Figure 1 for Annual Heating Loads and in Figure 2
for Annual Cooling Loads, and are reported numerically in Table 1 and Table 2.

RESULTS

It is important to emphasize that in Standard 140 no formal validation criteria are established to
determine the range of acceptable results (ANSI/ASHRAE 2007, Section 4.4.1). Thus while this
study does not demonstrate official acceptance ranges, it does show that SBEED closely follows
the same pattern of performance as these other eight simulation programs (Fig. 1 and 2).
Standard 140 is used here in part because it was also used previously in the development of our
earlier design tool HEED (Home Energy Efficient Design) which also uses the Solar-5
computation engine, but for residential buildings. Comparing the performance reported for both
SBEED and HEED also shows that they both closely follow the same pattern of performance
(Henkhaus, 2012).

SBEED fell within the normative range on 33 of the 54 cases that it ran. This includes 18 of the
27 heating load comparisons (Figure 1), and 15 of 27 cooling load comparisons (Figure 2). Note
that in all cases SBEED tends to be conservative, in that it estimates that a higher heating or
cooling load is needed compared to the average of the other eight programs. Thus, an actual
building would likely use less annual energy than SBEED predicted. This means that when
designing a Zero Net Energy building SBEED will have a higher probability of meeting that
goal.

Compared to the eight reference programs, SBEED showed the same magnitude and direction of
performance of in each successive case. This implies that when SBEED is used as a design tool,
each incremental design change to a building should produce changes in heating and cooling
energy that are of the correct magnitude and direction.

ASHRAE Standard 140-2014 Comparison

SBEED 1.0 Build 5: ANNUAL HEATING
ASHRAE Minimum ASHRAE Maximum SBEED Value
Mbtu/year

TESTCASE

Figure 1: Annual Heating Loads Comparison

 ASHRAE Standard 140-2014 Comparison
SBEED 1.0 Build 6: ANNUAL COOLING

Range Minimum Range Maximum SBEED Value

40.000

35.000

30.000

25.000
Mbtu/year

20.000

15.000

10.000

5.000

0.000

TEST CASE

Figure 2: Annual Cooling Loads Comparison

DISCUSSION OF RESULTS

Reference Programs:
Note that Standard 140 is based on results of eight reference computer programs that are now
20 or more years old, most of which are no longer considered state of the art in building
simulation. Missing from this list is EnergyPlus, which is now considered by many in the
U.S. to be the standard of the industry. In 2006 and 2010 EnergyPlus was run against these
same eight reference simulation programs, plus three more that were added (BLAST 3.0,
DOE2.1E, and DOE2.1E-RevWindow). In the 2010 test series using EnergyPlus 6.0.0.023
the test files generate results which lay outside bounds for eight the 62 cases. To date
Standard 140 has not been revised to include these three new versions of the original eight
programs or to include EnergyPlus or to include SBEED or HEED..

Weather Data:
All the eight original reference programs were run using an 8760 hour climate data file called
DRYCOLD.TMY, but since then the weather data formats have been revised, corrected, and
updated to TMY2 and now to TMY3. This file apparently originally used the Denver-Stapleton
Airport, Colorado TMY data.

SBEED could not use the original DRYCOLD.TMY file because it is not in EPW format, so
instead used the Denver-Stapleton Airport, Colorado TMY data in its currently published EPW
format (USA_CO_Denver-Stapleton.724690_TMY) available from the EnergyPlus Weather site.
(https://energyplus.net/weather). Thus weather data used in this current SBEED study may be
slightly revised from what was used in the original eight simulation programs.

EnergyPlus validation analysis was originally done using a BLAST weather file which in turn
had been converted into EnergyPlus format using the EnergyPlus weather converter. Since then,
the DRYCOLD.TMY weather file provided with Standard 140 has been directly converted into
the EnergyPlus format using the EnergyPlus weather converter. This produced significant
changes in results for some test cases using EnergyPlus with both the originally converted
weather file and results with the new weather file.

The EnergyPlus validation study reported that a comparison of the two weather files shows
several differences. First, the BLAST version has Daylight Savings Time option turned on while
the EnergyPlus version of the BESTEST weather file has the Daylight Savings Time option
turned off. This created differences in results for those test cases which have schedules which
change throughout the day, i.e. thermostat setback and nighttime ventilation cases (Cases 640,
650). Secondly, there were differences in the hourly outdoor wet-bulb temperature, sky
temperature, and diffuse and direct solar radiation. These changes are undoubtedly due to
differences in the psychrometric and solar radiation routines between the BLAST and the
EnergyPlus weather conversion programs.

Modeling Issues:
The specifications for Case 220 say that the opaque surface radiation properties should be
applied to all exterior opaque surface solar and infrared absorptances, and infrared emittances.
However the SOLAR5 engine in HEED and in SBEED applies these to the roof only.

Also in ASHRAE standard 140 the thermostat specifications say that heat shall be on when the
indoor temperature is less than 68 degrees, but SBEED turns heat on when interior temperature is
less than or equal to 68 degrees, so this will add a small amount of energy consumption to
Heating Energy.

CONCLUSIONS

SBEED (Small Building Energy Efficient Design) Version 1.0 Build 4 was used to model a
range of buildings as specified in ANSI/ASHRAE Standard 140-2014, Standard Method of Test
for the Evaluation of Building Energy Analysis Computer Programs. The ability of SBEED to
predict heating and cooling loads was tested using a test suite of 54 cases which included
buildings with both low mass and high mass construction, without windows and with windows
on various exposures, with and without exterior window shading, with and without temperature
setback, with and without night ventilation, and with and without free floating space
temperatures. The annual heating and cooling loads predicted by SBEED were compared to
results from eight other whole building energy simulation programs. SBEED was within the
normative range for 18 of the 27 annual heating load cases and within the normative range for 15
of the 27 annual cooling load cases. The nine heating cases that were out of range averaged less
than 6.5% overheating. The twelve cooling cases that were out of range averaged less than
10.4% overcooling. Thus 61% of the test cases were within the normative range, and all cases
that fell outside the range were on the safe side. This means that an actual building analyzed
using SBEED would likely use slightly less heating energy and slightly less cooling energy than
predicted.

Acknowledgements:

SBEED (Small Building Energy Efficient Design) was developed under contract with the
California Energy Commission by Murray Milne and Robin Liggett, Principal Investigators, with
Carlos Francisco Gomez, Senior Research Associate, and Donald Leeper, Senior Systems
Specialist. Testing and Evaluation was by Tim Kohut and Pablo LaRoche plus dozens of
colleagues around the country.

References:

ANSI/ASHRAE Standard 140-2014, Standard Method of Test for the Evaluation of Building
Energy Analysis Computer Programs, American Society of Heating, Refrigerating, and
Air-Conditioning Engineers. Atlanta, GA., 2014
HEED Validation Against the ASHRAE/BESTEST Standard (ASHRAE Standard 140), Grace
Tsai and Murray Milne, UCLA Department of Architecture and Urban Design,
http://www.energy-design-tools.aud.ucla.edu/HEED, 2003
HEED Validated Using HERS Bestest Tier 1 and 2: 2008, Murray Milne, UCLA Department of
Architecture and Urban Design, http://www.energy-design-tools.aud.ucla.edu/HEED,
2008
HEED Validation Reports: HEED 4.0, Build 27 and Build 29, Alicyn Henkhaus, EIT, UCLA
Department of Architecture and Urban Design, October 2012, HEED Summary
Validation Report 2012 includes the following validation reports:
 HEED Validation Against ASHRAE /BESTEST Standard 140, 2012
 HEED Validation Against HERS BESTEST Standard, 2012
 Comparison of HEED and EnergyPlus, 2012
 Validation Results Validation of PV Power Simulation in HEED, 2012
HERS BESTEST 1995. Home Energy Rating System Building Energy Simulation Test., R
Judkoff and J Neymark, National Renewable Laboratory, 1617 Cole Blvd, Golden
Colorado, 80401, 1995
EnergyPlus Testing with Building Thermal Envelope and Fabric Load Tests from
ANSI/ASHRAE Standard 140-2004. U.S. Department of Energy, Energy Efficiency and
Renewable Energy, Office of Building Technologies, 2006
EnergyPlus Testing with Building Thermal Envelope and Fabric Load Tests from
ANSI/ASHRAE Standard 140-2007. U.S. Department of Energy, Energy Efficiency and
Renewable Energy, Office of Building Technologies, 2010

SBEED is available at no cost from http://www.energy-design-tools.aud.ucla.edu/SBEED

Table 1: SBEED Annual Heating Loads Tested Using ASHRAE Standard 140-2014
ASHRAE Note: The statistics in the tables below are based on the Standard 140 Informative example results. These
statistics do not have any substantial importance and are not to be interpreted as acceptance criteria.

Table 1: ANNUAL HEATING LOADS

ASHRAE 140 Example Results SBEED Test
Min Max Mean Min Max
Case MWh/yr MWh/yr MBTU/yr MBTU/yr kBTU/sf MBTU/yr yes/no
600: BASECASE Low Mass Building 4.296 5.709 14.659 19.480 37.200 19.195
610: 600 w/ South Window Overhang 4.355 5.786 14.860 19.743 37.360 19.278
620: 600 w/ East and West Windows 4.613 5.944 15.740 20.282 38.400 19.814
630: 620 w/ E&W Window Overhang+Fins 5.050 6.469 17.231 22.073 40.100 20.692
640: 600 w/ Night Setback Thermostat 2.751 3.803 9.387 12.976 28.070 14.484 no no
650: 600 w/ Night Ventilation 0.000 0.000 0.000 0.000 0.000 0.000
900: BASECASE High Mass Building 1.170 2.041 3.992 6.964 13.230 6.827
910: 900 w/ South Window Overhang 1.575 2.282 5.374 7.787 14.750 7.611
920: 910 w/ East and West Windows 3.313 4.300 11.304 14.672 27.930 14.412
930: 920 w/ E&W Window Overhang+Fins 4.143 5.335 14.136 18.204 33.780 17.430
940: 900 w/ Night Setback Thermostat 0.793 1.411 2.706 4.815 12.610 6.507 no no
950: 900 w/ Night Ventilation 0.000 0.000 0.000 0.000 0.000 0.000
220: 600 w/ Opaque Window Low Mass 6.944 8.787 23.694 29.982 56.420 29.113
230: 220 w/ Infiltration Restored 10.376 12.243 35.404 41.775 79.750 41.151
240: 220 w/ Internal Gains Restored 5.649 7.448 19.275 25.414 47.390 24.453
250: 220 w/ Exterior Shortwave Absorptance 4.751 7.024 16.211 23.967 45.430 23.442
270: 220 w/ Clear South Windows 4.510 5.920 15.389 20.200 40.760 21.032 no
290: 270 w/ South Window Overhang 4.577 5.942 15.617 20.275 40.900 21.104 no
300: 270 w/ East and West Windows 4.761 5.964 16.245 20.350 41.620 21.476 no
310: 300 w/ E&W Window Overhang+Fins 5.221 6.165 17.815 21.036 43.98 22.694 no
320: 270 w/ Thermal Deadband Restored 3.859 5.141 13.167 17.542 34.630 17.869 no
395: 400 w/ No Windows, Low Mass Building 4.799 5.835 16.375 19.910 36.370 18.767
400: Opaque Windows, Low Mass, No Loads 6.900 8.770 23.544 29.924 55.500 28.638
410: 400 w/ Infiltration Restored 8.596 10.506 29.331 35.848 67.140 34.644
420: 410 w/ Internal Gains Restored 7.298 9.151 24.902 31.224 58.010 29.933
430: 420 w/ Extior Shortwave Absorptance 5.429 7.827 18.525 26.707 53.830 27.776 no
800: 430 w/Opaque Windows, High Mass 4.868 7.228 16.610 24.663 51.210 26.424 no

Eight cases are not included in this table because SBEED does not allow changes in the variables
being tested: Solid Opaque Windows, Surface Convection Coefficient, Interior Surface
Radiation, Cavity Albedo, and Added Sunspace.
Table 2: SBEED Annual Sensible Cooling Loads Using ASHRAE Standard 140-2014
ASHRAE Note: The statistics in the tables below are based on the Standard 140 Informative example results. These
statistics do not have any substantial importance and are not to be interpreted as acceptance criteria.

Table 2: ANNUAL SENSIBLE COOLING

LOADS ASHRAE 140 Example Results SBEED Test
Min Max Mean Min Max
Case MWh/yr MWh/yr MBTU/yr MBTU/yr kBTU/sf MBTU/yr yes/no
600: BASECASE Low Mass Building 6.137 7.964 20.940 27.174 58.820 30.351 no
610: 600 w/ South Window Overhang 3.915 5.778 13.359 19.715 45.500 23.478 no
620: 600 w/ East and West Windows 3.417 5.004 11.659 17.074 32.400 16.718
630: 620 w/ E&W Window Overhang+Fins 2.129 3.701 7.264 12.628 20.940 10.805
640: 600 w/ Night Setback Thermostat 5.952 7.811 20.309 26.652 58.820 30.351 no
650: 600 w/ Night Ventilation 4.816 6.545 16.433 22.332 44.250 22.833 no
900: BASECASE High Mass Building 2.132 3.415 7.275 11.652 28.030 14.463 no
910: 900 w/ South Window Overhang 0.821 1.872 2.801 6.388 17.730 9.149 no
920: 910 w/ East and West Windows 1.840 3.092 6.278 10.550 20.180 10.413
930: 920 w/ E&W Window Overhang+Fins 1.039 2.238 3.545 7.636 11.610 5.991
940: 900 w/ Night Setback Thermostat 2.079 3.241 7.094 11.059 28.030 14.463 no
950: 900 w/ Night Ventilation 0.387 0.921 1.320 3.143 2.440 1.259 no
220: 600 w/ Opaque Window Low Mass 0.186 0.835 0.635 2.849 4.670 2.410
230: 220 w/ Infiltration Restored 0.454 1.139 1.549 3.886 6.470 3.339
240: 220 w/ Internal Gains Restored 0.415 1.246 1.416 4.252 7.130 3.679
250: 220 w/ Exterior Shortwave Absorptance 2.177 3.380 7.428 11.533 18.570 9.582
270: 220 w/ Clear South Windows 7.528 10.350 25.687 35.316 72.320 37.317 no
290: 270 w/ South Window Overhang 5.204 8.089 17.757 27.601 58.370 30.119 no
300: 270 w/ East and West Windows 4.302 7.100 14.679 24.226 43.290 22.338
310: 300 w/ E&W Window Overhang+Fins 2.732 5.471 9.322 18.668 28.320 14.613
320: 270 w/ Thermal Deadband Restored 5.061 7.304 17.269 24.922 51.450 26.548 no
395: 400 w/ No Windows, Low Mass Building 0.000 0.016 0.000 0.055 0.060 0.031
400: Opaque Windows, Low Mass, No Loads 0.000 0.061 0.000 0.208 0.240 0.124
410: 400 w/ Infiltration Restored 0.000 0.084 0.000 0.287 0.370 0.191
420: 410 w/ Internal Gains Restored 0.011 0.189 0.038 0.645 0.970 0.501
430: 420 w/ Extior Shortwave Absorptance 0.422 0.875 1.440 2.986 2.710 1.398 no
800: 430 w/Opaque Windows, High Mass 0.055 0.325 0.188 1.109 0.690 0.356

Eight cases are not included in this table because SBEED does not allow changes in the variables
being tested: Solid Opaque Windows, Surface Convection Coefficient, Interior Surface
Radiation, Cavity Albedo, and Added Sunspace.