Protocols for Advanced M & V:

Helping You See Clearly

Webinar
April 30, 2020
Welcome and Acknowledgments

• This webinar is brought to you as part of the “Standardized,

Sustainable and Transparent EM&V – Integrating New
Approaches” project

• Thank You to project funders:

– U.S. DOE State Energy Partner Grant,
– CT Department of Energy and Environmental Protection,
– Lawrence Berkeley National Lab,
– Northeast Energy Efficiency Partnerships, Inc.
– Eversource and Avangrid (UI)
– State Partners: NYSERDA, Vermont Department of Public Services,
New Hampshire Public Utility Commission, Rhode Island Office of
Energy Resources

1
Please note

Audience is muted: Please use Chat Box for questions

We will unmute for Q&A Session at end of webinar and

distribute answers to questions if time is short

The webinar is being recorded

2
Introduction

Why are AM&V protocols and guidance important now? 3

Because

• The industry experience with AM&V is growing

• AM&V is an increasingly relevant tool for states to have
in their mix – e.g. P4P program design, CA NMEC
protocols, as support for time-differentiated savings
impacts, customer engagement and climate goals
• New Efficiency Valuation Organization (EVO- IPMVP) best
practice for AM&V offers global consistency and
credibility
• Proper guidance and protocols will ensure that AM&V is
used appropriately

4
Goals of Webinar

1. Share information on advanced M&V

o Guidance and Protocols
2. Introduce new resources for advanced M&V
o Available and Coming Soon
3. What role do protocols play in deploying advanced
M&V in building analytics?
4. Future directions
o More protocols work needed and Where AM&V is headed

5
Definitions

• Advanced M&V: Large data sets, near real-time, ongoing

feedback, non-linear analytical methods, whole building
meter-based savings calculations, frequent intervals.
(Similar but different from traditional billing analysis and
applicable to portfolios, some programs, and individual
sites, for program implementation and evaluation).
• Protocols: Set of concepts and commonly accepted
conditions ensuring credibility of a product. Technical
details.
• Guidance: How to meet conditions set forth by protocols.
Advice on best practices for applications. Often locally
agreed upon.

6
Agenda and Presenters

• An Evaluation and Portfolio Perspective on AM&V

– Kevin Warren, Warren Engineering
• A Program Planning Perspective: Software Protocols and
AM&V Implementation Guide
– Eliot Crowe, Lawrence Berkeley National Lab
• The Project Perspective: Technical Issues and New EVO
Publications
– Lia Webster, Facility Energy Solutions
• The State Perspective: Developing and Applying Guidance
– Carmen Best, Recurve
• Future Directions!
• Q&A

7
Kevin Warren
Warren Engineering

An Evaluation and Portfolio Perspective on AM&V

Major Use Cases

• ESCOs or EMIS Providers

– Use interval data analysis to prove the savings from a
building tune-up
• Impact Evaluation
– Determine the savings from a utility program after it has
happened or in real time
• Program Implementation and Tracking
– Embedded into the program delivery process

9
Ex-ante 2.0

• Pre/post billing analysis

• Continuous (or at least ongoing)
• All participants
• Embedded in program functions
• Used for more than just savings reconciliation
• Other methods may be used for estimating savings
prior to measure installation (TRM, engineering
calcs)

10
Program Characteristics that Influence
the Ex-ante 2.0 Approach

• Do we care only about savings at the program-level

(or average results for a large number of
participants) or do we care about facility-level
savings?
• Are participants relatively homogenous (residential)
or relatively unique?
• What is the average value of the savings for each
participant?

11
Relevant Prior Protocols

Savings = (Baseline Period Energy – Reporting Period Energy)

+/- Routine Adjustments +/- Non Routine Adjustments

Baseline
Applicable
Population Adjustment Protocols
Sectors
Technique

• UMP Chapter 8
Comparison
Homogenous Residential • SEE Action Impact Evaluation
group
Guide
• IPMVP
• UMP Chapter 16
Heterogeneous Nonresidential NRA
• UMP Chapter 19
• UMP Chapter 24

12
Flavors of Ex-ante 2.0

Ex-ante 2.0 Flavor Treatment of NRAs

Population with Embedded billing analysis with a comparison group

Comparison

Population w/o Embedded billing analysis without a comparison group

Comparison
Embedded Option Embedded billing analysis of all participants while
C attempting to identify and quantify NRAs at high rigor.

Raw Site Level No NRA

13
Do Ex-ante 2.0 Programs Require
Evaluation?
Ex-ante 2.0 Flavor Evaluation Approach

Population with Review analysis, review comparison group

Comparison

Population w/o Comparison group analysis, difference of differences

Comparison

Embedded Option
• Sample (after reviewing CUSUMs and CRRs)
C
• Use Option C for some but not all
• For Opt C, high rigor verify NRAs, missing data, dates
Raw Site Level • Adjust baseline if not existing conditions
• Calculate Realization Rates
• Review reserved savings analysis, site visits, and/or
M&V to answer “Why?”

14
Ex-ante 2.0 Gives Evaluators New Tools

• Early feedback to programs

• Better information on the timing of
savings
• New sampling methods using CRR
and CUSUM
– CRR = Claimed-to-Reserved Ratio
Claimed (ex-ante) / Reserved
– CUSUM Plots

15
Eliot Crowe
Lawrence Berkeley National Lab

A Program Planning Perspective:

Software Protocols and AM&V Implementation Guide
 Advanced M&V Process

Implementation Resource Guide

• Overview
• Tools & methods
• Tool selection
• Data gathering & preparation
• Workflow
• Documentation guidance
• Getting started

17
 M&V Tool Selection

• Many models/tools available

• Proprietary vs. transparent / open source
• Example free tools:
– ECAM
– RMV2.0
– OpenEEMeter
– Universal Translator
– NMECR

18
 M&V Tool Selection

• Need to customize models?

• Can the tool be easily configured to output
baseline model goodness of fit metrics?
• Need for individual building savings and
aggregated?
• Is the tool capable of “batch mode” data
input?
• Need to accommodate continuous meter
data feeds?
• Any desired model inputs beyond weather
and time?
• Has tool been vetted, for example in prior
pilots, third-party testing, or by other
means?
• Preference for additional features?
– Customer dashboard
– Opportunity ID
– Project management features
– Etc.
19
 How do you test an M&V tool?

• Predictive capability: Out-of-sample testing

• Robustness: Use a test dataset covering many buildings
• Trusted: Ensure that test method/results do not allow for ‘gaming’ the test

1. Training period ambient temp 3. Prediction period ambient temp

4….???
2. Training period consumption

20
EVO Advanced M&V Testing Portal

21
 EVO Advanced M&V Testing Portal

Download the test Use Prediction Period

data and create temperature data to Upload prediction
baseline models for generate modeled data and get results
Training Period hourly kWh values

22
 Test Metrics

Model Fitness Predictive Capability

Real (imperfect) data!

23
Did I Pass?

24

http://mvportal.evo-world.org/

24
 M&V Tool Testing

• Good tool = guaranteed results?

25
 Baseline Model Fitness

• R2, target >0.7

R2 .85, CV(RMSE) 7%, NMBE 0%

• CV(RMSE), target <25%

• NMBE, target <0.5%

• (MAPE is another to
consider)
R2 0.33, CV(RMSE) 67%, NMBE 0.27%

26
 Other baseline considerations

• Residuals / scatter chart

• Data management / meters cooling
heating
• Dates
• Weather data

Base load

27
 Savings Tracking and Visualization

Time series of savings

Time series of predicted vs actual energy use

Cumulative savings (CUSUM)

28
Lia Webster
Facility Energy Solutions

The Project Perspective:

Technical Issues and New EVO Publications
• AM&V Perspectives
• M&V and EM&V Protocols & Guidelines
• Technical issues in AM&V
• Areas under development
• Technical resources

30
AM&V Program Perspectives

Project Aggregated Impact

Focused Approach Evaluation
• Commercial & Industrial • Small Commercial & • Sample of Projects
sites Residential
• External factors
• Unique projects • Uniform population
• Program impacts
• Technically rigorous • Generous acceptance
criteria • Realization Rates
• M&V plan
• Control Groups • Ex-post savings
• Non-Routine Adjustments
• Portfolio level savings
• Accurate site-level
savings (Ex-ante)

31
M&V Protocol

IPMVP:
• Used worldwide

• Provides M&V terms & definitions

• Framework to determine savings

• Defines 4 M&V approaches

• Focus on projects

ISO 17741 / 50001:

• Superior Energy Performance (SEP)

• Limited application to AM&V

32
EM&V Protocols

DOE SEE Action:

• Provides terms & definitions

• Focus on programs

• Framework to determine net &

gross savings

• EM&V Approaches:

• M&V Approach (IPMVP Options)

• Deemed Savings

• Large Scale Consumption

Analyses (control groups)
33
EM&V Guidelines

DOE UMP
• Compilation of individual ‘protocols’
• 18 ECM specific, 5 ‘Cross-cutting’
• Focus on evaluation

EPA Guidebook on EM&V

• Compiles guidance from DOE
• Promotes best practices
State-by-State Guidance
• CA Standard Practice Manual
• Technical Reference Manuals
(TRMs)
• Public utilities commissions
• Other

34
M&V Guidelines

Based on IPMVP
• ASHRAE Guideline 14

• M&V Guidelines for Federal Energy

Projects (FEMP)

• Strategic Energy Management (SEM)

MT&R Guidelines

• State and utility guidelines

35
M&V Requirements:
IPMVP

IPMVP
Requirements for “Adherence”:

36
M&V Requirements:
IPMVP
IPMVP
Requirements for “Adherence”:
• Follow procedures & principles

• Detailed M&V plan & report

• Use IPMVP terminology & equations

• Consider Uncertainty in savings

• Operational verification

Option C: Whole Building specific:

• Regression modeling

• Energy data requirements

• Non-Routine adjustments
37
Advanced M&V industry is active!

P4P Utility AM&V Ongoing

Programs Software Research
• IOUs in California (NMEC) • ECAM • Model types

• Seattle City Light • LBNL RMV2.0 • Uncertainty methods

• NYSERDA / Con Ed • Universal Translator (UT3) • Efficacy of aggregated

approaches
• BayREN (CA) • Gridium
• Accuracy of adjustments
• Others… • OpenEEMeter
• NRE automated detection
• NMECR methods
• Others…

38
Current AM&V Programs

39
AM&V Software
Area Feature Free & Open Source Tools
Model Type(s) ECAM
Variables & Inputs Used RMV2.0
Model User Interface
OpenEEMeter
Level of User Adjustments
UT3 M&V Module
Equations of Model(s) Shown
NMECR
Interval Data Accepted
Hourly
Energy Data Used
Daily
Monthly
Level of Automation
Software Used
Overview
Code Language
Open Source
NRE detection
Data Coverage Assessment / Limiting
Tool Features Performance Period Weather Data
Residual Review for Autocorrelation
Model Statistics Provided
Avoided Energy Use 40
Savings Type
Normalized Savings
Key considerations for AM&V

Application Technical
Issues Issues
• Commercial vs. Residential • Variations in M&V tool capabilities
- Building level vs. Population based
• Need for customized models
• Avoided energy use vs. Normalized
savings • Savings uncertainty limitations

• Level of Automation • Model acceptance criteria

- Costs • “Bad” buildings
- Periodic reporting vs. Dashboard • Detecting Non-Routine events
- Need for customized models, QC
• Making Non-Routine adjustments
- Energy data access and cleaning

41
 Variations in M&V Tools

Form of Models:
• Change-point Models
• Time of week and temperature
• Variations

• Tools with multiple types of models

Data Used:
• Inclusion of Holidays
• Use additional variables (e.g., Occupied Sq.
Ft.)
ECAM
• Hourly, Weekly, or Monthly energy intervals

42
 Devil’s in the details…Savings Uncertainty

Fractional Savings Uncertainty Model Goodness of Fit

• From ASHRAE G-14 • Use regression statistics
• Based on concept that savings • Ensure quality modeling
uncertainty decreases over time procedures
• Auto-correlation problems with • Not as transparent as FSU
interval data
• Ensure savings are detectible
• Correction factors
• Under-estimates FSU

43
Mitigate Uncertainty & Maximize Savings

Reduce Saving Uncertainty

• Use stringent model acceptance
criteria
• Screen for non-routine events
• Fall-back M&V Approach:
- “Bad” buildings
- Critical projects

• Avoided energy use

- Less error than normalized savings
- Includes extremes, reflects actual
impacts

44
Mitigate Uncertainty & Maximize Savings

Reduce Saving Uncertainty Increase Program Savings

• Use stringent model acceptance • Accurate models capture lower
criteria levels of savings ~ more savings
reported
• Screen for non-routine events
• Lower % savings allowable ~
• Fall-back M&V Approach: broader range of eligible
- “Bad” buildings projects
- Critical projects • More accurate models ~ better
detection and remediation of
• Avoided energy use
non-routine events
- Less error than normalized savings
- Includes extremes, reflects actual • More accurate savings ~ higher
impacts realization rates

45
Minimize Uncertainty in Baseline Model

46

IPMVP’s Snapshot on AM&V

Screen for Non-Routine Events (NREs)
Baseline Period:
• Increased uncertainty in energy model
Implementation Period:
• Can obscure savings from ECMs
Reporting Period:
• Direct increase or decrease in Avoided energy use, or
• Added uncertainty in reporting period model

SBW Consulting 47
Areas of Ongoing Development
• Open-source software
• Methods for determining savings uncertainty
• Evaluation of aggregated approaches
• Automated detection methods NRE

RMV2.0

48
Technical Resources
Organization Document
BPA 3_BPA_Regression for M&V: Reference Guide
BPA 7_BPA_Verification By Energy Modeling
ASHRAE ASHRAE 14 - 2014
EVO IPMVP Core Concepts
EVO IPMVP Uncertainty Assessment Application Guide
EVO IPMVP’s Snapshot on Advanced Measurement & Verification
NW SEM Collaborative SEM Energy Modeling Method Selection Guide
Connecticut Department of Energy and Environment: Advanced
LBNL Measurement and Verification (M&V) Implementation Resource
Guide
Deep Retrofit Pay for Performance - A How-To-Guide & User
Seattle City Light
Manual for Commercial Customers

IPMVP Application Guide on Advanced M&V Methods and Non-

Routine Adjustments - Coming soon!

49
 Thank you!
[email protected]

50
Carmen Best
Recurve

The State Perspective:

Developing and Applying Guidance
DOE - NEEP Advanced M&V Webinar
April 30, 2020
Carmen Best, Director of Policy & Emerging Markets
53
Advanced History of
NMEC in California

54
Normalized
Metered
Energy
Consumption

Is a Means To
Streamline and Scale EE
to Double
Energy Efficiency in
California “The energy efficiency savings and demand reduction . . .
achieving the targets established pursuant to paragraph
(doubling of EE by 2030) shall be measured taking into
consideration the overall reduction in normalized
metered electricity and natural gas consumption where
these measurement techniques are feasible and cost
effective.”
– SB 350

55
CPUC Response
in the NMEC
Ruling ● The Ruling was modeled after the regulatory
requirements for behavior programs.

● Intended to provide more timely information to

customers, program administrators and regulators on
performance without significant re-casting of savings on
an ex post basis.

● Upfront agreement and review of methods appropriate

for the program and in some cases for the specific project

● Process of advice letter review served as the “case law”

documenting the detailed direction on the proposals
and methods coming from the Commission.

Ruling on High Opportunity Programs and Projects - CPUC 2015

56
Advanced = Enabling Effective
Application of M&V

57
M&V Protocols and standards . . .

IPMVP Option (C)

Professional Reproducible
Guidance Execution

Time of Week
& Temperature
Model

Framework Protocols
(2004) (2006)

. . . enable settlement.
58
Revenue Grade = Transparent, Consistent, Repeatable

• Python CalTRACK Engine

• Standard M&V Calculation Methods
• Open Source Apache 2.0
• Monthly, Daily, and Hourly
• How It Works: https://goo.gl/mhny2s
• Public Stakeholders Empirical Process
• Code Repo: https://goo.gl/qFdW4P
• www.CalTRACK.org

59
 Advanced = Enabling
Flexibility + Accountability

60
The Market is
Complex

Meter-based
Demand Flexibility is
Technology and
Business Model
Agnostic

61
CPUC NMEC Rulebook 2.0 - Overview

Details the Program Level Requirements including:

- M&V plan requirements
- Payments & Incentives
General Requirements include detailed information on:
- Commission roles for review
- Tools, Methods, Analytical Approaches and Calculation Software
- Savings claims
Definitions & Matrix of Approval Requirements

Weinbar: How to Make the New Mandatory NMEC Rulebook Work for Your Program

62
Population-Level NMEC

Programs must meet the Population-level NMEC

regulatory and filing requirements described in this
Program Fit document;

Energy savings determinations are made using an

Meter-Based NMEC approach based on pre and post-intervention
energy usage data observed at the meter

Measurement methods and calculation software are

Pre-Defined &
set before the program starts (and not subsequently
Consistent changed) and apply to all sites in a uniform fashion

For More Detail: CPUC Releases Version 2.0 of the Meter-Based NMEC Rulebook

63
Tools, Methods, Analytical Approaches & Calculation Software

Ex-post ...subject to Commission review of savings measurement methods and estimates,

for purposes of program and/or project-level feedback and for purposes of ex-post
Evaluation: impact evaluation

Savings All analytical methods, including tools, algorithms and software used in savings and
Calculations: incentive or compensation payment calculations, must be made available...

Measurement Savings determinations must be made by comparing at least 12 months of post-

intervention energy consumption to at least 12 months of pre-intervention energy
Period: consumption.

Data, methods and calculations must be made available to the PAs well as the
Transparency: Commission and its impact evaluators.

Documentation ..methods used to calculate savings for NMEC programs must be

documented...such that savings calculations are able to be replicated
and Replicability:

Consistent, Pre- For Population-level NMEC programs, the specific measurement method(s) and
calculation software must be determined before the program begins and applied
Set Method: uniformly...

Proprietary Savings measurement methods and calculation software that is public, and
especially those that are open-source, benefit from a stakeholder vetting process
Methods & that allows experts and practitioners to share their knowledge and use updated
Software: information to inform savings estimates.
64
 Rulebook for Programs and Projects Based on
Normalized Metered Energy Consumption
Version 2.0

Release Date: 7 January, 2020 Population NMEC M&V Plan &

Compliance Checklist

"Proprietary Methods & Software: Savings

measurement methods and calculation software
that is public, and especially those that are open-
source, benefit from a stakeholder vetting process
that allows experts and practitioners to share their
Impact Evaluation Specification
knowledge and use updated information to inform Comparison Group Methods
savings estimates. The Commission has supported
- Matching Approach & Criteria
the development of public, open-source processes to
- Statistical Metrics
develop NMEC methods (e.g. CALTRACK) and
encourages stakeholders to engage in these open-
source initiatives." p. 18

65
 M&V
● Democratized access to impact evaluation results
● More cost effective delivery of demand side strategies with
insights from M&V
● Transparent, consistent understanding of performance
● Incremental improvements in methods identified through
practice
● Scaled investments in energy efficiency and other demand
side strategies

66
 M&V =
Sales and
Marketing

Contractor Project DEMAND

Management Finance
FLEXIBILITY
Aggregators
(programs)

Consumer
Services and
Finance
Products

BUSINESS
MODELS

Savings Comfort Health

67
 Carmen Best
[email protected]
Director of Policy & Emerging Markets

DOE - NEEP Advanced M&V Webinar

April 30, 2020
Future Directions
Future Directions

• What guidelines are needed short term/long term?

• Future directions for AM&V?

70
 THANK YOU!

For more information, contact:

[email protected]
[email protected]
[email protected]
[email protected]
[email protected]

72