Technical Document

Data Recovery Service Guide

November 13, 2015

Data Recovery Service Guide
Tridium, Inc.
3951 Westerre Parkway, Suite 350
Richmond, Virginia 23233
U.S.A.

Confidentiality
The information contained in this document is confidential information of Tridium, Inc., a Delaware corpora-
tion (“Tridium”). Such information and the software described herein, is furnished under a license agreement
and may be used only in accordance with that agreement.
The information contained in this document is provided solely for use by Tridium employees, licensees, and
system owners; and, except as permitted under the below copyright notice, is not to be released to, or re-
produced for, anyone else.
While every effort has been made to assure the accuracy of this document, Tridium is not responsible for
damages of any kind, including without limitation consequential damages, arising from the application of the
information contained herein. Information and specifications published here are current as of the date of this
publication and are subject to change without notice. The latest product specifications can be found by con-
tacting our corporate headquarters, Richmond, Virginia.

Trademark notice
BACnet and ASHRAE are registered trademarks of American Society of Heating, Refrigerating and Air-Con-
ditioning Engineers. Microsoft, Excel, Internet Explorer, Windows, Windows Vista, Windows Server, and SQL
Server are registered trademarks of Microsoft Corporation. Oracle and Java are registered trademarks of
Oracle and/or its affiliates. Mozilla and Firefox are trademarks of the Mozilla Foundation. Echelon, LON, Lon-
Mark, LonTalk, and LonWorks are registered trademarks of Echelon Corporation. Tridium, JACE, Niagara
Framework, NiagaraAX Framework, and Sedona Framework are registered trademarks, and Workbench,
WorkPlaceAX, and AXSupervisor, are trademarks of Tridium Inc. All other product names and services men-
tioned in this publication that is known to be trademarks, registered trademarks, or service marks are the
property of their respective owners.

Copyright and patent notice

This document may be copied by parties who are authorized to distribute Tridium products in connection
with distribution of those products, subject to the contracts that authorize such distribution. It may not oth-
erwise, in whole or in part, be copied, photocopied, reproduced, translated, or reduced to any electronic
medium or machine-readable form without prior written consent from Tridium, Inc.
Copyright © 2015 Tridium, Inc. All rights reserved.
The product(s) described herein may be covered by one or more U.S. or foreign patents of Tridium.
Contents
About this guide .................................................................................................5
Document change log ................................................................................5

Chapter 1 JACE Data Recovery Service ............................................................7

SRAM and/or battery options .....................................................................7
Battery-less JACE scenarios........................................................................8
Battery-less versus battery trade-offs ................................................8
SRAM plus battery scenarios.......................................................................9
Unsuitable station for SRAM support ........................................................10
Unsuitable station example.............................................................10
Second example station - suitable for SRAM support? .....................11
Disabling SRAM support.................................................................11

Chapter 2 Requirements for JACE controller SRAM support ..........................13

Licensing .................................................................................................13
About the DataRecoveryService................................................................13
Data Recovery Service Editor..........................................................14
Data Recovery Service properties ...................................................17
Chapter 3 How SRAM support works in a controller.......................................19
Usage goal...............................................................................................19
SRAM based solution ...............................................................................19
Operation overview..................................................................................20
Recording (normal operation) .........................................................20
Station save effects ........................................................................21
Playback scenario (power lost or reboot) .........................................21
Chapter 4 Diagnostic tools for the DataRecoveryService ...............................23
Station output from the DataRecoveryService ...........................................23
Spy page for the DataRecoveryService......................................................25
Reformatting SRAM .................................................................................25
Reformatting the data recovery storage option in a JACE ................25

November 13, 2015 3

Contents Data Recovery Service Guide

4 November 13, 2015

About this guide
This document provides information about Niagara data recovery services.

Document change log

Updates (changes/additions) to this document are listed below.
• Updated November 13, 2015
Many minor changes throughout the document.
• Updated November 3, 2015
Updates include additional information to reflect Niagara 4.1 and JACE-8000 product release.
• Initial Niagara 4 release document: August 18, 2015

November 13, 2015 5

 Data Recovery Service Guide

6 November 13, 2015

Chapter 1 JACE Data Recovery Service
Topics covered in this chapter
♦ SRAM and/or battery options
♦ Battery-less JACE scenarios
♦ SRAM plus battery scenarios
♦ Unsuitable station for SRAM support

Niagara 4 includes support for “battery-less” JACE operation, where a JACE uses capacitor-charged SRAM
(static random access memory), to preserve RAM-resident data when a power outage occurs. This includes
station data not yet committed to non-volatile flash memory.
• Initially, this applied only to a QNX-based controller with an installed SRAM option card. Previously, such
controllers were always considered “battery-less”, where any installed backup battery was removed
when the option card was installed. However, with later releases, including Niagara 4, a JACE controller
can use both SRAM and a backup battery.
• The QNX-based JACE models have onboard SRAM as standard—no option card required. This includes
the JACE-8000 series controller, plus the JACE-3E, and NPM6E processor-based series (JACE-6E and
“retrofit board” JACE-603 and JACE-645 controllers). The JACE-8000 controllers ship “battery-less”.
The JACE-6E and JACE-3E controllers ship “battery-less” as well however, you can optionally install a
NiMH backup battery (identical to the one in JACE-6 series controllers).
• N4.1 supports the JACE-8000 controller model. AX-3.7 and AX-3.8 support all the controller models (ex-
cept the JACE-8000) and also all controllers with an installed SRAM option card. For JACE-3E support,
build AX-3.7.105 or later is required.
N O T E : The Security JACE platform does not include SRAM, nor does it support the SRAM option card/
DataRecoveryService in any build of NiagaraAX.
For hardware mounting details, refer the installation document that ships with each SRAM option card or
SRAM-equipped controller. This document summarizes usage scenarios and software operation details of
the SRAM memory feature.

SRAM and/or battery options

With any SRAM-equipped controller, including the “onboard SRAM” JACE controllers like the JACE-8000,
JACE-6E, and JACE-3E, there are two different ways to utilize SRAM. There is also a scenario in which you
may elect not to make use of SRAM, even if the controller includes it standard (onboard).

November 13, 2015 7

Chapter 1 JACE Data Recovery Service Data Recovery Service Guide

Battery-less JACE (with SRAM) JACE with SRAM plus backup battery JACE using only backup battery

JACE-8000/JACE-6E/JACE-3E (SRAM card JACE-6E/JACE-3E (SRAM card JACE-6E/JACE-3E or JACE-6 series

unnecessary) unnecessary)

This is the “battery-less” controller configu- Since Niagara 3.6.44 or later, an SRAM- Sometimes a station is a poor candidate for
ration for the Niagara implementation of equipped JACE can utilize both SRAM and SRAM support. Disabling SRAM operation
SRAM. No backup battery (NiMH or other- an installed backup battery, like the NiMH is best in this case, even if “onboard”
wise) is attached to the JACE. battery pack shown above. SRAM.
This configuration also applies to the JACE-
8000 controllers (not shown), which do not
support battery backup.

Battery-less JACE scenarios

A standard JACE-8000, JACE-6E or JACE-3E controller, or any SRAM-equipped JACE (via SRAM option
card), when installed without an integral, rechargeable battery pack (or external 12V battery), provides a
“battery-less” installation. This can offers advantages in certain situations. Some example scenarios are:
• Installation of any battery-equipped device is forbidden, due to job site or local regulations.
• Installation without a battery may allow a higher temperature environment rating, such as a JACE-6E or
JACE-3E (60 °C maximum, vs. 50 °C maximum if with a rechargeable NiMH battery pack).
• Over time, the periodic replacement of the NiMH battery in the JACE presents too many costly ob-
stacles. For example, a JACE may be installed in a difficult-to-reach location. And as the NiMH battery’s
condition declines, “battery bad” alarm notifications are received.
You can meet such issues by installing a standard JACE-8000, JACE-6E or JACE-3E controller, or if another
earlier model (e.g. JACE-6,-7), by removing its NiMH battery pack and installing the SRAM option card. This
makes the unit “battery-less”. As a consequence of SRAM-only backup support, JACE battery monitoring
no longer occurs—no more “battery bad” alarm notifications.
N O T E : A station running in a battery-less JACE has no seamless immunity to “power bumps”. Although all
station data, including components, histories, and alarms, are automatically restored to “pre-event” values
as part of station startup (following power restoration), the briefest power outage results in a controller re-
boot. For more details, see “Battery-less versus battery trade-offs”.
Note that a “battery-less” configuration for an SRAM-equipped controller is now one of two possible config-
urations where SRAM is used. For more details, see “SRAM plus battery scenarios”.

Ba t t er y- le ss ver s us b at t e r y t r ad e- offs
Any JACE controller with a charged backup battery holds a key advantage over a battery-less SRAM-
equipped JACE, in this regard:
• Station operation continues (uninterrupted) across very short power outages, that is “power bumps” last-
ing only a few seconds—without initiating an orderly shutdown.
[A UPS could be used to mitigate this, but this would re-introduce battery maintenance for the UPS.]

8 November 13, 2015

Data Recovery Service Guide Chapter 1 JACE Data Recovery Service

The NiMH battery provides enough power for this, immediately recharging when power is restored. If a
power outage lasts longer than its defined “shutdown delay” time, the NiMH battery allows sufficient time
for the JACE to perform an “orderly shutdown”, including saving the station’s database (config.bog) and all
recorded alarm and history records.
However, a JACE without SRAM but with a weak NiMH battery is exposed to a different power outage issue
—where the potential exists for data loss (since the last station save), due to insufficient battery power to
complete an orderly shutdown. Potentially, this could also occur if enough power outages occur in rapid suc-
cession—draining the controller’s battery to a low level.
A battery-less JACE solves that problem, as all station-generated data (changed from that stored in its non-
volatile flash memory at the time of power loss) is always preserved in SRAM. Upon power restoration, this
data is “played back” in the station during startup, then saved in its non-volatile flash memory.

S R A M d o e s n o t p re s e r v e d a t a o r f i l e s e x t e r n a l t o s t a t i o n
Please note that if a JACE power event occurs when station users have unsaved file changes, say in a Px file
or Nav file being edited, those unsaved changes are lost. This behavior may seem different from a battery-
equipped JACE entering an ordered shutdown—but it is not.
The practical difference is that a battery-equipped JACE may keep running over a short “power bump”. Sta-
tion users may be aware of such an event, and react by saving changes (click S a v e button in the active view).
Providing that communications are still established, the file edited may be saved. Or, power may be lost only
momentarily, and then remain stable until the user does a normal save.
N O T E : A battery-less SRAM-equipped JACE does not provide a similar save opportunity after a power
bump—it is already busy rebooting. Therefore, as a best practice, you should advise system users of bat-
tery-less SRAM-equipped JACEs to save often when editing items like Px graphics and Nav files.

SRAM plus battery scenarios

An SRAM-equipped controller can utilize both its SRAM and an installed backup battery, such as a JACE-6E
or JACE-3E controller with an optional internal NiMH battery pack, or a controller (with SRAM option card)
and its standard NiMH battery pack. This also applies to a QNX-based JACE controller with an external 12V
sealed lead-acid battery, either as its main backup battery (JACE-603, JACE-645) or as a battery in “parallel”
with its NiMH battery (JACE-7 ).
N O T E : This topic covers only those controller models that support battery backup.
The hybrid “SRAM plus battery” configuration offers:
• Immunity to power bumps, thanks to the backup battery—similar to a unit with backup battery only.
• A longer allowable “Shutdown Delay” time, up to 10 minutes for a unit with only a NiMH backup battery.
This is beyond the 30 seconds (recently 1 minute) maximum period. After continuous operation this long
on battery power, at shutdown the JACE saves its database and powers off. So, a unit with a reasonably
healthy NiMH backup battery could continue running over a power outage of up to 10 minutes. Note that
the station’s “DataRecoveryService” must be enabled to specify this longer shutdown delay in station’s
“PowerMonitorService”. If the unit is equipped with a sealed lead-acid (SLA) battery, this shutdown delay
can be specified up to 15 minutes.
• Data independence from backup battery condition, again a consequence of the ongoing SRAM opera-
tion. If the charge on the backup battery weakened such that the controller was unable to complete a da-
tabase save on a controlled shutdown, or even if immunity to power bumps was lost, data integrity still
remains. Upon reboot from a power restoration, the station’s DataRecoveryService restores (replays) the
previously recorded station runtime data from SRAM.
N O T E : A discharged battery could happen if multiple, consecutive power outages occurred, draining
the charge on the rechargeable backup battery. A longer “Shutdown Delay” time could contribute to
this. However, unlike with a controller not using SRAM, station runtime data is safe in this case.
Gain the features above by installing an optional NiMH battery pack in a standard JACE-6E or JACE-3E, or if
another earlier model, by retaining its NiMH battery pack when installing the SRAM option card. By default,

November 13, 2015 9

Chapter 1 JACE Data Recovery Service Data Recovery Service Guide

this hybrid configuration is standard with a “retrofit board” JACE-603 or JACE-645, assuming the control-
ler’s 12V SLA battery (in the controller’s enclosure) is retained.
Note in this hybrid configuration, JACE monitoring of backup battery(ies) continues—so “battery bad”
alarm notifications are still possible, and the regular replacement of backup batteries is still needed.
Again, the “SRAM plus battery” configuration for an SRAM-equipped controller is one of two possible con-
figurations where SRAM is used: the other is for a JACE controller to be “battery-less”.
N O T E : Although this hybrid configuration is typically the most desirable, note that some stations may be
poor candidates for SRAM operation, with SRAM support even counter-productive. In this case, a controller
platform that includes “built-in” SRAM (such as a JACE-6E or JACE-3E) can be configured to disable its Data-
RecoveryService, and use only its (optional) installed backup battery. Or, an SRAM option card can be re-
moved from another JACE controller, such that it also uses only its installed backup battery.

Unsuitable station for SRAM support

The DataRecoveryService writes current values as they occur to a block of SRAM. When a block is full, the
service copies it from SRAM to the controller’s flash memory. A station that creates rapid COV (change of
value) histories may fill the SRAM data blocks too frequently, triggering a database save possibly every cou-
ple of minutes. Ideally, such database saves to flash memory should occur no more than once an hour.
Saving the database too frequently results in inefficient use of controller CPU time and potential flash prob-
lems. Flash memory is designed to be written to a certain number of times. A number of variables contribute
to how often the database needs to be saved, including:
• Rate of changes that need to be persisited
• Size of the changes (histories, alarms, and setpoint changes differ in size)
• Amount of free flash memory space
Two examples use only the number of changes that occur per hour to demonstrate the need to consider the
COV rate when selecting station backup options:
• A JACE-6E station with 176 COV histories. Each writes every 5 seconds. In addition, the station has other
NRIO and BACnet components that add to the histories. 176 COV histories changing every 5 seconds
yields 126,720 changes per hour (176 * 12/min * 60 min/hr). Automatic database saves occur every 6 mi-
nutes. This number of changes is well beyond the recommendation for a controller using only SRAM and
flash memory to backup station data.
• A second JACE-6E station has 1200 interval histories, each with a collection interval of 10 to 15 minutes.
If all histories change every 10 minutes (worst case scenario) the result comes to 7,200 changes per hour
(1200 * 6 changes/hr). Now compare this number of changes to those in the first example: 126,720/
7,200. In the first example, changes occur nearly 18 times more often than in the second. Applying this
ratio to the second example, an automatic database save occurs approximately every 1.5 hours (6 mi-
nutes * 17.6 = 105 minutes), well within the acceptable range.
Applications that require frequent writes to flash memory should use a controller with a backup battery and
disable the use of internal SRAM/flash memory.

Unsuitable station example

An example test station running in a JACE-202 Express was found to fit this unsuitable category. This station
had 176 COV histories, each writing every 5 seconds, along with other NRIO components and BACnet com-
ponents (making little additional history usage). SRAM support in this station resulted in an ongoing auto-
matic save approximately every 6 minutes—about a magnitude away from ideal, which is no more than once
per hour.
176 COV histories changing every 5 seconds yields 126,720 changes/hour (176 * 12/min * 60 min/hr).

10 November 13, 2015

Data Recovery Service Guide Chapter 1 JACE Data Recovery Service

Second example station - suitable for SRAM support?

Now consider the suitability of a station with 1200 interval histories, each with a collection interval of 10 to
15 minutes. Planning for the “biggest load” configuration, that is all histories at 10 minute intervals:
1200 histories changing every 10 minutes yields 7,200 changes/hour (1200 * 6/hr).
Note this example station results in a rate of history changes nearly 18 times less than the first (Unsuitable
station example), where the ratio of these two “total changes per hour” is 17.6 (126,720/7,200).
Let’s assume everything else is equal between this station and the first (unsuitable) example, including the
same amount of free flash space. Then in this case, ongoing automatic database saves would occur approxi-
mately every 1.5 hours (6 minutes * 17.6, or 105 minutes)—well inside the “ideal range”.
N O T E : A number of different variables factor into the actual operation of SRAM support, including:
• Rate of changes that need to be persisted.
• Types of changes (histories, alarms, and setpoint changes all have different sizes).
• Amount of free flash disk space on the controller.
Therefore, the examples above do not reflect a full range of different scenarios.
Also see the following related sections:
• The next section, “Disabling SRAM support.”
• For detailed background information on the operation of SRAM support, see “How SRAM support in a
JACE works”.
• For tools to monitor operation, see Diagnostic tools for the DataRecoveryService”.

Disabling SRAM support

N O T E : This topic covers only those controller models that support battery backup.
In situations where a station is not compatible with SRAM support, you can equip its host JACE controller
with a suitable backup battery and disable SRAM operation. Depending on the controller series, do this in
one of several ways:
• If a JACE-6E or JACE-3E, install the optional NiMH battery pack. In its station’s D a t a R e c o v e r y S e r v i c e ,
set the “Service Enabled” property to false. In the station’s P l a t f o r m S e r v i c e s container, make sure the
“Battery Present” property is true.
• If a JACE platform using an SRAM option card, remove that option card and install its NiMH battery pack.
Then using the platform S o f t w a r e M a n a g e r, uninstall the platDataRecovery module to remove the
D a t a R e c o v e r y S e r v i c e from the station. Again, in the station’s P l a t f o r m S e r v i c e s container, make sure
the “Battery Present” property is true.

November 13, 2015 11

Chapter 1 JACE Data Recovery Service Data Recovery Service Guide

12 November 13, 2015

C h a p t e r 2 R e q u i re m e n t s f o r J A C E
co ntrolle r S RAM s up po rt
Topics covered in this chapter
♦ Licensing
♦ About the DataRecoveryService

The following is required for SRAM support in a JACE controller, using either “onboard” SRAM or the SRAM
option card, along with the platform DataRecoveryService:
• A JACE-8000, JACE-6E, or JACE-3E controller with onboard SRAM, or
A JACE-603 or JACE-645 with onboard SRAM, licensed to run a NiagaraAX station, or
A JACE-2,-6,-7 controller running AX-3.6 or later, with an available option card slot, and
– An SRAM option card, installed in that option card slot.
– The “platDataRecovery” module must be installed in the JACE
(this module is automatically installed in platforms with "onboard" SRAM).
• The JACE must also be licensed for this feature—see “Licensing”.

Licensing
The JACE needs a license entry as shown here, to support the SRAM option card and DataRecoveryService:
<feature name="dataRecovery" expiration="2030-12-31"/>
JACE controllers with onboard SRAM should have the dataRecovery license feature by default. For JACEs
with SRAM option cards, ensure the feature is present in the license. If the license feature is missing, SRAM
will not work - either the DataRecoveryService in its station will be missing or will otherwise remain in a fault
condition.

About the DataRecoveryService

The DataRecoveryService is the station platform service that provides SRAM support for any JACE equipped
with SRAM. Providing the platDataRecovery module is installed, this service automatically appears under
PlatformServices.

November 13, 2015 13

Chapter 2 Requirements for JACE controller SRAM support Data Recovery Service Guide

NOTE:
• If the JACE is not properly licensed, this service remains in fault.
• This service includes a “Service Enabled” configuration property. The DataRecoveryService does not re-
place the PowerMonitorService.
N O T E : Starting in N4.0, the D a t a R e c o v e r y S e r v i c e E d i t o r view includes additional configurable fields.
For example, S t a t i o n S a v e L i m i t , S t a t i o n S a v e L i m i t P e r i o d , G e n e r a t e A l e r t o n R e p l a y and P l a t f o r m
Alarm Support.
Fi gu re 1 Data Recovery Service Editor in PlatformServices of SRAM-equipped JACE

The figure above shows the default view (for an AX release) for the service: the D a t a R e c o v e r y S e r v i c e
E d i t o r.
Note the example above reflects a scenario where a station save has occurred since the service was created.
Some SRAM “data recovery blocks” have already been flushed to flash (“Persistent Storage Size” is not 0.00
KB).

Data Recovery Service Editor

This Data Recovery Service Editor is the default view of the Data Recovery Service.
The Data Recovery Service Editor view has the following three main areas:
• Data Recovery Settings
• Blocks Configuration
• Data Recovery Blocks

14 November 13, 2015

Data Recovery Service Guide Chapter 2 Requirements for JACE controller SRAM support

Data Recovery settings

Include the following:
• Service Enabled
Defaults to true, to enable SRAM support via this service. If a JACE with onboard SRAM (e.g. JACE-6E
or JACE-3E), you can set this to false to disable SRAM support—in which case it relies on its installed
backup battery and its P o w e r M o n i t o r S e r v i c e to preserve station data upon loss of power. (If an SRAM
option card-equipped JACE, you can simply remove the SRAM option card and uninstall the platDa-
taRecovery software module).
• Service Status
The current status of the DataRecoveryService, which is typically “Ready”. Other states include “Start-
ing”, “Configuring”, “Replaying”, “Saving”, “Stopping”, “Stopped”, “Fault” and “Unknown”.
• Last Station Save Time
Reflects the last time a station save occurred (config.bog written to flash memory). This save may (or may
not) have occurred as a result of the DataRecoveryService.
• Last Station Save Successful
Boolean that reflects if last station save attempt was successful, as either “true” or “false”.
This save may (or may not) have occurred as a result of the DataRecoveryService.
Note in the case of a newly-created DataRecoveryService, this is “false” until the next save occurs.
• Station Save Limit
Configurable in N4.0 and later. Number of station saves that are allowed to occur during the Station Save
Limit Period before it is determined that the station is spending too much time saving. Exceeding the lim-
it throws the Data Recovery Service into fault since too much data is being generated.
• Station Save Limit Period
Configurable in N4.0 and later. The period of time for Station Save Limit. If enough saves occur during
Station Save Limit Period to exceed the Station Save Limit then the service goes into fault. For example,
more than 5 station saves in 3 minutes period triggers a fault.
• Persistent Storage Size
Reflects the total size of all the “flushed to flash” data block files (“.drdb” files) that exist in the station’s
/dataRecovery folder, in KB. Initially, this will be 0, until the first SRAM block flushes to flash. It will then
increment by that KB amount for each subsequent SRAM block flushed.
Note this value is continually compared to the “Persistent Capacity” property in the Blocks Configuration
property section.
• Generate Alert On Replay
Configurable in N4.0 and later. Boolean (true/false) value, generates an alert (low priority alarm type) that
will indicate that a Data Recovery Replay (power was lost) occurred. This is a persistent artifact that will
show up in the alarm console, since it can be useful to know when power loss occurred.
Default is false. If set to true, upon any controller boot sequence in which SRAM recorded data is dis-
covered and played back, a corresponding alert is routed to the Alarm Class named in the D a t a R e c o v -
e r y A l a r m S u p p o r t container. The following figure shows details for such an example alert.

November 13, 2015 15

Chapter 2 Requirements for JACE controller SRAM support Data Recovery Service Guide

Fi gu re 2 Example Alarm Record details for an alert generated by DataRecoveryService

• Data Recovery Alarm Support

Configurable in N4.0 and later. This is the standard container slot for routing platform service-generated
alarms or alerts, in this case an alert from the D a t a R e c o v e r y S e r v i c e upon any controller boot sequence
in which SRAM recorded data is discovered and played back. These properties work in the same fashion
as those in an alarm extension for any control point.

Blocks Configuration
These status properties include the following:
• Total Size
Reflects, in bytes, the total amount of SRAM buffer memory available to the service. For example, this is
“524288” for the 512 KB SRAM option card.
N O T E : For JACE-8000 controllers this is “262144” bytes for the 256 KB SRAM memory.
• Number of Data Recovery Blocks
Reflects the number of data block partitions of SRAM used, for example, 3.
• Active Directory
Reflects the directory used in SRAM for the active data block.
• Persistent Directory
Reflects the full flash file directory path used to store flushed “.drdb” files,
which equates to: /dataRecovery
• Full Policy
Reflects the current policy when an SRAM data block becomes full (currently “Flush”).
• Persistent Capacity
Reflects the size limit, in KB, for the total of all “flushed to flash” data block files (“.drdb” files). If this limit
is exceeded (see property “Persistent Storage Size”), the service automatically triggers a station save op-
eration. For related details see “Station save effects”.

16 November 13, 2015

Data Recovery Service Guide Chapter 2 Requirements for JACE controller SRAM support

Data Recovery Blocks

This area provides expandable bar graphs for each of the SRAM buffer data blocks, to visually represent the
current amount of used space, overhead space, and available free space, along with numerical values.
By default, the currently active SRAM block is expanded, showing a bar graph of current buffer usage.

F ig ure 3 Example SRAM data block, show both states “Active” and “Flushing”

The image above shows an example active SRAM block first near full, then “flushing” momentarily to flash—
an operation that lasts only a second or two. Another SRAM block becomes active when this happens, and it
is used until it fills and needs to flush to flash.
Above the bar graph of each block, its S t a t u s is described, typically as either: “Active”, “Idle”, or sometimes
“Flushing”, with other states “Purging”, “Awaiting Idle”, “Flush Queued”, “Defragmenting”, “Reserved”,
“Fail”, and “Unknown”.
Below the bar graph of each block, numerical amounts display, in bytes, for its total Capacity, currently Used
space, calculated Overhead Space, and available Free Space.

Da ta R eco very Se rvi ce p rop erti es

In addition to the (default) Data Recovery Service Editor view, the Data Recovery Service also has properties
on its P l a t f o r m S e r v i c e P ro p e r t i e s view, many of which are shown here.

November 13, 2015 17

Chapter 2 Requirements for JACE controller SRAM support Data Recovery Service Guide

Fi gu re 4 JACE-8000 Platform Service Properties view of DataRecoveryService

Most of these properties are also on the D a t a R e c o v e r y S e r v i c e E d i t o r default view.

This view alsoincludes the following read-only informational fields (not pictured above):
• Too Many Saves — Boolean (true/false). Default is false.
• Station Save Limit (editable on the Recovery Service Editor view) -
• Station Save Limit period (editable on the Recovery Service Editor view)

18 November 13, 2015

Chapter 3 How SRAM support works in
a co ntroll er
Topics covered in this chapter
♦ Usage goal
♦ SRAM based solution
♦ Operation overview

This section explains how SRAM support in functions in a controller.

The following sections are included under this topic:

Usage goal
When recovering from a power outage (where a “controlled shutdown” did not occur), the goal is to retain
all runtime station data in volatile DRAM (Dynamic RAM) that had changed prior to power lost, that is, since
the last station save to the JACE’s NVRAM, also known as “flash” memory. Changes apply to all normally
persisted station data, including any changes to components, histories, and alarms.
Simply increasing the flash write frequency (for example upon each value change) is not viable because of
eventual damage to flash memory components. Additionally, the latency of flash writes is significant; such
writes may not complete if power is lost.
Replacing all flash memory on a JACE with SRAM is cost prohibitive; however, a memory caching scheme us-
ing SRAM was developed. See the section “SRAM based solution”.

SRAM based solution

The patent-pending SRAM support feature employs both hardware and software, as follows:
• Hardware: SRAM buffer with dedicated capacitor for memory refresh (backup). JACE-8000 has “on-
board” 256 KB of SRAM, while JACE-3E, and NPM6E-based controllers (JACE-6E, JACE-603, JACE-645)
have “onboard” 512 KB of SRAM, backup capacitor plus supporting circuitry. The SRAM option card pro-
vides all of this on an installable option card, for JACE-6,-7 series controllers.
• Software: The JACE-8000 requires N4.1, the dataRecovery license, and the platDataRecovery-
module. The other JACEs require AX-3.6 or later, the dataRecovery license, and the platDataRecov-
ery module. The module works with the SRAM buffer to continuously save changes as “deltas”.
Essentially, this feature automatically “records” all changes during runtime, for “playback” upon station
startup after a power outage.
Providing that the JACE is properly licensed, no other configuration is required for this combination to pro-
vide SRAM support. This feature automatically works for a JACE-8000, JACE-6E, or JACE-3E, as well as for
other platforms with the SRAM option card (providing the platDataRecovery software module is installed).
An interface “window” is automatically provided in the station running on the JACE, via the dynamically cre-
ated “DataRecoveryService” under its PlatformServices container. A special view on this platform service al-
lows verification of operation, and if ever necessary, a means of troubleshooting.
N O T E : The DataRecoveryService includes the “Service Enabled” configuration property, which is true by de-
fault. The use case is for a JACE-3E or NPM6E-based controller (e.g. JACE-603) in which you want to disable
SRAM support, using only its installed backup battery instead.

November 13, 2015 19

Chapter 3 How SRAM support works in a controller Data Recovery Service Guide

Operation overview
Providing the JACE controller is equipped with SRAM and is properly licensed, the platDataRecovery mod-
ule dynamically creates a “DataRecoveryService” in the running station’s PlatformServices container. This
service presides over the SRAM option card, which in turn acts as a buffer for flash memory.
The new service partitions the SRAM into multiple buffers, or data “blocks”. In the initial usage there are 3
blocks. Only one these blocks is ever “active” at any time.
Operation can be described in the following modes:
• Recording (normal operation)
• Station save effects
• Playback scenario (power lost or reboot)

Re cord in g ( norm al op erati on)

Immediately after a station has started, any subsequent change in any of its three object spaces (component,
alarm, history) is immediately “recorded” by the DataRecoveryService, writing to the currently active SRAM
block. In bytes, each block has a “total capacity”, a calculated amount of needed “overhead”, and the cur-
rent available “free” area. Exceptions to this recording are noted below.
NOTE:
• Unsaved user edits to files, e.g. Nav or Px files (working in the Px Editor) are not recorded, and thus are
lost upon any power event if running battery-less. See “SRAM does not preserve data or files external to
station”.
• Slots of objects that have the “Non-Critical” config flag set are also not recorded. Note by default, for
most slots, this new slot config flag (AX-3.6 and later) is cleared. See “Non-Critical config flag”.
When the active SRAM block becomes full, i.e. its “free” area is not large enough for a record write, its data
is written out (“flushed”) to a file in the JACE’s flash memory, then that SRAM block is cleared. Concurrently,
while this transfer/flush occurs, another SRAM block is activated and used. This multi-threaded buffer meth-
od enables recording station changes that would otherwise be lost or blocked.
Over time, many SRAM blocks may fill, where each results in another file in flash on the JACE. These files are
automatically named in a numeric sequence with a “.drdb” extension (data recovery database), stored in the
station’s file space under the \dataRecovery directory. For example, this folder may contain files “1.drdb”,
“2.drdb”, “3.drdb”, and so on, with “1.drdb” being the oldest. This sequence becomes important later,
when the DataRecoveryService restores (plays back) data.
At some point, the collective size of the station’s “.drdb” files may exceed the “persistent capacity” limit
used by the DataRecoveryService. This is a “persistent space full” condition. If this occurs, an automatic sta-
tion save occurs. Effects on the DataRecoveryService are the same as if a station save is manually issued, or
if an automatic save from the platform “auto-save” frequency occurs. See “Station save effects”.

Non-Critical config flag

There may be cases where new slots or the data values of those slots is not considered critical, and does not
need to be saved by the DataRecoveryService. To accommodate for this, a new “Non-Critical” config flag
can be set on any slot (AX-3.6 and later). By default, for most slots, this flag is cleared meaning that it is not
set.
N O T E : When this flag is set on newly-created slots, the value of such new slots is not saved by the DataRe-
coveryService. If running “battery-less” and power is lost before the station is saved, these values will be
permanently lost.
However, if power is not lost, then the slots are still saved as part of a routine station save. But when the val-
ues of the slots change, those changes are not recorded by the DataRecoveryService; they are only saved
when a “station save” occurs. Not saving this non-critical data through the DataRecoveryService may offer
some performance advantages.

20 November 13, 2015

Data Recovery Service Guide Chapter 3 How SRAM support works in a controller

St a t ion s av e effec t s
When saving a station running the DataRecoveryService, the following things occur. Note this applies re-
gardless of how the save was issued—for example, a manually invoked command (S Sa v e S t a t i o n ), or an auto-
matically issued save, e.g. a “persistent space full” condition by the DataRecoveryService, or a station save
that occurs as part of a station copy operation, reboot command, or from a “controlled shutdown” while run-
ning on backup battery power:
1. The normal station save method is used to capture all changes to all the station’s object spaces (compo-
nents, alarm, history) to flash memory, saved as the files config.bog, alarm.zip and history.zip.
2. Upon a successful save, all “.drdb” files holding buffered data are erased from flash, as these are no lon-
ger necessary.
3. The active SRAM data block is also cleared; however, one block operates in a “reserved state”. This is
needed to capture any changes that may occur while the station is in the “saving” state. Otherwise, data
loss could potentially occur if power was lost during or immediately after a save.
Again, note that buffered data in flash and SRAM is not erased until the save (config.bog) is successfully writ-
ten to flash. Thus, data recovery records are not lost if a power loss occurs during a save.

Pl ayb a ck s c enari o (p owe r l os t o r reb oo t)

• If power is lost, a battery-less JACE is immediately off, with the last active SRAM block preserved by ca-
pacitor charge, plus typically one or more flash-preserved “.drdb” files. This could also occur if battery-
equipped controller was unable to successfully complete an orderly shutdown. The JACE remains in this
dormant state until power is restored, when the JACE boot process begins normally.
• If a reboot command is given, the last active SRAM block is present, but typically no flash-preserved “.
drdb” files—because a reboot typically1 first issues an “orderly shutdown”, starting with at station save.
See “Station save effects” for details. The JACE boot process begins normally.
During station startup, after loading into DRAM the station’s last saved database (config.bog), the DataRe-
coveryService starts up and detects that recovered data exists in the preserved SRAM block, and also possi-
bly in flash-preserved “.drdb” files.
The DataRecoveryService then performs the following actions:
1. It replays (writes to DRAM) all the flash files in sequence, beginning with the oldest.
2. Once flash records have been replayed, the service replays the records found in the SRAM block, as they
are the most recent.
Note that in some cases this playback to DRAM can take a few minutes—this depends on how much data
there is to replay. Station startup then completes, and an immediate save of the DRAM-restored station to
flash (config.bog) occurs. All previous “.drdb” files are erased in flash, as well as SRAM data blocks, and Da-
taRecoveryService operation begins anew. See “Recording (normal operation)” for details.

1.
A reboot issued from system shell is one exception—no orderly shutdown occurs before such a reboot. There may be
.drdb files saved when this reboot happens. If so, these are “replayed” the same as if power is lost.

November 13, 2015 21

Chapter 3 How SRAM support works in a controller Data Recovery Service Guide

22 November 13, 2015

Chapter 4 Diagnostic tools for the
DataRecoveryService
Topics covered in this chapter
♦ Station output from the DataRecoveryService
♦ Spy page for the DataRecoveryService
♦ Reformatting SRAM

The DataRecoveryService provides the following diagnostic tools that may be useful if troubleshooting.
Also included here, a procedure for reformatting the SRAM option card (if necessary).

St ati on o utp u t f rom th e D at aR ec o ver ySe rv ic e

When left at the default “message” log level, the DataRecoveryService produces minimal station output re-
lated to operation. Primarily, related messages are seen at station startup, especially if following a power
loss event or reboot command.
MESSAGE [11:18:15 13-Aug-10 EDT][sys.registry] Up-to-date [250ms]

MESSAGE [11:18:20 13-Aug-10 EDT][sys.registry] Loaded [2935ms]

MESSAGE [11:18:29 13-Aug-10 EDT][sys] Baja runtime booted ("/ffs0/niagara")

MESSAGE [11:18:29 13-Aug-10 EDT][sys] Loading "/ffs0/niagara/stations/J6_TestW/

config.bog"...

MESSAGE [11:19:34 13-Aug-10 EDT][sys] Loaded (64809ms)

MESSAGE [11:19:56 13-Aug-10 EDT][alarm.database] Loading...

MESSAGE [11:19:58 13-Aug-10 EDT][alarm.database] Loaded [2196ms, 32 alarms, 104 pages]

WARNING [11:19:58 13-Aug-10 EDT][platDataRecovery.service] data recovery detected,

replaying...

MESSAGE [11:20:11 13-Aug-10 EDT][sys] DataRecoveryService restoration check complete

(18368ms)

MESSAGE [11:20:12 13-Aug-10 EDT][sys] Services Initialized (1010ms)

MESSAGE [11:20:12 13-Aug-10 EDT][sys.mixin] Updated [112ms]

MESSAGE [11:20:14 13-Aug-10 EDT][history.db] Starting async warmup of history config

index...

MESSAGE [11:20:14 13-Aug-10 EDT][history.db] Async history config index warmup

completed in 342 ms.

MESSAGE [11:20:37 13-Aug-10 EDT][web.server] HTTP Server started on port 80

MESSAGE [11:20:38 13-Aug-10 EDT][fox] Service started on port 1911

November 13, 2015 23

Chapter 4 Diagnostic tools for the DataRecoveryService Data Recovery Service Guide

MESSAGE [11:20:39 13-Aug-10 EDT][sys] Niagara Runtime Environment: 3.6.20

MESSAGE [11:20:39 13-Aug-10 EDT][sys] *** Station Started (27273ms) [153291ms total]
***
niagara>
MESSAGE [11:20:41 13-Aug-10 EDT][sys] Saving station...

MESSAGE [11:20:55 13-Aug-10 EDT][history.db] Saved history archive (4197ms)

MESSAGE [11:21:02 13-Aug-10 EDT][sys] Saved /ffs0/niagara/stations/J202_TestW/

config.bog (19654ms)
The example above shows related messages mixed in with other station startup entries.
Using “spy”, you can increase the station’s log level, by setting it to “trace” on items:
• platDataRecovery.manager
• platDataRecovery.service
This produces much more debug information in station output. An example small snippet below reflects
trace level output when an SRAM data block “flushes” to a flash file.
C A U T I O N : Turning trace logging on either of the logs above may produce extremely large levels of output,
and should not be left in this log level state. This applies particularly to large stations.
TRACE [10:31:12 13-Aug-10 EDT][platDataRecovery.service] External append(history:,
encoded key bytes: appended /J202_TestW/AuditHistory...

TRACE [10:31:12 13-Aug-10 EDT][platDataRecovery.manager] Size of used block exceeds

free space, forcing flush of block

TRACE [10:31:12 13-Aug-10 EDT][platDataRecovery.service] Narcissistic append

(encoded key bytes: AA==, data bytes: 0000b10b000e...

TRACE [10:31:12 13-Aug-10 EDT][platDataRecovery.service] Attempt to append data with

key encoded as bytes: AA== successful

TRACE [10:31:12 13-Aug-10 EDT][platDataRecovery.service] Flush operation successful.

TRACE [10:31:12 13-Aug-10 EDT][platDataRecovery.service] Attempt to append data with

key encoded as bytes: appended /J202_TestW/AuditHistory...
You can also increase the log level of other items related to (or used by) the DataRecoveryService, including
the following:
• alarm.dataRecovery — For logs about the AlarmService’s use of the DataRecoveryService
• history.critical — For logs about the HistoryService’s use of the DataRecoveryService
As well as these additional DataRecoveryService related items:
• sys.critical
• sys.critical.load.item (item = add, change, facetsChange, flagsChange, recategorize,
remove, rename, reorder)
• sys.critical.progObj
• sys.critical.save.item
(item = add, change, facetsChange, flagsChange, recategorize, remove, rename, reorder)
N O T E : The same caution applies as before about trace level on these items—some may produce very large
levels of station output.

24 November 13, 2015

Data Recovery Service Guide Chapter 4 Diagnostic tools for the DataRecoveryService

Spy page for the DataRecoveryService

An extensive “spy” page on a station’s DataRecoveryService is available in Workbench at the following loca-
tion: S p y → s y s M a n a g e r s → D a t a R e c o v e r y M a n a g e r.
F ig ure 5 Spy page for DataRecoveryService

This spy page, partially shown in above, provides numerous statistics about station data being backed up by
the service. Details about these statistics are outside the scope of this document.

Reformatting SRAM
When using a “serial shell” connection to a JACE controller with SRAM, a special “alternative boot” option
is available. A sub-option allows you to reformat the onboard SRAM installed in the JACE.
You must be near the JACE to make this direct serial shell connection, and also be able to easily power it off
and on as needed.
N O T E : In most cases the following reformat procedure will never be necessary. When shipped, the SRAM
option card is already properly formatted, and does not contain any stored data.
However, in rare cases it may be necessary to reformat using this procedure.

R eform atti ng the d ata rec ove ry stora g e op tion i n a JA CE

N O T E : This procedure is written for those controller models that have an SRAM option card installed. The
alternate boot menu options may differ slightly for controller models that have onboard SRAM (or other data
recovery storage option).
Step 1 If needed beforehand, backup the JACE to your PC using normal Workbench platform tools.
Step 2 Power the JACE off, and install the serial shell jumper (if not already installed).
Connect the necessary serial cable between the JACE and your PC’s COM port.
Step 3 On your PC, start a terminal emulation program, for example HyperTerminal, and open a previ-
ously saved setup for JACE-2,-6,-7 communications, using that COM port. Settings are listed
below.
• Bits per second: 115200

November 13, 2015 25

Chapter 4 Diagnostic tools for the DataRecoveryService Data Recovery Service Guide

• Data bits: 8
• Parity: None
• Stop bits: 1
• Flow control: Hardware
Step 4 With your HyperTerminal session active, apply power to the JACE.
After a short delay, text should appear in the HyperTerminal window similar to below:
N O T E : Wait for the prompt:
Press ESC to choose alternate boot options...
and then press ESC.
IPL for NPM 2 (PPC405EP) v3.03 ECC

Press <ctrl-c> to stop autoboot...

Autobooting...

Loading image from on-board and flash

npm2xx startup version 6.41

launching devc
waiting for /dev/ser1

creating symlink for /dev/serconsole

Welcome to QNX Neutrino 6.4 on the NPM 2xx (ppc405)

starting /ffs0 filesystem

ETFS_FS_512
Press ESC to choose alternate boot options...
Step 5 Press ESC to go to the alternate boot menu, a portion of which is shown below.
N O T E : Also, as explained in the alternate boot menu, be careful about which options you skip!
9. Skip starting niagara daemon

10. Skip SRAM option card mount

11. Format SRAM option card before mount

c. continue with boot

Enter choice:
Step 6 At the “Enter choice” prompt, type 11 and press ENTER.
The alternate boot menu repeats, only missing the selected item 11 (“Format SRAM option card
before mount”).
Step 7 Type c (to continue with boot) and press ENTER
You see “continuing boot...” along with other normal boot messages for TCP/IP initialization.
When the boot completes (at the login prompt), there should be entries indicating that a format
operation occurred, as shown in the following:

26 November 13, 2015

Data Recovery Service Guide Chapter 4 Diagnostic tools for the DataRecoveryService

SRAM option card detected in slot 1

Formating SRAM option card before mount

SRAM option card mounted successfully at /sramopt

starting ntpd...
MESSAGE [15:36:48 29-Jun-2010] [tid=1] niagarad: starting, baja_home=/niagara

MESSAGE [15:36:48 29-Jun-2010] [tid=1] webserver: web server started [tid = 4]

MESSAGE [15:36:48 29-Jun-2010] [tid=1] app registry: station registry starting

MESSAGE [15:36:48 29-Jun-2010] [tid=7] engine watchdog: app J202_TestW watchdog

thread started [tid = 8]

MESSAGE [15:36:48 29-Jun-2010] [tid=7] station: station J202_TestW starting

login:
Step 8 When the “login” prompt appears, power off the JACE.
If needed (and applicable), you can now remove the SRAM option card to use in another JACE
controller.
N O T E : If removing an SRAM option card from a JACE-6,-7 controller and not reinstalling, be sure
to uninstall the platDataRecovery module from the JACE controller, using the platform S o f t -
w a r e M a n a g e r.
Step 9 Exit from the HyperTerminal application, selecting to S a v e if you wish to reuse this setup again.

November 13, 2015 27