At New NIST Facility, Response Robots Must Measure Up

The growing number of visitors with mechanical escorts traveling to the newest building on the Gaithersburg, Md., campus of the National Institute of Standards and Technology (NIST) come expecting a disaster. And their hosts, members of NIST's Emergency Response Robots Project, never disappoint.

With careful attention to detail, lead engineer Adam Jacoff and his team have appointed much of the interior and exterior of NIST's year-old Robotics Test Facility with features inspired by catastrophe—earthquakes, hurricanes, explosions, nuclear-reactor meltdowns, and other cataclysmic events. Each suite—about three dozen in all—is designed for testing and measuring capabilities of robots built to assist search-and-rescue teams, bomb squads, and others who respond to natural and man-made disasters or other hazardous situations.

At this arena of disarray-by-design, robot manufacturers, first responders and military personnel, and researchers aiming for next-generation response-robot capabilities can determine how well and how reliably their particular machines are likely to carry out tasks at disaster scenes. For example, can a robot maneuver through rubble, scale a collapsed wall, or climb a ladder? Can it find victims trapped in wreckage, wield a hose, close a valve leaking toxic gas, operate power tools, or disable an improvised explosive device? Does the power source last as long as specified, and what is the range of the robot's communication system?

The 892-square-meter (9,600 square-foot) Robotics Test Facility is where many in the robotics and first-responder communities come for answers. At this proving ground, tests for measuring response-robot performance are prototyped, refined, and, ultimately, proffered for community-wide adoption as consensus standards.

"You need to measure how both the current practice of robotics and the state of the science and the emerging technology are addressing the particular challenges of disaster response," Jacoff explains. "This is best done in an environment that enables collaboration, facilitates adoption of best-in-class solutions, and helps the entire industry raise its sights and pursue the next set of advances that will lead to more capable systems."

The new facility is a big step up from the converted NIKE missile site where the performance-testing program got its start. Since 2005, 15 NIST tests have been adopted as standards by ASTM International and about 60 more are under various stages of development or review. The assortment of robots tested also has expanded beyond remotely operated ground vehicles to include aquatic vehicles and small unmanned aerial systems.

"Standard test methods are essentially just agreed-upon ways to test robotic capabilities," Jacoff says. "So we are not developing a specification for a 'standard robot' of any kind, like an equipment standard. Rather, we are developing a standard way to test remotely operated robotic systems so that performance can be measured and compared."

Realistic, reliable, and reproducible, these benchmark standards are credited with helping the young field of disaster robotics to make tangible progress since the technology was first deployed—at the collapse of the World Trade Center towers following the terrorist attacks on Sept. 11, 2001. Since then, robots have been used at more than 30 disaster sites around the world. The increase in deployments reflects improvements in capabilities, especially for reconnaissance tasks. But growing even faster is recognition of the many as-yet unrealized opportunities for robots to render essential, timely assistance in unpredictable hazardous environments. Getting there will require leaps in capabilities.

Today's robots, according to Gill Pratt, program manager at the Defense Advanced Research Projects Agency (DARPA), are like one-year-old children beginning to walk and interact with the world.

Not content to allow the field to linger in infancy, Pratt came up with the idea for the DARPA Robotics Challenge (DRC). The $2 million, winner-take-all competition is intended to "transform the field of robotics and catapult forward development of robots featuring task-level autonomy that can operate in the hazardous, degraded conditions common in disaster zones."

Early on, DARPA turned to NIST and asked it to lead development of the DRC tests. NIST's job was to develop a mock-up of a disaster scenario prompted by Japan's Fukushima Daiichi nuclear meltdown, caused by the 2011 Great East Japan earthquake and tsunami. With DARPA's input, Jacoff's team distilled lessons learned from the Fukushima Daiichi response into eight basic tasks and converted these tasks into benchmark performance tests.

The new Robotics Test Facility opened just seven months before the DRC trials, which were held Dec. 20-21, 2013, at Florida's Homestead-Miami Speedway. Almost immediately, the new building began to fill with an assortment of building materials. And soon thereafter, NIST staffers were sawing, hammering, fastening, and assembling them into what began to resemble a jungle gym for robots. A jumble of cinder-blocks seemingly strewn about the facility's rear exterior would become the route that competing robots traversed in DARPA's staged disaster. Inside, engineers and technicians readied other tests that would challenge robots to locate and shut off valves, wield a tool to break through a wall, and accomplish other DRC tasks.

By early December, the team was packing up the simulated disaster scenario for shipment to Florida. Like all of NIST's robot performance tests, the eight DRC tests were designed to be stacked on standard pallets and arranged for easy reassembly at what Jacoff called "one of the biggest robotics evaluations on Earth."

Of 16 U.S. and Japanese contestants, eight robot teams emerged from the December trials. These teams will go on to compete in the finals, to be held June 5-6, 2015, in Pomona, Calif. NIST and DARPA are now discussing plans for an even more rigorous round of performance challenges.

"The DRC Finals will provide more difficult, faster-paced tests designed to mimic an actual disaster scenario," Pratt explained in an article. And unlike the trials, the finalists will not be connected to tethers to deliver electricity to the robots, nor to safety lines to protect them from falling or otherwise damaging themselves.

NIST's work for DARPA epitomizes the collaborative ethos that Jacoff and team have cultivated since their earliest days of developing robot test methods.

"Our aim always has been to provide a reliable, useful testing resource for all parties across the robot development spectrum, from researchers to manufacturers all the way through to the end users," Jacoff says. "We measure success, capture failures, and deliver lessons learned. Through it all, we remain friends."

To date, more than 100 response robots have run the gauntlet of NIST test methods. Scores more have been tested when the NIST team takes its arena on the road for the annual RoboCupRescue competitions for robotics researchers, which Jacoff co-founded, or to other response robot evaluation exercises geared to emergency responders and soldiers.

Increasingly, operators of response robots come to the NIST course to test drive their mechanical assistance and to sharpen their skills. Today, response robots are controlled remotely by trained human operators, but they are gradually becoming more adept at performing basic navigational tasks unaided.

The goal is to advance to a more cooperative working relationship—called supervised autonomy—where robots perform many simple tasks on their own but continue to rely on humans for judgment and situational awareness. To help measure progress toward this goal, NIST has developed a tiered framework for evaluating how well response robots decide and perform on their own.

"Turning over more of the decision-making and workload to robots would allow non-experts to operate response robots, which could speed up response times significantly," explains Elena Messina, leader of the Manipulation and Mobility Group within NIST's Intelligent Systems Division. At Fukushima, for example, operators required several weeks of training before robots could be first deployed after the site was inundated by the tsunami.

"With greater autonomy, robots could be deployed in the immediate aftermath of a hazardous event, when lives still can be saved and when conditions are too dangerous for search and rescue teams to enter," Messina says.

Inspiration for NIST's tests comes from robot performance requirements gleaned from various user communities—bomb squads, search teams, military reconnaissance units, and others. Ideas for new requirements, however, also may spring from a breakthrough innovation that robotics researchers or manufacturers may introduce while running their newest creations through the paces on the test course.

Testing and training at NIST helps to ensure that users are realistic in their expectations—no robot has yet to excel in all tests—and, at the same time, provides a reality check for marketing claims. Typically, a robot will undergo 20 to 30 tests, chosen by a manufacturer, prospective customer, or researcher to establish benchmark capabilities for intended applications or missions. The combination of tests also helps in determining performance and cost trade-offs.

Test results can guide robot procurements as well as set tangible improvement targets that researchers and manufacturers can aim for in the next iteration of their technology.

Defense and civilian agencies alike consult the NIST test results to inform their purchase decisions. For example, the Fire Department of New York has been a frequent customer. Most recently, it used the NIST test course to scope out robot capabilities and refine mission requirements before investing in a robot platform to support hazmat operations.

The Defense Department's Joint Improvised Explosive Device Defeat Organization used NIST performance-test results to guide a $32 million procurement of ultra-lightweight reconnaissance robots for tactical military operations. Similarly, the National Capital Region Bomb Squad Working Group, consisting of 15 federal, state, and local bomb squads, based recent robot purchases on NIST performance test results.

After using quantifiable data on robot capabilities to specify their robot purchases, several agencies also have required that robot vendors perform "acceptance testing" under the same NIST standard test methods before the machines can be put into service.

The NIST robot test course sprouted from a Department of Homeland Security (DHS) project to develop standardized methods to test and measure the abilities of urban search-and-rescue robots. In addition to DHS and the Department of Defense, the Department of Justice has also funded the NIST special program.

The value of these DHS-NIST-ASTM International Standard Test Methods is now multiplying as organizations around the world are replicating the test apparatuses, conducting their own evaluations, and collaborating with NIST to prototype and validate new tests. Similar test facilities in the U.S. include Southwest Research Institute in San Antonio, Texas. The newest to come on line is the New England Robotics Validation and Experimentation Center at the University of Massachusetts Lowell. It joins NIST-inspired courses in Japan and Germany. Next up is a bomb squad training facility in Perth, Australia.

In addition, scores of organizations have installed subsets of the NIST test methods to support development and testing of specific capabilities, leading to innovation and hardening of new robotic skills that will improve disaster-response capabilities and help to keep responders out of harm's way, according to Jacoff.

– written by Mark Bello