Papers by James Gallagher
Data access protocols on which many emerging data systems are being based are often tightly coupl... more Data access protocols on which many emerging data systems are being based are often tightly coupled with the semantics of the discipline for which the system is being developed. This greatly facilitates interoperability within the data system, but it also generally results in substantial constrains on interoperability with data systems in other disciplines or, for disciplines with a very large range of data types, on interoperability within the discipline itself. In order to build a data access protocol that may be used by a broad range of disciplines, the Open source Project for a Network Data Access Protocol (OPeNDAP) has approached the problem by focusing on the syntax of the data rather than on their semantics. In particular, the OPeNDAP data access protocol consists of a rigid prescription for the syntax of the data and a mechanism for the communication of the data semantics, but no constraint on semantic content. In this way, the protocol becomes a base on which discipline dat...
During the upcoming Summer 2016 meeting of the ESIP Federation (July 19-22), OpenDAP will hold a ... more During the upcoming Summer 2016 meeting of the ESIP Federation (July 19-22), OpenDAP will hold a Developers and Users Workshop. While a broad set of topics will be covered, a key focus is capitalizing on recent EOSDIS-sponsored advances in Hyrax, OPeNDAPs own software for server-side realization of the DAP2 and DAP4 protocols. These Hyrax advances are as important to data users as to data providers, and the workshop will include hands-on experiences of value to both. Specifically, a balanced set of presentations and hands-on tutorials will address advances in1.server installation,2.server configuration,3.Hyrax aggregation capabilities,4.support for data-access from clients that are HTTP-based, JSON-based or OGC-compliant (especially WCS and WMS),5.support for DAP4,6.use and extension of server-side computational capabilities, and7.several performance-affecting matters.Topics 2 through 7 will be relevant to data consumers, data providers andnotably, due to the open-source nature of a...
Remote Sensing, 2016
In order to have confidence in the long-term records of atmospheric and surface properties derive... more In order to have confidence in the long-term records of atmospheric and surface properties derived from satellite measurements it is important to know the stability and accuracy of the actual radiance or reflectance measurements. Climate quality measurements require accurate calibration of space-borne instruments. Inter-calibration is the process that ties the calibration of a target instrument to a more accurate, preferably SI-traceable, reference instrument by matching measurements in time, space, wavelength, and view angles. A major challenge for any inter-calibration study is to find and acquire matched samples from within the large data volumes distributed across Earth science data centers. Typically less than 0.1% of the instrument data are required for inter-calibration analysis. Software tools and networking middleware are necessary for intelligent selection and retrieval of matched samples from multiple instruments on separate spacecraft. This paper discusses the Multi-Instrument Inter-Calibration (MIIC) system, a web-based software framework used by the Climate Absolute Radiance and Refractivity Observatory (CLARREO) Pathfinder mission to simplify the data management mechanics of inter-calibration. MIIC provides three main services: (1) inter-calibration event prediction; (2) data acquisition; and (3) data analysis. The combination of event prediction and powerful server-side functions reduces the data volume required for inter-calibration studies by several orders of magnitude, dramatically reducing network bandwidth and disk storage needs. MIIC provides generic retrospective analysis services capable of sifting through large data volumes of existing instrument data. The MIIC tiered design deployed at large institutional data centers can help international organizations, such as Global Space Based Inter-Calibration System (GSICS), more efficiently acquire matched data from multiple data centers. In this paper we describe the MIIC architecture and services.
2014 Oceans - St. John's, 2014
called the "Research Coordination Network: OceanObs" (RCN) is addressing approaches for improving... more called the "Research Coordination Network: OceanObs" (RCN) is addressing approaches for improving interdisciplinary research capabilities in the ocean sciences. In addition, the RCN continues tasks that focus on improved observation capabilities and on community outreach and education. A working group has addressed issues and challenges of open data. Another working group is focused on improvements in observations where both in situ and remote sensing provides essential information. A use case for the discussion is considers observation of coastal waters. There is also a monthly webinar, "Blue Marvel-Ocean Mysteries" that supports Capacity Building and Outreach. The ultimate goal of these activities is to identify and encourage better ways to make earth observations and to improve availability to society as knowledge and understanding for improved policy and decision-making. This paper reviews the status of the RCN developments.
Earth Science Informatics, 2015
By broad consensus, Open Data presents great value. However, beyond that simple statement, there ... more By broad consensus, Open Data presents great value. However, beyond that simple statement, there are a number of complex, and sometimes contentious, issues that the science community must address. In this review, we examine the current state of the core issues of Open Data with the unique perspective and use cases of the ocean science community: interoperability; discovery and access; quality and fitness for purpose; and sustainability. The topics of Governance and Data Publication are also examined in detail. Each of the areas covered are, by themselves, complex and the approaches to the issues under consideration are often at odds with each other. Any comprehensive policy on Open Data will require compromises that are best resolved by broad community input. In the final section of the review, we provide recommendations that serve as a starting point for these discussions.
004.0.08, 27 October 2005, Added a note that the type names are case-insensitive on page 3.2.1; C... more 004.0.08, 27 October 2005, Added a note that the type names are case-insensitive on page 3.2.1; Clarified the typical use of XDR so that it’s clear that the number of elements of an array is sent twice in the case of an array of atomic types (Byte,..., Float64) but only once in the case of the remaining types (see page 7.3.2.1); An error in the grammar for Sequences which was discovered previously was left out of a previous revision. Sequences are limited to at most a single inner Sequence ‘per level. ’ See page 7.2.2.4. 004.0.07, 3 May 2005, Corrected the description of the Host header in Section 6.2.2. The header was incorrectly described as containing the DNS name or IP address of the client. In fact (and in conformance with HTTP/1.1) it contains the name/number of the server. 004.0.06, 12 April 2005, Changed designation from ‘Proposed... ’ to ‘Draft Community Standard; ’ Added a clarification of ‘stride ’ in a hyperslab (Section 4.1.1); Corrected the grammar for Attributes to in...
004.0.09, 28 August 2007, Added missing double quotes to grammar rules for array-dim (now called ... more 004.0.09, 28 August 2007, Added missing double quotes to grammar rules for array-dim (now called projectionarray-dim), corrected an error in the grammar for a CE (although servers can accept an id without a projectionarray-dim, the grammar did not make that explicit) and corrected an error in the characters REQUIRED to ba accepted for an id (most servers accept more characters, but the required set is fairly restricted) on pages 21 and 22; In Section 7 (page 23) the footnote listed the values for carriage return and line feed in reverse order; Corrected the grammar for a DDS in Section 7.2.2 (page 28) ’data-source ’ → ’dataset’; Added a note in Appendix A.2 regarding RFC 2616’s suggestion that ’tolerant ’ applications ignore the CR and use only LF as a separator when they process a CRLF pair- we suggest not assuming HTTP software will follow this advice; In Section 7.1.4 (page 24) ’includ ’ → ’include’; In Section 8 (page 35) The value of the ’XDODS-Server ’ header has been changed ...
: As part of a phased approach toward an Integrated Ocean Observing and Prediction System, NOPP i... more : As part of a phased approach toward an Integrated Ocean Observing and Prediction System, NOPP in 1999 selected a few preliminary steps for development. One of these steps was to plan and implement "a community-based 'system' to broaden and improve access to ocean data". The project described herein was proposed in response to the 2000 NOPP BAA and has as its long-term goal the development of such a system; i.e., the implementation of a network based system that would provide for the discovery of and seamless access to oceanographic data. The ultimate goal is a system that will provide immediate access to a vast array of real-time and historical oceanographic archives by all segments of the oceanographic community from the one-time user to the modelers and managers requiring regular and automatic ingestion of the highest quality data and data products. The objectives of this project are: 1. The development of a data access protocol capable of handling the exchange...
AIP Conference Proceedings, 1993
As part of the NASA funded REASoN project, A Thematic Data Portal to Satellite-Derived Ocean Surf... more As part of the NASA funded REASoN project, A Thematic Data Portal to Satellite-Derived Ocean Surface Properties, the University of Rhode Island in conjunction with OPeNDAP Inc. is developing a suite of graphical user interfaces (GUIs) to facilitate access to data sets of satellite-derived ocean surface properties. These data sets are served from a variety of institutions in the US and abroad. The suite of GUIs currently (1 March 2008) consists of eight major oceanographic data sets. On average, one new GUI is being added every three weeks. That which distinguishes these data sets is their size and their complexity. In fact, some of the more complex 'data sets' actually consist of collections of data sets. For example, the ocean color data set, served from CSIRO in Australia, consists of CZCS, SeaWiFS, MODIS and merged MODIS and SeaWiFS data sets and the 4km Pathfinder SST data set, served from JPL, consists of a number of SST products based on the AVHRR GAC data stream, rang...
Ecological Informatics, 2010
Environmental sensor networks are now commonly being deployed within environmental observatories ... more Environmental sensor networks are now commonly being deployed within environmental observatories and as components of smaller-scale ecological and environmental experiments. Effectively using data from these sensor networks presents technical challenges that are difficult for scientists to overcome, severely limiting the adoption of automated sensing technologies in environmental science. The Realtime Environment for Analytical Processing (REAP) is an NSF-funded project to address the technical challenges related to accessing and using heterogeneous sensor data from within the Kepler scientific workflow system. Using distinct use cases in terrestrial ecology and oceanography as motivating examples, we describe workflows and extensions to Kepler to stream and analyze data from observatory networks and archives. We focus on the use of two newly integrated data sources in Kepler: DataTurbine and OPeNDAP. Integrated access to both near real-time data streams and data archives from within Kepler facilitates both simple data exploration and sophisticated analysis and modeling with these data sources.
Data Science Journal, 2003
In the process of implementing a protocol for the transport of science data, the Open Source Proj... more In the process of implementing a protocol for the transport of science data, the Open Source Project for a Network Data Access Protocol (OPeNDAP) group has learned a considerable amount about the internal anatomy of what are commonly considered monolithic concepts. In order to communicate among our group, we have adopted a collection of definitions and observations about data and the metadata that make them useful: differentiating between "semantic" and "syntactic" metadata, and defining categories such as "translational" and "use" metadata. We share the definitions and categorizations here in the hope that others will find them as useful as we do.
Abstract: The OpenGIS standard for web-based geographic information systems (GIS) has shown promi... more Abstract: The OpenGIS standard for web-based geographic information systems (GIS) has shown promise for implementing wide-area distributed GIS. However, making existing numerical data available as maps can be timeconsuming and expensive, especially for large data archives. The map transformation process requires extraction of a subset from data files, interpolation, representation as a raster image and in some cases reprojection. We use the Distributed Oceanographic Data System (DODS) as an underlying subsetting ...
Eos, Transactions American Geophysical Union, 1993
ABSTRACT Software that allows remote users to browse satellite-derived images and in-situ data ar... more ABSTRACT Software that allows remote users to browse satellite-derived images and in-situ data archives of interest to oceanographers via Internet has been developed by researchers at the University of Rhode Island/s Graduate School of Oceanography. The tool, called ``Xbrowse,'' can access a recently assembled archive of Advanced Very High Resolution Radiometer (AVHRR) Sea Surface Temperature (SST) imagery covering the Western Atlantic from 1979 to present, with new passes being added daily. From January 1979 through March 1982, the coverage is spotty. From April 1982 to present, every pass downloaded on the Wallops Island receiving station is available. Each pass in the archive has been remapped to standard geophysical coordinates.
Abstract: The OpenGIS standard for web-based geographic information systems (GIS) has shown promi... more Abstract: The OpenGIS standard for web-based geographic information systems (GIS) has shown promise for implementing wide-area distributed GIS. However, making existing numerical data available as maps can be timeconsuming and expensive, especially for large data archives. The map transformation process requires extraction of a subset from data files, interpolation, representation as a raster image and in some cases reprojection.
Uploads
Papers by James Gallagher