-
Metadata and provenance management
Authors:
Ewa Deelman,
Bruce Berriman,
Ann Chervenak,
Oscar Corcho,
Paul Groth,
Luc Moreau
Abstract:
Scientists today collect, analyze, and generate TeraBytes and PetaBytes of data. These data are often shared and further processed and analyzed among collaborators. In order to facilitate sharing and data interpretations, data need to carry with it metadata about how the data was collected or generated, and provenance information about how the data was processed. This chapter describes metadata an…
▽ More
Scientists today collect, analyze, and generate TeraBytes and PetaBytes of data. These data are often shared and further processed and analyzed among collaborators. In order to facilitate sharing and data interpretations, data need to carry with it metadata about how the data was collected or generated, and provenance information about how the data was processed. This chapter describes metadata and provenance in the context of the data lifecycle. It also gives an overview of the approaches to metadata and provenance management, followed by examples of how applications use metadata and provenance in their scientific processes.
△ Less
Submitted 14 May, 2010;
originally announced May 2010.
-
The Earth System Grid: Supporting the Next Generation of Climate Modeling Research
Authors:
David Bernholdt,
Shishir Bharathi,
David Brown,
Kasidit Chanchio,
Meili Chen,
Ann Chervenak,
Luca Cinquini,
Bob Drach,
Ian Foster,
Peter Fox,
Jose Garcia,
Carl Kesselman,
Rob Markel,
Don Middleton,
Veronika Nefedova,
Line Pouchard,
Arie Shoshani,
Alex Sim,
Gary Strand,
Dean Williams
Abstract:
Understanding the earth's climate system and how it might be changing is a preeminent scientific challenge. Global climate models are used to simulate past, present, and future climates, and experiments are executed continuously on an array of distributed supercomputers. The resulting data archive, spread over several sites, currently contains upwards of 100 TB of simulation data and is growing…
▽ More
Understanding the earth's climate system and how it might be changing is a preeminent scientific challenge. Global climate models are used to simulate past, present, and future climates, and experiments are executed continuously on an array of distributed supercomputers. The resulting data archive, spread over several sites, currently contains upwards of 100 TB of simulation data and is growing rapidly. Looking toward mid-decade and beyond, we must anticipate and prepare for distributed climate research data holdings of many petabytes. The Earth System Grid (ESG) is a collaborative interdisciplinary project aimed at addressing the challenge of enabling management, discovery, access, and analysis of these critically important datasets in a distributed and heterogeneous computational environment. The problem is fundamentally a Grid problem. Building upon the Globus toolkit and a variety of other technologies, ESG is developing an environment that addresses authentication, authorization for data access, large-scale data transport and management, services and abstractions for high-performance remote data access, mechanisms for scalable data replication, cataloging with rich semantic and syntactic information, data discovery, distributed monitoring, and Web-based portals for using the system.
△ Less
Submitted 13 December, 2007;
originally announced December 2007.
-
Secure, Efficient Data Transport and Replica Management for High-Performance Data-Intensive Computing
Authors:
Bill Allcock,
Joe Bester,
John Bresnahan,
Ann L. Chervenak,
Ian Foster,
Carl Kesselman,
Sam Meder,
Veronika Nefedova,
Darcy Quesnel,
Steven Tuecke
Abstract:
An emerging class of data-intensive applications involve the geographically dispersed extraction of complex scientific information from very large collections of measured or computed data. Such applications arise, for example, in experimental physics, where the data in question is generated by accelerators, and in simulation science, where the data is generated by supercomputers. So-called Data…
▽ More
An emerging class of data-intensive applications involve the geographically dispersed extraction of complex scientific information from very large collections of measured or computed data. Such applications arise, for example, in experimental physics, where the data in question is generated by accelerators, and in simulation science, where the data is generated by supercomputers. So-called Data Grids provide essential infrastructure for such applications, much as the Internet provides essential services for applications such as e-mail and the Web. We describe here two services that we believe are fundamental to any Data Grid: reliable, high-speed transporet and replica management. Our high-speed transport service, GridFTP, extends the popular FTP protocol with new features required for Data Grid applciations, such as striping and partial file access. Our replica management service integrates a replica catalog with GridFTP transfers to provide for the creation, registration, location, and management of dataset replicas. We present the design of both services and also preliminary performance results. Our implementations exploit security and other services provided by the Globus Toolkit.
△ Less
Submitted 28 March, 2001;
originally announced March 2001.
