HDF5 came into being in 1998 as a result of a collaboration among the National Center for Supercomputing Applications (NCSA) and the three National Laboratories - Sandia National Laboratory (SNL), Lawrence Livermore National Laboratory (LLNL), and Los Alamos National Laboratory (LANL). In an effort to develop common data management standards in support of the Accelerated Strategic Computing Initiative (ASCI), HDF5 was created to address specific ASCI requirements and at the same time is becoming a national standard for scientific data management.
At all three labs, but especially at SNL, HDF5 has become a mainstay for data management for many important applications. The current project is a three-year cooperative agreement between The HDF Group and ASC to continue supporting HDF5 for users at SNL and LLNL. Particular attention is given to three types of applications: (1) applications built upon the SAF data model implementation, such as the SIERRA framework supporting multi-physics coupling, (2) applications that use HDF5 directly, and (3) applications that currently use netCDF and have an interest in migrating to netCDF 4, which will use HDF5 as its underlying format.
This project has the following objectives in support of ASC:
- To conduct regular releases and maintenance of HDF5 on platforms of importance to ASC applications at SNL and LLNL
- To port HDF5 to new key platforms at SNL and LLNL.
- To fix bugs with high priority to SNL and LLNL ASC applications
- To add features to the library, tools and format with high priority to SNL ASC applications
- To analyze performance and tune HDF5 to meet SNL ASC application needs
- To support developers and end users within SNL
Porting involves making sure that HDF5 can be used appropriately on the system architecture, configuring of the HDF5 library to build correctly on a new system, and integration of the library with other key software, such as the system's file system and libraries that HDF5 depends on.
Maintenance on a particular system involves keeping the latest release of HDF5 available and working on the system, periodic testing of the HDF5 library on the system, and making sure that the library is compatible with new versions system components, such as compilers and file systems. For SNL, maintenance also involves addressing issues involving parallel file systems, maintaining important snapshots of releases under development to allow The HDF Group and SNL to test performance and feature improvements.
Recovering files after crashes.
This task is recognized as an important and immediate need for ASC applications, and a sizeable portion of the resources are being devoted to this task. Because this task is difficult and complex, it will probably not be possible to arrive at a general solution with these resources. Instead, our effort will be targeted at understanding and addressing the specific needs of ASC.
Because chunking performance is very important for the SAF and EXODUS applications, resources will continue to go to improving chunking performance, especially in cases where small writes are involved.
Small amounts of R&D activities will be devoted to improving performance in write-only scenarios, and to investigating the possibly implementation of variable-sized chunking.
User support, meetings.
User support involves rapid email and telephone to Lab staff, including system staff as well as applications developers and users. Lab staff are able to participate in weekly HDF5 planning meetings as needed, and The HDF Group staff visit the Labs on-site periodically to offer tutorials and to meet with Lab staff over matters of interest involving HDF5.
For more information, click the links on the left.
- - Last modified: 16 May 2011