I recently had the privilege to share a bit of my work on the HDF Group blog by way of a guest post. In our lab we generate increasingly large and diverse simulation datasets, and maintaining sanity and productivity in spite of these scale ups have been a challenge. This post details one approach to the problem, and it makes heady use of HDF5 for efficient data storage and retrieval.

If you have an interest in Big Data challenges and aren’t already subscribed to the HDF Group blog, one of those two things should change. :D

— david