The JSA and the Grid: How "infinite" computing power enables a new archive model for PI-led observatories

Economou, Frossie

The JCMT Science Archive (JSA), a partnership between the James Clerk Maxwell Telescope and the Canadian Astronomy Data Center, is now entering its second year of deployment and is continuing to evolve. The JSA offers raw data, nightly products and advanced data products from heterogeneous sub-millimeter instruments taken under a range of conditions and for different types of programmes, from small PI-led runs to large surveys.

Our experience with the development and operation of the JSA has demonstrated that that Grid Computing and Grid Virtualisation offer compelling benefits even for observatories dealing with data sets that nowadays would be considered less than terrifying ( < 100 TB/year). For one, they enable us to adopt an "agile data release" model: instead of heavily curated but rare releases of data products, the JSA adopts a "reduce-often release-always" approach where data is constantly reprocessed and the latest product is immediately exposed to the user and available for VO harvesting. In combination with a robust pipeline environment, cloud computing allows us to turn one of the most challenging data sets in astronomy, that of ground-based sub-millimeter instrumentation, into science-quality automatically-produced 1-D, 2-D and 3-D aggregate data products.

We also believe that our experience with the JSA provides a model on how "PI-led" telescopes (those not built for a specific survey purpose) with diverse science programs and instrumentation suites can successfully publish their products in a science archive. A partnership with an experienced data center provides a smooth route to Virtual Observatory publication (at least to the extent that existing VO protocols allow it).

Return to oral presentation list