Design and implementation of the LOFAR Long Term Archive data model and interfaces

Renting, Adriaan

The LOFAR Long Term Archive (LTA) is the system designed for storage, searching and further processing of the LOFAR scientific data products. It houses LOFAR data that needs to be stored on timescales longer than the few weeks that the LOFAR central processing systems can provide. Its purpose is to make this data available to the scientific community, allow for easy querying and further processing of the data. Its two main components are a centralized database and portal together with distributed Petabyte scale data storage and processing facilities.

The data model of the archive itself is designed to describe the processes and intermediate data products that led to the creation of each stored dataset. The resulting data product description allows for full understanding of how a data product was created and trace its origins back to the observations from which it originates. Raw and intermediate data is often too large to be stored after initial processing. This means the data products that do end up being stored in the LTA need to be accompanied by a clear description of which processes and data they originated from.

Together with the data model interfaces and procedures for storing and retrieving data together with detailed data formats have been defined. This has been done to allow easy public access and future functioning as a legacy archive. The data model and several of the interfaces, procedures and data formats of the LTA will be presented in this paper and the choices that were made which led to their current design.

Return to poster list