Digital Curators wage war on terabytes

Joy Davidson | 15 November 2005

Digital curators wage war on terabytes

As information overload makes meaningful data storage more difficult, a fledgling research centre aims to bring order out of chaos.

Julie Nightingale
Tuesday November 15, 2005
The Guardian

It's an oft-heard complaint that, as a society, we are drowning in information - a data mountain of survey responses, statistics, research findings and experiment outcomes, the details of which become blurred and their true significance buried.
If it's a situation that Joe Public dimly recognises then it's a tediously familiar one in academia where, facilitated by technology, the generation of data has rocketed. In science alone, the amount of information produced is reckoned to double every couple of years.

Gigabytes have given way to terabytes as the standard volume measurement (one terabyte = 1,000 gigabytes and roughly 100,000 times the capacity of the average home computer) and their big sister petabyte (1,000 terabytes) is coming up on the rails.

The search for knowledge has similarly boomed. The UK Data Archive curates the largest collection of digital data in the social sciences and humanities in the UK and is the custodian, among other things, of population censuses. In 2000, the centre was handing out around 6,000 data sets to researchers annually. Five years later, the number has climbed to 20,000.

For a researcher, ploughing through the volume of information isn't the only issue. Pinpointing the significant findings of others that will inform your own work is made more challenging when the data set you're pursuing is poorly labelled, stored in haphazard fashion, missing the back-up info that would verify its authenticity and accuracy or interred in an out-of-date software format.

Vital documentation

It may sound like carelessness on the part of boffins but it can happen at the highest levels; it has long been rumoured, for example, that Nasa mislaid vital documentation from the early years of the space programme in the 1960s, specifically the findings from its first satellite launches.

This less-than-rigorous approach is not a surprise, says Chris Rusbridge, director of the Digital Curation Centre at the University of Edinburgh.

"Yes, in a sense, it is simply a question of good housekeeping but it is housekeeping that has to last 50 years or more. You may have to keep notes in association with the data for long periods and you don't know what environmental changes may occur that might affect the data. How many pictures do we all have in our photo albums that we simply don't recognise because no one wrote who, what and where on the back?"

Funded by the JISC and the Research Councils UK E-Science programme, the centre was set up in 2004 to improve digital curation - effectively, the process of maintaining and adding value to a trusted body of information held in digital form for current and future use. The DCC is focusing specifically on scientific and scholarly data.

The centre will devise new software tools and processes and also act as a repository for ones already developed in universities and other research centres around the country, making them more widely available and disseminating good practice. The work is in its early stages but the academic community has identified key areas for its focus.

Director of the UK Data Archive, Kevin Schurer and his team of experts in data preservation and dissemination are looking to the centre to help them crack one particular time-consuming problem: 'migration' - transferring data to a new format when the one in which it was created becomes obsolete or is updated.

"Migration is a fundamental issue for us," Schurer explains. "At the moment, we keep things in a non-proprietary format - ASCII or RTF - so we are not reliant on specific software, then we carry out a technology watch. If something is changing we 'migrate' the files en masse. We have a system that allows us to identify all files of the same type at once so if something needs to be upgraded we can pick them all out." Even so, he adds, it remains a laborious task.

Complexity and variety

The British Atmospheric Data Centre also has issues related to volume of data and formatting. The centre currently stores around 120 terabytes of atmospheric and meteorological information gathered by aircraft instruments or satellites, among others, and including data used by the Met Office for its weather forecasts.

"All of the data I have stored will be small compared with the amount to come," points out Bryan Lawrence, head of the centre.

The complexity and variety of formats in which environmental scientific data is held makes transfer and comparison difficult. As if that wasn't enough, every new scientific discovery appears to generate its own set of formats.

What's needed, says Lawrence, is a management tool that acts as a framework, listing every aspect of data analysis that the researcher should be considering and suggesting possible processes that ensure the data collected is accurate.

The centre is concentrating initially on data but plans to incorporate other digital material, such as images, electronic books and websites, into its remit.

As the rate of exploration in science shows no sign of tailing off, the problem for academia is ongoing, says Rusbridge.

"Technology is advancing at a great rate and new capabilities come in that bring in new kinds of challenges. A few years ago it was impossible to imagine that we'd be working with petabytes of data but scientists are unstoppable."


Digital Curation Centre:
UK data archive:
British Atmospheric Data Centre: