Load testing repository platforms

Very interesting post on Stewart Lewis's blog:

"One of the core aims of the ROAD project is to load test DSpace, EPrints and Fedora repositories to see how they scale when it comes to using them as repositories to archive large amounts of data (in the form of experimental results and metadata). According to ROAR, the largest repositories (housing open access materials) based on these platforms are 191,510, 59,715 and 85,982 respectively (as of 18th July 208). We want to push them further and see how they fare."

Stuart goes on to mention a paper on "Testing the Scalability of a DSpace-based Archive", that pushed it up to 1 million items with "no particular issues"! They have their chunky test hardware in place, and plan to do the stress test using SWORD, so that they can throw the same suite of data at each repository.

He says

"More details will be blogged once we start getting some useful comparative data, however seeing as the report cited above took about 10 days to deposit 1 million items, it may be some weeks before we’re able to report data from meaningful tests on each platform."

This is an extremely worthwhile exercise, particularly for those thinking of using repository software platforms for data, where the numbers of objects can very rapidly mount up.