Research Data Transfer Zone
The Problem
With the volume of data generated from research projects increasing rapidly, researchers from more and more fields and disciplines are facing issues inherent with managing large volumes of data.
One such issue is transferring data to and from advanced scientific facilities so that researchers can conduct their experiments and carry out their research using the most cutting-edge technology. But in the past, data transfer capabilities have not always kept up with the capacity of scientific facilities.
One of the ways that institutions are aiming to ease the burden of managing/transferring such high volumes of research data is through improved network infrastructure.
The Solution
SES worked with Jisc and Diamond Light Source (Diamond) to establish a coherent framework that will significantly increase the end-to-end data speeds between SES partners and Diamond.
The purpose of the Research Data Transfer Zone (RDTZ) was to reduce the end-to-end network impediments that affect the speed at which research – rather than business – data can be transferred between experimental, computational and storage facilities.
The RDTZ follows a similar framework to the Science DMZ model documented by the US Department of Energy (ESET) and leading universities in the USA
The process
SES assembled resources from Jisc, Diamond Light Source and university IT networking and leadership teams to identify and work with researchers that already experienced data transfer issues with Diamond. The aim was to implement a RDTZ to explore how this framework may address these challenges in an efficient and user-friendly way.
The outcome
The University of Southampton was the first institution to pioneer the setup alongside SES, with the pilot project demonstrating an increased capability in data download speed.
The initial pilot demonstrated the ability to reduce overall download time scales from 10-30 days to just a few hours, discouraging researchers from resorting to out-of-date data transfer methods, such as commonly used portable hard drives, and instead using the power of the Janet network infrastructure to move their data.
The project concluded in April 2018 with an event at the University of Southampton to raise awareness of the RDTZ and its recent achievements and the potential to expand the zone.
May 2021 update
Since the completion of the RDTZ project, other universities have approached the challenge of research data transfer. With most institutions now connected to the Janet network at a minimum of 10Gbit/s and some recently being upgraded to 100Gbit/s, ensuring your organisation has an effective local network architecture, the right software tools, and optimised file store has become ever more important.
Imperial College London is one university connected to Janet at 100Gbit/s, where RDTZ principles have been applied. Their researchers have access to accounts on their HPC system and research data store. As part of the Covid-19 response, the combination of a local RDTZ and the high capacity link to Janet enabled the transfer of a 5TB Covid-related dataset between the European Bioinformatics Institute (EBI) and Imperial College London in less than three hours at a data rate of around 500MB/s (4Gbit/s).
Other universities that have worked to improve data transfer speeds include QMUL and UCL.