RequirementData are frequently spread across different administrative domains and often reside on a range of incompatible resources such as GridFTP, SRB/iRODS, SRM, SFTP, FTP, WEBDAV, HTTP, and local FILE. Copying potentially large quantities of data in a scalable manner between different types of resource remains a key challenge. Existing technologies do go some way to providing the required capabilities (e.g. 3rd party transfer between two GridFTP severs). However, support for the wide range of protocols that are currently in use across different institutions and scientific facilities remains limited. A common approach used to tackle this problem often involves manually orchestrating the buffering of bytes between a data source and sink via an intermediary, usually the users own desktop. However, this presents a number of challenges; a) the intermediary must be constantly available throughout the duration of a transfer, b) buffering large volumes of data introduces bandwidth and concurrency issues, especially for clients on limited networks and subject to restrictive firewalls.
DTS Service (Asynchronous messaging and worker node pool)The service being developed is based on well established enterprise message broker technologies and the Apache Commons Virtual File System1,2. VFS provides data streaming between incompatible protocols via byte IO and is highly extensible for new and emerging protocols2. Message brokering provides asynchronous communication and routing of transfer request messages to an expandable cloud of dedicated worker nodes implemented with VFS. If the service is being overburden and falls behind in its processing, all that is needed is to turn-up a few more worker instances to listen to the queue. Importantly, workers may be deployed at different institutions and within different network domains to a) improve resilience, since workers can be added and removed without affecting each other, and b) maximize transfer efficiency, since workers can be strategically placed to serve a particular data source and/or sink. In doing this, the DTS design meets geographical-topological concerns that may exist from a deployment perspective by allowing the hosting of such services to be either centralized (across multiple facilities) or confined to a single institution / network.
DTS Message ModelFor the service messaging model, we have (for now) resorted to a composite schema for defining our message formats that combines both JSDL1 and DMI2 elements with our own proprietary extensions. Ideally, we would like to produce a standards compliant message format (rather than resorting to our proprietary format). We are currently working in this area to hopefully address this. Please refer to our wiki for details: http://code.google.com/p/dtsproject/wiki/DMI_JSDL_Issues
DTS ArchitectureThe basic components of the proposed system can be summed up as follows:
DTS ClientInteracts with the user to allow the submission of data transfer operations. It allows the user to browse the source and target URIs so as to select the data staging elements that make a up a data transfer operation.
DTS Web ServiceThis is the external interface of the DTS system. It allows DTS clients to submit jobs, obtain progress information on existing jobs as well as to issue control requests (pause a job, abort a job etc). This module is also responsible for updating the job data store to reflect changes in the status of a given job.
DTS WorkernodeThis is the workhorse of the system. Its primary function is to process an incoming DTS message and to provide feedback to higher levels in the system on the status of such an operation. It is anticipated the DTS cloud will consist of 1 to ‘n’ DTS worker nodes.
The PrototypeWe have implemented a prototype for the DTS and this wiki page will show you have to build and run it.
The DtsModules wiki lists all the modules that the DTS application uses.
