Secure and Reliable File Transfer


Transferring payload of significant size through a reliable and secure manner has become a common need. People are trying to use existing method like ebMS or AS2 to handle this but find that the payload size these method can deal with is limited. Here in the article, we are going to share the technology we developed for a project (click to visit the project page) funded by Innovation Technology Fund (ITF) which is aimed to establish an infrastructure that can support transmission of digital content of any size in a secure, reliable, transactional, and managed manner .

The options available for file transfer

The need to exchange file is not anything new, different technologies are developed to cater for the needs. The slides below illustrates the possibilities, together with the short coming for each.

A new protocol is born

The design principle is simple - break, and join. We start by defining a new message protocol which we call it SFRM (Simple File Reliable Message). The design has borrowed the technology from the standard for reliable message exchange, such as acknowledgment request, digital security, etc. Apart from this, the protocol defined three types of messages - Meta Message, Regular Message (i.e. the message that carries the payload) and Acknowledgment. The "Meta Message" is the first thing that the sender will deliver. It enables a "hand shaking" process to ensure the recipient has enough disk space to receive the file. Afterwards, the sender side will chop down the payload into segments and deliver to the recipient side as a "Regular Message", the protocol has defined the reliability and security measure with reference to AS2. One of the different between SFRM and AS2 is the way to acknowledge a successful transmission. Instead of sending an acknowledgment for each message, SFRM will enquire the status of a collection of segments in a single "Acknowledgment Request". The receiver will then return response a message that contains the status of the segments concerned. This approach lower the number of message exchange required. The steps involved can be summarized as follows,

On top of the protocol, we have a solution

With the protocol defined, we have it implemented as a plugin for H2O. This plugin provides API for easy integration, what's more, the plugin makes uses of the active-tasks mechanism to control the throughput of file transfer. Different kind of application can build on top of this, let me show you one of the application we developed. The screencast below demonstrates a frontend client, which has H2O and SFRM embedded, transmitting file to another H2O server having the SFRM plugin installed. The user didn't need to know the API, they can enjoy the benefits of delivering a large file in a reliable and secure manner.


This ScreenFlow video requires a more recent version of the Adobe Flash Player to display. Please update your version of the Adobe Flash Player.

Discuss this in our forum

Posted by Ronnie Kwok on 04/22 at 03:05 PM