Moving large volumes of smaller files – Webinar follow-up

In our June webinar last month we discussed how FileCatalyst is able to transfer large volumes of small files. This is accomplished by creating larger archives comprising of the smaller files, rather than sending the files individually.

This blog is intended to further elaborate on some features I didn’t discuss in the webinar and as well as some new features we are working on that will help to accelerate this process even further.

One drawback of sending files in the archive is that they are not accessible until after the archive is extracted. To overcome this issue with the current feature set, one can segment the file set into multiple tasks using filters. For example, a user may want to send files that are from 0 – 100KB in size with one task, and files that are 100KB or greater with another. Or one may segment the files/tasks based on wildcard or regular expressions. By dividing the files into several tasks and using the single archive option only for the smaller files, or files matching certain expressions, one can gain more efficiency because several archives are being built and extracted concurrently.

Coming this September, FileCatalyst HotFolder will have the ability to transfer multiple files simultaneously within a single task. This will allow the transfers to be broken down even further. It will also be able to pick up new additions to the folder users are transferring in real time, even when well into an existing transfer.

Another advancement coming in September is a drastic increase in the speed at which FileCatalyst recursively builds the file list to be transferred, specifically when the files reside on a Windows network share. For example, in the current version of FileCatalyst HotFolder, it may take 10 – 15 minutes to build a list of about 200,000 files on the network. In the new version it takes only 8 seconds!

These capabilities can scale into the millions of files, territory where other solutions start to choke, and may take several days to complete.

So although FileCatalyst is known for its ability to transfer very large files at line speed, it is also extremely efficient at transferring massive amounts of small files, with more great advancements coming soon!