Mastering Metadata Key To Shifting Data Fast, Says Arcitecta

Companies that move and analyze huge volumes of data are always on the lookout for faster ways to do it. One Australian company says it has created a protocol that can "transmit terabytes per minute across the globe."

The company, Arcitecta, which has an almost 25-year history, has just announced the new Livewire protocol, which is part of their data management platform, Mediaflux, used by institutions including the Australian Department of Defense, drug maker Novartis, and the Dana Farber Cancer Institute.

According to CEO Jason Lohrey, Livewire itself has already made an impact for some of the largest data movers. "One of our customers transmits petabytes of data around the globe, he told The Register.

The Livewire transmission protocol creates a native protocol tunnel through HTTP/S or TCP/IP, which won the company a pair of awards for its performance at SupercomputingAsia's 2021 Data Mover Challenge.

Several teams competed to shift the largest amount of data in the most efficient manner via interconnected multinational 100GbE networks, with a mind to scaling to future network speed increases. 

Moving lots of data quickly from one datacenter to another is an increasingly relevant challenge as ever growing amounts of data is generated. While many industries find computing at the edge a solution to the problem of needing to move data, such a strategy isn't always the best solution. 

Parallelization is the key that Lohrey said makes Livewire able to move data so quickly. That parallelization is a key component of Mediaflux, which manages data in such a way that it can be split among virtualized storage spaces.

• Oracle users fail to get that moving apps to cloud means business transformation – Gartner
• Insteon's vanishing act explained: Smart home biz insolvent, sells off assets
• AWS CEO: We're not spinning out, likely to seek acquisitions
• Yandex speaks out from front line of Western sanctions against Russia

Every aspect of the data pipeline, Arcitecta said, is executed in parallel using Livewire. That includes storage, which reportedly resulted in one Mediaflux/Livewire transmission being clocked at 65TB per hour – off tape.

"Mediaflux Livewire leverages the power of metadata to optimize data movement via parallelized data transfers across limited, highly-contended or latent networks and eliminate redundant file transfers," the company said in a statement.

Because Livewire is the protocol, it isn't restricted to Mediaflux and can operate as a standalone product as well. Pricing for the standalone version of Livewire or Mediaflux weren't immediately available.

The Data Mover Challenge included several other winners who were all working on solutions to the same problem. Among them were Team MUSASHINO, which was awarded most innovative solution by inventing a new internet protocol, and Ciena-iCair-UETN, which was declared the overall winner for creating a framework that used machine learning to optimize a data transfer framework. ®