Silicon Valley: Is The Pied Piper’s Algorithm The Real Deal?

Silicon Valley, American television sitcom focusing on the lives of six young people launching a startup (in the Silicon Valley, of course), just premiered it’s season two, and we found an awesome little gem for the show’s fans.

To fill you in, the show follows a fictional startup Pied Piper, “A Middle-Out Compression Solution Making Data Storage Problems Smaller”, which is working on a compression solution and trying to make it in the Valley. if you have had any contact with the startup world, I highly recommend the show because it plays with the cliches of the startup community quite beautifully and will have you rolling on the floor laughing, while at the same time being pretty accurate in it’s depiction.

Who Or What Is Pied Piper, Anyways?

But how accurate is it when it coms to technology? Pied Piper (which even has its own web page) is working on deploying an integrated, multi-platform functionality of all applications of their algorithm intended to provide a suite of compression services across diversified market segments.

The solution is built on a universal compression engine that stacks on any file, data, video or image no matter what size. Using our revolutionary “middle-out” algorithm, we find long-range structure in your files to create the most efficient encoding of your data.

…So, Does It Work?

Ethan Chow, entrepreneur, technologist and engineer  who has his own web page EthanChow.Me, analyzed the solution proposed in the show to see how off it is. The paper he wrote, “Written in 5 hours, fueled with existing knowledge, impaired by scotch and sodas” examines the technical implications of a thirty one second clip from the popular television show Silicon Valley.

Primarily, the Hooli employees are false in proposing he’s not using DCT banks, it is clearly documented on the whiteboard that they believe Pied Piper is using discrete fourier transform(DFT) banks. Granted, he is using discrete cosine transform(DCT) to decorrelate the impulse response generated from the initial uncompressed data but for the filter bank he’s applying a DFT methodology. Although this does not rule out the possibility that he is implementing a polyphase filter bank which could mean that the DFT and DCT is working concurrently to optimize but from the scene, only proof of a DFT construct is seen.

Essentially what that means is that the DCT process takes out many of the smaller coefficients while still being a true lossless compression. This results in a reduction of the depth of the search and compression process.

[…]

He’s not using a DFT “spiderweb”, he’s constructed a DFT butterfly which admittedly is less sexy of a term than “spiderweb” but a “spiderweb” is improper terminology.

Understandably he’s developed a oversampled DFT filter bank, it’s most optimal in dealing with high-dimensional, nonlinear data as the fictitious company Pied Piper has noted they do so well in comparison to Dropbox. This data is most commonly represented in formats such as large HD video files or dynamic audio files. Using this methodology, it defends from common noise/error found in encoding and decoding during the compression process such as, but are not limited to: echo, reconstruction error, and computational complexity.

WHAAAAAAAAT?

Didn’t get that? No worries. In conclusion, no, we will not be seeing the Pied Piper algorithm come to life and the show did not actually revolutionize the field of compression. But they did quite a good job at it, says Ethan:

Televisions show will never have the accuracy to satisfy an engineer but Silicon Valley gave an engineer a good run for his money. However inaccurate these sources of entertainment are, they will continue to be entertainment, especially for this electrical engineer/entrepreneur/software developer.

And that’s something 🙂 You can find the whole paper here, and I strongly recommend you see the show as well!