Select ZK Proof Advancements Now Integrated in Filecoin
MARCH 19, 2024 - The Decentralized Storage Alliance (DSA) recently released dramatic cryptographic processing improvements that stand to benefit zero-knowledge proof (zk proofs) processing within the Filecoin network as well as throughout the Web3 infrastructure stack. These advances optimized several key computational algorithms, as well as improved CPU and GPU processing flows to reduce cost and improve performance by up to 80%. The advances reduce what used to take many servers and combine it into a single server thereby gaining additional processing efficiencies. Filecoin development teams have integrated select improvements directly into Filecoin, making them accessible to the over 3,000 Storage Providers in the Filecoin network.
The Decentralized Storage Alliance (DSA) brings together technology industry leaders to help enterprises make the transition to decentralized storage technologies through reference architectures, standards, education, advocacy, and best practices. The work was part of an initiative by the DSA to improve performance and energy usage within the Filecoin network and was led by Supranational, a US-based firm that designs and develops hardware accelerated cryptography for verifiable and confidential computing. The team improved the process to onboard data onto the Filecoin network with emphasis on phases of the process that are heavily zk-SNARK dependent, which is a specific type of zk proof.
ZK proof processing is particularly time and resource intensive. And while the Filecoin network already supports GPU acceleration to assist with zk-SNARK processing, these recent advances further optimize the estimated over 100,000 GPUs in use – representing around $100M of deployed capital – operating within the Filecoin network.
Filecoin Network is the Highest User of ZK-SNARKS in Web3
ZK-SNARKs allows one party to prove to another that a certain statement is true, without revealing any information about the statement itself. Zk proofs in general, and zk-SNARKs in particular, have become fundamental building blocks for storing trusted and verifiable data within blocks and data sectors contained within blockchains.
ZK-SNARKS are heavily used in the sealing process of the Filecoin network, which is the process by which data gets onboarded or loaded on the network in a cryptographically secure, but verifiable manner.
Every day, over 1.7 million zk-SNARK proofs are created and verified on the Filecoin network – making the Filecoin network the highest known user of zk-SNARKs in the world and which is larger than Ethereum, zCash, Mina and many other networks combined.
This evidence is derived from gas metrics from FilFox, a Filecoin Network block explorer, which shows ~170,000 ProveCommitSectors operations and ~90,000 SubmitWindowedPoSt operations. The first operation processes 10 zk-SNARKs each and the second processes an additional zk-SNARK – the total of which then adds up to 1.7+M zk-SNARKs.
Optimization of ZK-SNARK Computational Primitives
A key to achieving the 80% reduction in cost and latency in generating Filecoin’s SNARKs was the use of improved algorithms and software implementations. Optimizations to the GPU algorithms were enabled for the specific computations that are being employed in the Filecoin SNARKs. These include multi-scalar multiplications (MSM), number theoretic transformations (NTT), and Poseidon hashing, three core computations found in many zk proof systems.
- Multi-Scalar Multiplications (MSM) – MSM is a key computational primitive in zk proofs. The algorithm is used to calculate the sum of multiple scalar multiplications, and it is often used to commit to a polynomial in SNARK systems. MSM operations were accelerated through the use of optimized assembly that improves the efficiency of large integer arithmetic.
- Number Theoretic Transformations (NTT) – NTT is a mathematical approach that facilitates efficient polynomial multiplication. This implementation makes use of Fast Fourier Transform-style algorithms giving the algorithm a computational complexity of O(nlogn). NTT operations were accelerated by developing a more efficient implementation of the core NTT algorithm.
- Poseidon Hashing – Poseidon is a ‘SNARK-friendly’ cryptographic hash function. It is often used in SNARK systems as it requires up to 8x fewer constraints per message bit than previous ‘SNARK-friendly’ hashes such as the Pedersen hash. Poseidon hashing was accelerated through an improved kernel implementation that leverages more efficient data transfers.
This work advanced the state-of-the-art performance for Groth16 proof generation. Groth16 is a widely used proof system throughout the blockchain ecosystem and as such this project represents a breakthrough in the performance of ZK cryptography more broadly for blockchains and beyond. The work is open sourced and can be used to accelerate other zk proof-based systems.
Sealing/Onboarding Data on the Filecoin Network
Two phases of the sealing or data onboarding process on the Filecoin network rely heavily on zk-SNARKs in their operations. One phase computes a number of zk-SNARK proofs, and the other verifies them as part of the continual proof phase.
Whereas the term “block” is used for blocks of transaction in transaction-based blockchain networks, the “sector” is used for data stored on the Filecoin network. Before a sector can be onboarded onto the network, a Storage Provider within the network must seal the sector or in other words, encode the data in the sector to prepare it for the proving process.
Sealing a sector is done via a Proof-of-Replication process which is a computation-intensive operation that results in a unique encoding of the sector. Once data is sealed, Storage Providers (1) generate a proof-of-replication, (2) run a SNARK on the proof to compress it, and (3) submit the result of the compression to the blockchain as a certification of the storage commitment.
Project Origins and Team
Supranational, the developer behind these performance improvements, is a long time contributor to the Filecoin ecosystem. Supranational develops blst, the core cryptography library used in Filecoin’s consensus protocol, and also assisted with initial optimizations to Filecoin’s zk proof system before the network’s launch.
Recently, the team has been working on sppark, a library designed to accelerate a variety of zk proof systems. The sppark library is leveraged in the recent improvements to Filecoin’s proving system, and is also used by other Filecoin ecosystem participants including Lurk Lab’s implementation of the Nova proof system.
“The software optimization work for this project was extensive,” said Kelly Olson, one of the leaders of the team, “While there is always room for additional improvements, the current software is quite efficient and the cost of producing these proofs is low enough that it is no longer a concern for storage providers.”
Release and Availability of the ZK-SNARK Enhancements
All Storage Providers can now take advantage of select improvements that have been integrated directly into Filecoin.
“These enhancements will reduce costs for the entire network, resulting in further cost savings for end users of decentralized storage technologies" said Daniel Leon, Founding Advisor of the DSA.
Storage providers are able to purchase common single-server hardware solutions from vendors with these enhancements included. These servers are plug-and-play ready meaning that they can be powered and connected to the network and perform sealing operations with limit to no customizations.
Future Possibilities
Building off of these enhancements, Supranational, in collaboration with Protocol Labs and the DSA, have defined a ‘SNARK-as-a-Service’ API which will allow these proofs to be outsourced to providers who specialize in proof computations, thereby further simplifying operations and enabling specialization in the Filecoin ecosystem.
This type of zk-SNARK compute specialization is one of the firsts of its kind in the Web3 / blockchain space but is likely to be replicated across networks that are heavily zk proof dependent. Different algorithmic enhancements and circuit-specific implementations might have to be employed but the model for doing so using GPUs is one that holds great promise for reducing costs and compute times across the entire Web3 ecosystem
**
About Supranational
Supranational designs and develops hardware accelerated cryptography for verifiable and confidential computing. The team has decades of experience in algorithmic optimization across CPU, GPU, FPGA, and ASIC platforms. Accelerated cryptography developed by Supranational is currently in production in blockchain networks such as Ethereum, Filecoin, Optimism, Polygon, Aptos, Sui, Chia, and more. For more information visit https://www.supranational.net/ or e-mail hello@supranational.net.
About Decentralized Storage Alliance (DSA)
The Decentralized Storage Alliance (DSA) is a professional organization launched in October 2022 by a community of companies who imagine a better path to storing data through decentralized storage solutions. Its charter is to drive industry growth, bridge the gap between Web2 and Web3 storage technologies, and highlight advances in technology an
Daniel León: Dec 17, 2024
Ken Fromm: Nov 25, 2024
Daniel León