I've used ISA-L as the backend implementation for erasure codes in OpenStack Swift.
There are two versions of ISA-L. The open-source one includes some optimized erasure code libraries. The other version is free (as in beer) AFAICT but it requires signing a license agreement with Intel. This version also includes some other optimized libraries for hashing, crypto, and stuff like that.
https://github.com/openstack/liberasurecode is a plugable C library that you can use for erasure coding. It supports some simple EC algorithms itself and works with jerasure and ISA-L.
This appears to be an implementation of Reed-solomon error correction code... at the assembly level. (!! handwritten assembly? I don't see that too often nowadays)
Which is basically the mathematics behind RAID-5, RAID-6, and even hard-drive / SSD sector-level error correction. Parchive is an older open-source project that implements the same concept:
RAID-5 uses simply parity, RAID-6 uses something slightly more complicated, but they are not close in complexity (or capabilities) of Reed-Solomon codes -- although, if there was a Raid Level that had 1024 drives of which any 16 can fail, it would probably employ Reed Solomon codes.
> Several methods, including dual check data computations (parity and Reed-Solomon), orthogonal dual parity check data and diagonal parity, have been used to implement RAID Level 6
> if there was a Raid Level that had 1024 drives of which any 16 can fail, it would probably employ Reed Solomon codes.
Reed-Solomon codes are generalized for any number of drives with any number of failures.
There's a bitwise-efficiency thing with powers-of-two as well. I forgot the details, but I did the math back in college. I'll have to find my notes on this subject... but yeah, the matrix works for virtually everything.
I forgot whether or not Reed-Solomon becomes just a simple xor-parity check in the case of 1-failure, but it wouldn't surprise me if it were related.
