Well that looks horrifying… Yeah using a file stream and matching as you get data, when you run out of data for a match then just hold it until you get the next set of data and then pick up where you left off, a genserver listening to the stream will be perfect for that. It looks like a very serialized format though so no real boost in concurrent reading so this really might be the best way, but it is entirely doable.
