What format is this data? You said it is dynamic but ‘how’? You’d need to know some bounds on how to split it, at the very least if a ‘section’ of data cannot be more than, oh, 1 meg then you can split at 1 meg boundaries with each section parsing 2 splits at a time with overlap handling to the next or so. Or if you do not want to parse a single file concurrently but rather serially then you can get data in chunks and parse on the fly in whole, or…?
