@OvermindDL1 If you follow this approach, you definitely need to use backpressure. However, I don’t think an extra process is needed here; we already have one (besides the current process), namely the process owning the file.
An alternative approach:
- Pick a chunk size that is larger than the largest piece you need to match on.
- Keep track of a ‘buffer’ bytestring.
- You can pattern match on this buffer’s front, and chop the parts of that are consumed.
- Whenever this buffer becomes shorter than the chunk size, you append the next file chunk to it, and in this way consume the file lazily.
