I would like to do some binary pattern matching on very large files (1 - 20gb files). How do I do this using streams? The spec of some of the files I’ll be parsing is dynamic so I can’t always count on something being the same size in the same place every time.

Might need more of an example of what you are wanting to do, like pseudo-code or something? You should be able to just parse it out as normal though. And how large are they?

They range from 1 to 20 gb. Basically I need to pattern match on the binary to extract a massive amount of meta-data. I don’t want to use File.read! because that loads the whole file into memory and my VMs don’t have 20gb of ram.

You are probably looking for File.stream!(filename, [], chunk_size), which creates a stream that you can consume (and pattern match on) in the sizes of the given chunk.

[image] jeramyRR: Basically I need to pattern match on the binary to extract a massive amount of meta-data. I don’t want to use File.read! because that loads the whole file into memory and my VMs don’t have 20gb of ram. What format is this data? You said it is dynamic but ‘how’? You’d need…

[image] Qqwy: You are probably looking for File.stream!(filename, , chunk_size), which creates a stream that you can consume (and pattern match on) in the sizes of the given chunk. Yep this if you know the chunk size, but how ‘dynamic’ is the parsing? EDIT: Also, what are you doing with the…

@OvermindDL1 , here is the format: National Imagery Transmission Format The National Imagery Transmission Format Standard (NITFS) is a U.S. Department of Defense (DoD) and Federal Intelligence Community (IC) suite of standards for the exchange, storage, and transmission of digital-imagery…

Well that looks horrifying… Yeah using a file stream and matching as you get data, when you run out of data for a match then just hold it until you get the next set of data and then pick up where you left off, a genserver listening to the stream will be perfect for that. It looks like a very seria…

The problem is that I have no idea where to set the chunk size for file.stream. As you can see this spec is all over the place.

[image] jeramyRR: The problem is that I have no idea where to set the chunk size for file.stream. As you can see this spec is all over the place. Do whatever is efficient for the data size, 1 meg or 10 megs or whatever all should be fine, I’d just send the data over to another process (a gen…

@OvermindDL1 If you follow this approach, you definitely need to use backpressure. However, I don’t think an extra process is needed here; we already have one (besides the current process), namely the process owning the file. An alternative approach: Pick a chunk size that is larger than the larg…

Binary pattern matching when input is a stream

Questions & Help Questions

jeramyRR March 24, 2017, 8:01pm 7

@OvermindDL1, here is the format:

The first link under The following documents define the standard: will show you the format