What is the fastest way to split a large string into multiple chunks by size e.g breaking 10MB long string into multiple chunks of 5KB each?

Use binary pattern matching together with list comprehension: for <<chunk::size(chunk_size)-binary <- input>>, do: chunk

@hauleth I did a quick test and it seems like this method will throughway a leftover chunk. for <<chunk::size(5)-binary ← “hello world”>>, do: chunk [“hello”, " worl"]

I found an answer from StackOverflow which goes like this: "hello world" |> String.codepoints |> Enum.chunk_every(5) |> Enum.map(&Enum.join/1) # ["hello", " worl", "d"] However, I not sure is there any performance implication here that I should take into consideration.

[image] csokun: However, I not sure is there any performance implication here that I should take into consideration. well, you could use Stream.chunk_every, so it would lazily chunk your string instead of eagerly. For long strings like you said you are working with, it should improve performa…

The stackoverflow is likely to be quite a bit slower. handling utf8 is slow, but if not needed: Here is a version based on the list comprehension but which takes the leftover into account/ defmodule Chunker do alias Chunker def chunk(string, size \\ 5), do: chunk(string, size, []) defp c…

Hi @kelvinst I tried to port a method createFileFromText from azure-storage npm to my ex_azure_storage lib. So to be honest I don’t know where the string comes from :slight_smile:

Random thought here but if how UTF-8 uses a variable number of bytes bothers you or causes issues, you could use UTF-32 which is fixed-length! Alternatively you might have to implement something which is smart enough to know how many bytes each grapheme requires. You might want to check out elixir/…

@cmkarlsson impressive I’ve learnt something today thanks to you :smiley:

I see, yeah, I was just curious cause normally big amounts of data like that come from files or uploads, that could be streamed themselves, avoiding to load the whole thing to memory. But in your case, as it’s a ported lib function that takes a loaded string, you don’t much control over that, so yea…

Elixir Programming Language Forum

How to split string into multiple chunks by size

Questions & Help Questions

dimitarvp May 13, 2021, 3:15pm 3

This could produce invalid Unicode strings though?