Why can't I do configuration in runtime.exs in my dev environment?

How are you reading the config value? If you’re reading it during compile-time, then it’s baked into the app and trying to modify at runtime will have no effect.