Checkout docker 19.03 (not GA as of this comment but there are release candidates) for secure ways of doing the id_rsa “copy”. Note you can also use .netrc file to use https instead of ssh for cloning from github.
There is a good blog post about using it for ssh and secrets for netrc here https://medium.com/@tonistiigi/build-secrets-and-ssh-forwarding-in-docker-18-09-ae8161d066
You could also use it for hexpm:
RUN --mount=type=secret,id=hex.config,target=/root/.config/rebar3/hex.config rebar3 compile
Then run with with:
$ docker build --secret id=hex.config,src=~/.config/rebar3/hex.config .
There is also a new build cache that allows you to store artifacts like fetched hex deps separate from the image and automatically bring them in at build time. Plus mounting . instead of COPY . ..
I got it working but was hacky to do having rebar3’s _build be stored outside of ./ so it could be cached by the new docker build cache and then you don’t lose your built dependencies or git clones either. I’m sure similar can be done with mix.
I have written all this up with more details and examples but I’m still waiting for 19.03 to be released before publishing 
