Oh if you have good Clojure experience then although the syntax here is significantly different you will pick up the idioms very quickly. You might even be able to start with phoenix and elixir concurrently without much issue. 
Just keep in mind, Phoenix is mostly just a set of plugs, nothing is magical, feel free to look in its source code to see how anything is done.
