# Julia + elixir **URL:** \ **Category:** Discussions\ **Tags:** interoperability, programming-languages, julia\ **Created:** [August 30, 2020, 9:40pm UTC](https://forum.elixirforum.com/t/julia-elixir/34064 "2020-08-30T21:40:40Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![ityonemo](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/ityonemo/32/11341_2.png) [@ityonemo](https://forum.elixirforum.com/u/ityonemo)\ **Post date:** [August 30, 2020, 9:40pm UTC](https://forum.elixirforum.com/t/julia-elixir/34064/1 "2020-08-30T21:40:40Z")
Spinning this off in a new thread because I think it’s interesting and merits its own discussion. Agree with @hauleth that the basis should be spinning up a c-node inside of julia, but it’s going to need a bit more than that. A few questions: - How to manage Julia versions to run on - Julia has a --worker node setting, should we use that? - How do you load up the correct dependencies? Also, julia annoyingly uses global dependencies, more like python than elixir, which has project-level dependencies. - Is there something really high performance that one could do like memmapping memory segments and passing data across that way? How do you detect if that capability exists? (windows? Linux? MacOS? etc) Separately, would anybody be interested in doing a collab on this for SpawnFest 2020? ---
**Author:** ![derek-zhou](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/derek-zhou/32/19943_2.png) [@derek-zhou](https://forum.elixirforum.com/u/derek-zhou)\ **Post date:** [August 30, 2020, 7:20pm UTC](https://forum.elixirforum.com/t/julia-elixir/34064/2 "2020-08-30T19:20:42Z")
> [@On "Why Elixir?"](https://forum.elixirforum.com/t/on-why-elixir/34038/26): > > parse a poorly structured CSV that came out of a shell script dropped from an ETL pipeline, into a high performance pipeline of matrix-matrix multiplication on a GPU There is Julia for those kinds of things. ---
**Author:** ![ityonemo](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/ityonemo/32/11341_2.png) [@ityonemo](https://forum.elixirforum.com/u/ityonemo)\ **Post date:** [August 30, 2020, 7:29pm UTC](https://forum.elixirforum.com/t/julia-elixir/34064/3 "2020-08-30T19:29:38Z")
OK, I’m slightly biased because I came to Elixir directly from Julia because I didn’t think Julia had a “deploying real user-facing things” story despite being 1.0… maybe I’m the only one, ha. Julia matches extremely well with the academic mindset and the mindset of machine learning professionals (even though some of them don’t know it yet), which is why it’s in the beginning stages of blowing up in that field. It also meshes really well with elixir in terms of its programming paradigms (functional, escape hatches for statefulness, capitalized modules, secretly a lisp). I think stronger julia ↔ elixir interop is a thing that could be really amazing, and is currently an unsolved problem. I also know that there have been lots of rumblings in the elixir community about wanting this… ---
**Author:** ![derek-zhou](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/derek-zhou/32/19943_2.png) [@derek-zhou](https://forum.elixirforum.com/u/derek-zhou)\ **Post date:** [August 30, 2020, 8:42pm UTC](https://forum.elixirforum.com/t/julia-elixir/34064/4 "2020-08-30T20:42:53Z")
I only started learning Julia a few days ago. So far, I think it is good at what it is supposed to be good at, not much for other things. For one the core api documentation is no as easy to navigate as elixir’s. I wholeheartedly agree better interop between elixir and julia would be beneficial to both communites. ---
**Author:** ![hauleth](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/hauleth/32/18942_2.png) [@hauleth](https://forum.elixirforum.com/u/hauleth)\ **Post date:** [August 30, 2020, 9:03pm UTC](https://forum.elixirforum.com/t/julia-elixir/34064/5 "2020-08-30T21:03:41Z")
I think (I haven’t tested this out) that running Julia as C-Node would be fairly straightforward. ---
**Author:** ![srowley](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/srowley/32/28720_2.png) [@srowley](https://forum.elixirforum.com/u/srowley)\ **Post date:** [August 30, 2020, 9:18pm UTC](https://forum.elixirforum.com/t/julia-elixir/34064/6 "2020-08-30T21:18:33Z")
I would love to see really strong Julia interop and for Julia to adopt Elixir’s expectations/standards for high-quality documentation and tooling. Those two languages compliment each other really well. But I also agree that a lot of the user base is usually happy to just hack something together in Python and call it a day. An Elixir/Julia alternative could possibly occupy a niche for applications that need to be durable and therefore more thoughtfully engineered. ---
**Author:** ![hauleth](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/hauleth/32/18942_2.png) [@hauleth](https://forum.elixirforum.com/u/hauleth)\ **Post date:** [August 31, 2020, 9:35am UTC](https://forum.elixirforum.com/t/julia-elixir/34064/7 "2020-08-31T09:35:18Z")
> [@ityonemo](#): > > How to manage Julia versions to run on > [@ityonemo](#): > > How do you load up the correct dependencies? Why should we care? Really. This is all on the “Julia developer” side, not on “Elixir developer” side. > [@ityonemo](#): > > Is there something really high performance that one could do like memmapping memory segments and passing data across that way? If such integration would use UNIX sockets for communication instead of C-Node (fairly easy to do with `socket` module) then one could pass `mmap`ed FD to the second process. However IIRC this is still not available on Windows (even with introduction of UNIX socket on Windows). > [@ityonemo](#): > > Separately, would anybody be interested in doing a collab on this for SpawnFest 2020? Seems interesting. You can count me in. ---
**Author:** ![LostKobrakai](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/lostkobrakai/32/3072_2.png) [@LostKobrakai](https://forum.elixirforum.com/u/LostKobrakai)\ **Post date:** [August 31, 2020, 9:40am UTC](https://forum.elixirforum.com/t/julia-elixir/34064/8 "2020-08-31T09:40:25Z")
I feel like for many questions one can also look at things like `rustler` for how integration with elixir/mix/rebar3 can look like. ---
**Author:** ![ityonemo](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/ityonemo/32/11341_2.png) [@ityonemo](https://forum.elixirforum.com/u/ityonemo)\ **Post date:** [August 31, 2020, 3:00pm UTC](https://forum.elixirforum.com/t/julia-elixir/34064/9 "2020-08-31T15:00:06Z")
Well im also the author of zigler, so this won’t be my first rodeo with elixir language interop. ---
**Author:** ![ityonemo](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/ityonemo/32/11341_2.png) [@ityonemo](https://forum.elixirforum.com/u/ityonemo)\ **Post date:** [August 31, 2020, 3:29pm UTC](https://forum.elixirforum.com/t/julia-elixir/34064/10 "2020-08-31T15:29:49Z")
> Why should we care? Really. This is all on the “Julia developer” side, not on “Elixir developer” side. I think if you’re going to want to write an app that you deploy somewhere “in prod”, you are not going to want to struggle with, say virtualenvs. I’ve discussed this with the Julia founders, and honestly, the current Julia deploy strategy for solving this is “just containerize it”. Since Elixir is the more opinionated system, and acting as the ‘orchestrator’ anyways, I think it’s reasonable to take the position that in a mixed language deployment, (that will necessarily have to share container, for example), Elixir should be the “responsible parent” and manage those sorts of things. I _think_ julia has an escape hatch where you can set the deps path, so forcing julia deps into the elixir project tree is “the right thing”. ---
**Author:** ![derek-zhou](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/derek-zhou/32/19943_2.png) [@derek-zhou](https://forum.elixirforum.com/u/derek-zhou)\ **Post date:** [August 31, 2020, 3:52pm UTC](https://forum.elixirforum.com/t/julia-elixir/34064/11 "2020-08-31T15:52:01Z")
> [@ityonemo](#): > > Elixir should be the “responsible parent” and manage those sorts of things. I _think_ julia has an escape hatch where you can set the deps path, so forcing julia deps into the elixir project tree is “the right thing”. Agreed. Just don’t make it overly complicated and hidden. Also if people want to stick to the global setting of the Julia environment they should be able to opt out of this. ---
**Author:** ![hauleth](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/hauleth/32/18942_2.png) [@hauleth](https://forum.elixirforum.com/u/hauleth)\ **Post date:** [August 31, 2020, 4:30pm UTC](https://forum.elixirforum.com/t/julia-elixir/34064/12 "2020-08-31T16:30:37Z")
> [@ityonemo](#): > > that will necessarily have to share container, for example That is the place where we disagree. Each of the applications should have their own containers and do not share stuff if not needed. I need to check, but I think that sharing FDs via UNIX socket could work between containers. > [@ityonemo](#): > > Elixir should be the “responsible parent” I do not think that we should patronise such applications. From the viewpoint of the Elixir developer this may seem like a good idea, but from the Julia developer, who is accustomed to their own tooling, this may not seem like something worth their time. > [@ityonemo](#): > > I _think_ julia has an escape hatch where you can set the deps path IIRC there is such possibility. > [@ityonemo](#): > > so forcing julia deps into the elixir project tree is “the right thing” I wouldn’t say so. I think that there should be library for the two projects to communicate with each other, but making Julia part of Elixir deployment is not worth the hassle in most cases. Julia (like R) is designed to be simple and easy for scientists, and they want to be able to just `Pkgs.install("foo"); using Foo;` and call it a day. Managing that from within Elixir would be IMHO hell of unneeded journey (as you would for example need C and Fortran compilers in your machine, that would need to be managed by Elixir as well). ---
**Author:** ![ityonemo](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/ityonemo/32/11341_2.png) [@ityonemo](https://forum.elixirforum.com/u/ityonemo)\ **Post date:** [August 31, 2020, 4:48pm UTC](https://forum.elixirforum.com/t/julia-elixir/34064/13 "2020-08-31T16:48:24Z")
> [@hauleth](#): > > I need to check, but I think that sharing FDs via UNIX socket could work between containers. I think that it is possible, by hook or by crook to share FDs through containers via unix socket. Do CNodes work between containers? Also, I think you would want your Elixir app to supervise the Julia processes and julia can certainly go rogue and hog its own thread management to the point where it wouldn’t be able to service a kill request over the cnode channel. If they’re in the same container, the elixir vm has the “out” of being able to slaughter julia at will. In any case it would be easier to not bother with the dependencies so maybe I’m overthinking it, but I think that the target audience for a julia + elixir combined is not a julia developer that is a data scientist or academic researcher, it’s an elixir developer that wants to deploy something that a data scientist or academic researcher has written, or a data scientist or academic researcher that actually cares about sane deployment. So those problems are going to have to be solved. ---
**Author:** ![hauleth](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/hauleth/32/18942_2.png) [@hauleth](https://forum.elixirforum.com/u/hauleth)\ **Post date:** [August 31, 2020, 5:15pm UTC](https://forum.elixirforum.com/t/julia-elixir/34064/14 "2020-08-31T17:15:17Z")
> [@ityonemo](#): > > Do CNodes work between containers? These do. C-Nodes are treated the same way the regular “Erlang nodes”. So these even works over network (additional advantage - you can run your Erlang application on simpler machine, and run Julia code on GPU-backed beast). > [@ityonemo](#): > > I think you would want your Elixir app to supervise the Julia processes Absolutely not, this is why we have system supervisors like systemd. Erlang isn’t the best solution out there for managing external processes, especially when you run it as unprivileged user. This also solves problem of “unresponsive node” you mention. > [@ityonemo](#): > > If they’re in the same container, the elixir vm has the “out” of being able to slaughter julia at will. No if Julia would for example hog RAM. From within Elixir you also cannot run application in cgroup or extensively limit it capabilities (as running Erlang VM as root is generally poor idea). ---
**Author:** ![derek-zhou](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/derek-zhou/32/19943_2.png) [@derek-zhou](https://forum.elixirforum.com/u/derek-zhou)\ **Post date:** [August 31, 2020, 5:17pm UTC](https://forum.elixirforum.com/t/julia-elixir/34064/15 "2020-08-31T17:17:13Z")
> [@hauleth](#): > > Managing that from within Elixir would be IMHO hell of unneeded journey Agreed on the no reinventing the wheel. A julia project within an Elixir project, within the same git repo, would be nice though. Sort if like how a JavaScript project lives in a phoenix project. The julia sub project can still be autonomous. I would love to use exunit to drive regression tests for julia. ---
**Author:** ![hauleth](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/hauleth/32/18942_2.png) [@hauleth](https://forum.elixirforum.com/u/hauleth)\ **Post date:** [August 31, 2020, 5:21pm UTC](https://forum.elixirforum.com/t/julia-elixir/34064/16 "2020-08-31T17:21:42Z")
> [@derek-zhou](#): > > A julia project within an Elixir project, within the same git repo, would be nice though. Nothing prevents you from having both codebases in one repo. That doesn’t change there to be honest. The difference will be how you launch and manage components of the application. ---
**Author:** ![derek-zhou](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/derek-zhou/32/19943_2.png) [@derek-zhou](https://forum.elixirforum.com/u/derek-zhou)\ **Post date:** [August 31, 2020, 5:49pm UTC](https://forum.elixirforum.com/t/julia-elixir/34064/17 "2020-08-31T17:49:57Z")
> [@hauleth](#): > > The difference will be how you launch and manage components of the application. Ok. Ideally I want to: - launch the elixir component by itself using standard elixir method for debuging - launch the julia component by itself using standard julia method for debuging - launch elixir+ julia components together using standard elixir method for production and testing ---
**Author:** ![dokie](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/dokie/32/2015_2.png) [@dokie](https://forum.elixirforum.com/u/dokie)\ **Post date:** [September 3, 2020, 4:53pm UTC](https://forum.elixirforum.com/t/julia-elixir/34064/18 "2020-09-03T16:53:40Z")
Joining this discussion late on but wanted to say that we are using both Elixir and Julia in production in a kubernetes cluster communicating back and forth over HTTP api’s. By adopting a functional approach in Julia with no state management we are able to have acceptable performance. ---
**Author:** ![hcs42](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/hcs42/32/20320_2.png) [@hcs42](https://forum.elixirforum.com/u/hcs42)\ **Post date:** [September 15, 2020, 3:43pm UTC](https://forum.elixirforum.com/t/julia-elixir/34064/19 "2020-09-15T15:43:07Z")
We use Erlang+Julia in production. We use Julia’s package compiler to create shared object files that contain the compiled version of our Julia code, its dependencies, and the Julia runtime itself. Then in Erlang we start a new operating system process (erlang:open\_port/2) and run our Julia code. Our Julia code is listening for requests on its standard input and sends the responses to its standard output. (This part was inspired by [https://github.com/hdima/erlport.](https://github.com/hdima/erlport.)) We plan to modify our code to use sockets instead of standard IO. ---
**Author:** ![ityonemo](https://forum.elixirforum.com/user_avatar/forum.elixirforum.com/ityonemo/32/11341_2.png) [@ityonemo](https://forum.elixirforum.com/u/ityonemo)\ **Post date:** [September 19, 2020, 3:04am UTC](https://forum.elixirforum.com/t/julia-elixir/34064/20 "2020-09-19T03:04:54Z")
Thank you both very much for your valuable insight. I wonder if @dokie would benefit from something a bit less burdensome than all-out http parsing. @hcs42 I’m surprised and impressed that statically compiled Julia works well enough to do that, static compilation has come a long way from the 1.0 days I guess! I just quit my job and am evaluating what to do next. I think it won’t be an elixir-julia bridge, but that would be second on my list. [Next page](https://forum.elixirforum.com/t/julia-elixir/34064.md?page=2)