You could not use buttons outside of forms. Then you don’t need the JS and your website will work without JS. Also phoenix_html.js, which defines what you described, is tiny (<3kB uncompressed): phoenix_html/priv/static/phoenix_html.js at main · phoenixframework/phoenix_html · GitHub
phoenix.js is much larger because it’s powering the client side part of phoenix channels. If you’re not using channels on the client you should be fine to exclude phoenix.js.
