As already said, Assent doesn’t require Pow/PowAssent. It’s a low-level multi-provider framework abstracting away OIDC, OAuth 2.0, and OAuth 1.0. The only dependencies are related to JWT parsing and HTTP client (I strive to keep a minimal dependency graph). PowAssent is build on top of it, and deals with all the complexity of user registration.
I know you said you are going to support password, but for anyone else wanting to do something like this and only want provider authentication I would recommend not using phx.gen.auth or Pow at all. You can just add the user info to the signed/encrypted session cookie (with an expiration timestamp!) or use a JWT when succesfully authenticating, and store whatever provider was used as a cookie. When the session info/JWT expires just redirect to the provider again to reauth. The less surface you add, the better for security.
Finally as for Pow, it’s a complete package. I’m planning a larger rewrite of how it works to improve observability and also have built-in LiveView support. I’m currently updating Pow to support Phoenix 1.7 with all the breakling changes 1.7 introduces.
Unfortunately only got so much time to maintain my open source projects 
