The implementation is currently checking for the presence of a limit in the query.
When a limit is applied, the query used to load the association is run in a lateral subquery, which would have the effect of applying the distinct separately to each parent 
.
If you’d like to send a PR that ensures distinct queries are always applied per-parent, the code that needs updating is: here and here
