You could use an additional intermediate table as a “handle” to the many kinds of things:
user_things_and_affinities:
user_id
thing_id
affinity_id
things:
type
building_id
tool_id
location_id
person_id
activity_id
etc
This way, for M users and N things there are only N rows with mostly NULL instead of O(M*N).
You could also copy some data to the things table from each kind of record - for instance, a “display name” so that UI code doesn’t need to load the associated record just for a name.
