This is linked to how functions are differentiated in the BEAM. Functions are differentiated by the module they reside in, their name and arity (number of arguments the function accepts). As such serialize/1 is the same function and the various function clauses a contraction of an if-else.
If having these 3 cases are import the define the 3 different functions in the behaviour individually e.g. serialize_nil/1, serialize_lists/1, serialize_item/1. The behaviour can auto-generate a serialize/1 which calls the relevant callbacks.
If you have many different structs perhaps look into using Protocols.
