Emails and activities - why is this attached to "people" column and how can it be set to the 'email' column only?

Hi community, can you help - I had expected from testing and the documentation that the Emails and Activities timelines are attached to an email address in an email column. But further testing is showing that new timeline items are attached to any person/team in a people column as well. Can i stop this at a global level?
Essentially that makes emails and activities completely useless wherever we want to assign an item to a monday user
We want to track emails and activities for clients (items and related items linked by mirrored email field) but we do not want a monday user who handles multiple clients to see all of the activities across their clients in the widget for any client! Same for teams!
I cannot see any standard filter for this! I don’t want to abandon either E&A or assignment of records to people? Please if anyone has a solution i’d be very grateful