Yeah, as far as I know there is no way around this due to [frustrating] limitations with Monday’s People column configuration options.
Monday basically views all users added to any/every People column as task assignees—therefore, the item they’re added to appears on their My Work page. 
Here’s a feature request asking for that to be addressed; please upvote it.
This relates to my other huge gripe about People columns: the fact that you cannot add a user who does not have some degree of access to your board (i.e., all users you want to put in a People column have to be given some level of access to the board.)