Permissions for contact's emails to limit activities showing per contact and per user.

I would like to be able to add account restrictions on the activity visibility from emails sent and received by specific contacts.
We can restrict user visibilities but that is across all boards. I want to be able to limit which contacts show to which users.