Delete Calendar Event when Removing Assignee

We are using a Google Calendar Integration, and I am able to create a calendar event when I assign a person to a task. However, I need to be able to delete the event from Google Calendar if I remove the assignee. In the same way, if an assignee is switched to someone else, I need the calendar event to also switch in Google Calendar.

Thoughts on how I can make this work? Seems without this functionality the Google Calendar integration is only half-baked…