Two People columns and Two Number Columns in Stacked Bar Chart

I’m looking to develop a stacked bar chart using to people columns and two number columns. The number columns need to link to one specific people column in the report. Here’s how they should link.

BAU People Column - BAU Number Column
NH People Column - NH Number Column.

Any person could be assigned to one or both of these people columns for the project. What I am finding is I cannot specify the linkage to these two column. Meaning if Drew is Assigned the BAU and Jared the NH. The bar chart will add the numbers from both the BAU column and the NH column for both Drew AND Jared. I’ve tried filters, I tried all the x and y options. I cannot seem to get them to calculate correctly.