It might work without the olives. However, the olives didn't taste as "olivey" after stewing in the pan, and the chicken didn't take that much flavour from them, either.

It might be a risk to try it, but I suspect that it might work for him so long as the olives don't make it to his plate (or are placed on only one side of the pan).