Your situation is complicated by the project manager's status as an outside consultant who might be reluctant to take the risk of damaging his relationship with your staff members by participating in such a meeting (and who might, quite reasonably, point out this is outside the scope of his current contract). Nonetheless, I don't see any alternatives. A conversation that doesn't include the employee strikes me as cowardly and would probably strike an employee as violating the legal principle of having the right to face your accuser (as it were).
If you didn't set this expectation at the beginning of the project, it's a bit more problematic, which means right now would be a good time to announce it, explaining it as a just-too-late idea that's better than not having it at all. Not doing it means reporting managers would have to assess employees' performance without knowing how they performed -- a far worse outcome.
My best suggestion is to explain the situation to everyone involved, letting them know when the first occurrence will take place. Since it's a retrofit, I'd suggest giving everyone at least six weeks to prepare.
One more thought: There's a good chance that, as first-timers, no one in your group will do this well, so you might consider sitting in on the first round to help facilitate proceedings and smooth over the inevitable rough edges.
This story, "Don't let matrix management exclude employee performance," was originally published at InfoWorld.com. Read more of Bob Lewis's Advice Line blog on InfoWorld.com.