I'm going to suggest that the Vancouverites book reservations for the 7 AM ferry from Tsawwassen to Swartz Bay. The ride is 1 hour, 35 minutes, and we can meet up with the US crew somewhere in Victoria (where their ferry will land, I believe). I don't know Victoria well enough to suggest a meeting place, Hayley we'll need your expertise for this.
By this schedule we can be downtown by 9 AM-ish, once we hear from the US folks on when they think they can get there, we can see how well this would fit together. I realize it will be an early start to the day, but life is short ...