How to remove a broken intermediate oil pump drive shaft

The oil pump on my '98 explorer sport 4.0 sohv engine ceased and the shaft broke. i have it all apart new shaft and pump ready to re-install but I can't get the upper half of the shaft out of the engine. Any idea? I can't see if the oil filter mount can be removed the exaust is in the way. any kind of help would be great since the hayes book doesn't give anykind of info at all.