Honda CBR 600F4i - How to remove bar ends

I have a 2004 Honda CBR 600F4i...on a recent ride I lost one of my bar ends. I went down to the local Cycle Gear shop and purchased some replacement ends. The problem is that the replacement ends do not secure to the bars int the same manner. The OEM ends were secured to the bar via a bolt...replacement end has a rubber component that gets squished and expands thus expanding and securing itself. I need to remove the component from the OEM bar end but am having difficultly doing so. Does anyone have any advice in how to remove that component?