Mazda 3 shaking semi violently around 4th gear

I have a 2008 mazda 3 (Automatic with sport shift), and have noticed that around 4th gear (in automatic) the car shakes semi violently. This also occurs around the 5th gear as well. However, if I let off of the accelerator ever so slightly, the shaking stops. This doesn't always happen, but some days it occurs quite frequently, and others it does not even happen. Does anyone have any idea what might be causing the issue?