Check the air pressure and yes, swap the tires from side to side. More than likely the tire that pulls has a "dominant belt" in it. There 's no real explanation for it other than there's no fix for it.
When I ran across one in a new tire installation, if it was ok with the customer I'd switch it to the rear, but most of the time we'd switch it for another new tire and send the bad one back as a new defect.
Used tires are always a bag of snakes especially when mixing and matching to get the best out of a set.
Bob