You should rev match with any down shift to prevent strain & stress on the trans.
If you know how to shift clutches wear more from starting from a stop than shifting.
If your worried about clutch wear, DONT USE IT! With the right technique up an down shifts can be done very smoothly. Do it wrong and count on tearing up a trans and even possibly an engine.