No don't give up!
If you really know that this is what you wan't to do-do it!Don't worry if it seems that people know more than you,it doesn't matter.You will learn like myself-i was 16 & absolutely knew NOTHING about cars when i started out,in a few years i was a qualified mechanic!
And there are still things i don't know how to do and everyone is like this.
Nearly 10 years later and i still enjoy it and still learn everyday,also get to work and drive some of the coolest cars around.
The best way to start is to get ya hands dirty!start working on an old engine and try and figure it out.
GOOD LUCK