It depends on the size of your hand. If you have a big hand, then wear a big glove and if you have a small hand ware a small glove or you don't have to worry about thinking and get the "one size fits all" :]
It depends on the gloves. Sometimes the size is not affected at all, just the padding.