I have a selection of different sized tools from different machines and I just use whatever seems best for the job in hand. I suppose that the bigger the tool the less problems with vibration. But it often comes down to whatever tool will get into some small space.
Many of my carbide tools are the brazed-on tip variety. I like these because they have positive rake. My replaceable carbides are ex-industrial and have negative rake. Ok for heavy jobs.
Stick with whatever your comfortable with.
Dave.