It just might be that Wong. You are not getting the correct edge shape presented to the work.
I do in fact use the back cutting face as well. If I see my tram is slightly out, I use that to my advantage, as the return cut (without putting any further cut on) can just take 'wisps' of material off, and that is what gives you the mirror finish. I will run my fly cutter fairly fast, well above 500 RPM most times, but at a very slow feed.
Plus lubrication always works well, especially with ali, as that is liable to pick up on the cutter and swipe it across your nice mirror finish, I use WD40 for lube on ali, brass or phos bronze and cast iron doesn't need it, and for normal steels (I wouldn't recommend a fly cutter on really tough stuff) I just use normal water based coolant/oil mixture from a squeezy bottle. Basically, to obtain a very nice finish, you have to attempt to keep the removed material from being carried around to the next cut, so by using a lube, the fine swarf is thrown away by the liquid being thrown around by the cutter, the ones that require no lube, automatically eject from the surface by themselves.
Bogs