Did you increase the spindle speed (RPM) to keep the surface speed up? As you halve the diameter you need to double the RPM for the same cutting conditions, makes a big difference!
The other possibility is the tool's below/above centre height, this has a bigger effect as the diameter goes down - picture the angle from the work's centre to the tool edge, closer in the steeper it gets so if low you lose the effect of the tool's top rake. Similar if above centre, the work rubs on the tool below the cutting edge unless you have loads of relief!
Just my ha'pennorth,
Dave H. (the other one)