Hi I like the switch mod
Me too, shame it didn't work!
How about a belt to the outside of the banjo, since you've got that round? a vee belt will work on that much wood surface area without need for a vee. A small rod in wood bearings for an arbor (jackshaft) and a small vee-belt pulley between them at the other end. A couple of machinery bushings (stop collars) on the arbor to trap it. Chuck the arbor end. That takes the side load off the drill/motor.
The outside isn't really round yet. I was going to try cutting the outside first, but decided it'd probably be better to try get the thing more balanced by cutting the inside first. Plus it'd get in the way when I eventually get around to cutting the outside. I did consider a jack shaft to reduce the rpm but that was starting to get complicated.

I ended up pinching the speed controller from the jigsaw. If it doesn't catch fire I shouldn't have any trouble putting it back when i'm finished. The drill still has runaway speed, but it's much more adjustable than just with the trigger, so i'm able to fiddle with it and the speed of cut so that it maintains a steady and slower speed when working.

Using the battery drill to control the cut. Much easier than the crank and it doesn't sway the slide like when cranking it.

Managing a fairly heavy cut, about 3-4mm.

It's leaving a fairly okay cut. Bit hairy on where its cutting into the endgrain but it isn't tearing chunks out and its nothing a bit of sanding can't clean up.
Had a bit of a weird day today and didn't get alot done. I probably could've cut more but at the time I was thinking I might regrind the tool to reach all the way to the end. But I just realised that I probably wont need to cut it all the way since it can be cleaned up with one of those flush cutting router bits.
Also, just for the record, i'm switching the thing off when fiddling with the speed controller since it's all very exposed. If it was a more permanent setup i'd have built an enclosure for it.