More advice - thanks!
Jonny, I was going to bolt the rotary firmly to the drill press table (no mill, as such), flip the chuck jaws down and grip the rotary's table in it, indicate it in and then line up between the jaws, drill first hole, then crank the rotary through 90*, drill, rinse and repeat!
I think the existing (rear) holes are a bit small (I'll use M8 capscrews for fixing as that's what the T-nuts take) , and perhaps too close to the edge of the chuck and wouldn't leave enough room for the counterbores - if the drill wandered they could even emerge from the side of the chuck, not the front face...
Thanks again,
Dave H. (the other one)