Trouble is if I remove it from the lathe and it doesn't fit because its too tight, I'll have it to
cock I mean clock up again.....

Measure, measure, measure.....I convinced myself that the diameter was going to be ok.....I measured with my trusty caliper, 3.307 inches, measured the spigot on the mill head again, yep 3.307 inches....its got to be right....? Ok, checked with some outside calipers, hmmm...feels good....
So on removing from the chuck, I inched up the mill head, moved it aside and offered up the block....
Oohhh....its gonna be tight...
Well it fit like a glove, if anything its a tad tight, I think there may be some high spots inside the socket,where the spigot fits...well, its in now. I'm happy with the result. All the measuring and careful machining paid off....
I just need to remove it now to tidy up the chucked end, machine a socket in it and add all the tapped or clear holes and the nut pockets.....