I think I finally got it

: You are trying to locate the ER-holder front end with thin cone and back (far) end with draw-bar? If this is the case, please read on and comment, otherwise ignore my ramblings:
1) I hope you are not trusting the draw bar to center end of the collet holder?

* Most likely the thread on the collet holder is off-center and skewed. If the draw bar assembly is too good fit it might bend it out of alignment.
* You want both registers on the same piece (I.E. front cone and rear "ring". You don't want to trust
thread to locate anything radially or axially.
* If I got this right you have three registers: Cone, ER-holder side draw bar and upper side of the draw bar. There is no practical way you get this straight repeatedly. You must put
two registers on the collet holder and allow the draw bar collet holder side have will of it's own.
Believe me on this one one. Thread is great pulling stuff together but thread itself should not generally be trusted to loacte anything if it can be avoided.
2) How true is the collet holder? If you:
* clean up the collet holder, collet and nut
* chuck a piece of ground stock
* and turn the whole assy from ground stock (I.e. center the ground stock on lathe)
DO NOT support the collet holder on the far end - I'm guessing that the trouble is there
How would the collet holder turn out to be?
I think that objective should be to locate the front end of the holder with cone and far end with a "ring". And all these on the same integral part, to keep the stuff concentric. If you make these with sliding fit they will "live" when you put them under load.
I would machine the cone into ER-holder ( I need to bash these chinese holders all the time, because mu milling machine has MT3 with dogs). Measure carefully, but normally they have enough meat in the end. If not see if this could me made out of ER32 MT4 or MT3 holder. The reasoning is that thing "rings" are not great idea under multidirectional loads. They will shift/turn. You either need strong enough "collet" closer to nominal 16 mm ID or such that was the design of this collet type
I also would check the diameter at the end, these collet holders don't always conform standard fits. I would machine there a recess and press strong enough bushing there (either press fit or strong "stud" loctite), and then I would skim the outside of this bushing on the same setting that the cone is machined to keep then concentric.
I would not trust the draw bar to locate the end of the holder. Too many variables. I actually would machine the end that comes next to ER-holder very much smaller diameter to allow the ER-holder far end to locate it's own.
Pekka