Nothing really wrong with it in my book, fluffy is right about the feed back of the mill verses the drill press.
If money and or space are the concern then the mill has to stay and the drill press goes, you can’t mill on a drill but you can drill on a mill.
Another advantage im sure your aware of is the X Y feed table, unless you have a stage jig, drilling on the press is always going to be a manual set up.
I use machines that are designed for a certain job and find ways for them to perform another job all the time, its all about adapt and over come routine.
Now I slightly abuse my drill press now and again, as I am sure a lot of us do, I would never risk hurting the mill though. Its too precious.
Hope this helps. Anthony.