Its more likely to damage the usually softer material that they are inserted into. The reason for the knurl is to make them slightly over diameter so the grip in the hole. Removing and refitting will wear the hole so they won't grip as well each time they are re inserted.
J