You're replacing them entirely? Then don't worry about damaging them. Just use a steel punch and a big hammer. Obviously be care with the slide, but don't even worry about the sight itself.
XDs have gotten better, but sometimes still have overly tight sights that will bend the t-handle on sight pushers or even break parts on lower quality and some of the good universal ones too.
EDIT. Wow I didn't even notice this was a couple months old.