It does appear to be some sort of insert. If you seriously want it out, and are willing to deal with whatever shape the hole is in, machine it out. You should be able to drill it a good bit of the diameter, then bore it until it is just a thin shell. Peel it out.