Is that a tapered pin? If not, is it deformed (peened)? Still amazed the pin is so tight. Seems to me like you need to make an anvil to support the work. Maybe you could use a plain old V block, it may provide enough clearance in the notch underneath the pin to get it started. My first instinct is to try a pin punch.