Head rest insert

So after struggling with the head rest of the second row seat for ever, the plastic insert with the push lock broke at the neck. I bought the replacement part but cannot figure out how to remove the plastic stem from the seat. It is broken below the leather level and cannot get it out with pliers. Any help would be appreciated.