The screw was a tiny socket set screw which I didn't find until afterwards.
You can make a split puller (thought it is more of a pusher) to get the bearing out without removing the shaft. I've used someone else's to do that, my own version was made to do it with they shaft out.
I don't like the sound of all this bashing lark, you don't want to loosen the P bracket mounting in the hull.
