I have no idea why FM fails to upgrade the bootloader - try perhaps asking at the FlashMagic forum.
--
You shall program the provided bootloader upgrade binary into block 0, and then run it, and autobaud on it (send 'U' say from a terminal). It reprograms block 1 while running from block 0.

Alternatively, you can program the lower 2000h of it directly to block 1, but you need to zero out the first three bytes (if I remember correctly - I don't have my notes on this topic at hand).
--
You can of course write a short snippet to replace the bootloader and switch the banks to "normal application" configuration.

The P89V51RD2 boots into block 1, so you need to switch to block 0. The best way to do it is to perform a software reset, by writing 1 into SWR=FCF.1 (see chapter 6.2 of datasheet). This is what the default bootloader does, after the initial timeout.

The trouble with your attempt is the following: if you only set BSEL to 1, banks will indeed switch, but after that instruction the instruction from block 0 at address 3 is executed, and not the "ljmp 0" from block 1!

JW