The reset is active high on this part and that is the main point of my comment. If you are willing to accept the use of the crappy resistor / capacitor type (R/C) reset timing circuit then for a high true reset the normal circuit configuration is to put a resistor to GND from the RST pin and capacitor to VCC from the RST pin. I've seen folks use 1uF cap and 47K resistor for this type of reset circuit. My general recommendation though is to not use an R/C reset circuit and use a suitably selected reset generator type chip.

When you try to activate the ISP mode do check that the RST pin is held high from VCC on and kept there by the programming device. Also check to make sure that your XTAL1 and XTAL2 pins are actively oscillating just after the VCC comes on. ISP with this part will not work unless both these conditions are met.

Michael Karas