Physical access means that it may not be a good way to take over a huge number of units containing a specific chip.

But stealing crypto keys from a single chip can allow an attacker to make lots of interesting things.

Like cracking game consoles, allowing them to rip games.
Or allowing the extraction of encrypted movies (DVD, BD, ...)
Or allowing someone to tap into communication between other units using the same cryptographic methods and keys.
Or stealing credit card information from point-of-sales terminals.
...

For every chip out in the open, there will be developers who have suitable tools for interfacing.

And some chips may have backdoors that are accessible through normal firmware update channels - lots of equipment have over-the-air updates, or can update from USB thumb drives.

The important thing here is that this isn't the worlds only chip with security issues. A number of chips don't have explicit "backdoors", but have broken copy-protection implementations that gives the exact same end result - people can rip contents, or can modify the chip contents.