Yes. You have it right here.

Not only the code tends to become larger and in most cases a tad slower but also the code itself (the library code) tends to grow a very hairy look.

It is really annoying to try to read some highly portable code which is stuffed up to throat with #ifndef ... #elif and #whatever directives. Unfortunately those are unavoidable if one intents to make portable code. Those are a nightmare to debug....

The trick is to isolate the MCU dependant parts and then design the library in a way that those are supplied by a separate set of source/object code. This way the main logic stays free of #directive fiddlings and the port is easier to adopt to a new MCU.

One example (again) would be the LCD driver library. The MCU dependent parts are those that manipulate the control and the data wires. So there will be a set of functions that do only (and only) that. So we have one function that initializes the wires, another that sets the control wires up/down and yet others that read/write the data. One can even leave those out of the library since these are very simple and code them on the main module (making them public). This way one really does not need a single directive in the "business" code.