Most people who build a modem using a 8051 don't use it to generate and detect tones. They use a modem chip that may use SPI or similar to interface with the 8051. The 8051 then implements the AT commands you normally use when communicating with a modem. So the 8051 may receive a "ATD 12345" and then order the modem chip to take the line and listen for call progress tones. Then the 8051 waits until timeout or the modem chip detects the tone on the line. Then the 8051 orders the modem chip to generate DTMF tones, one-by-one. Then the 8051 orders the modem chip to listen for call progress again, to try to figure out if the number is busy, or if the other side have picked up the line.

It's only for very slow speeds that it may be meaningful to let the 8051 do tone generation/detection itself. One reason is the dynamic range of the phone line. You have to be able to hear quite weak tones. But at the same time, you may receive strong tones that are mixed with weak tones from another call. And you don't want to generate square-wave tones, so you either have to produce reasonably good sine wave or you need to filter the generated tones.

If detecting call progress, you must be able to detect multiple frequencies at the same time - and estimate their amplitude. If needing DTMF detection, you must be able to listen for 8 different frequencies concurrently - besides any possible need to hear call progress tones.

Simplest is to implement an old 300 baud modem. But they are not so useful, so why spend the time implementing one? Especially since there exists cheap modem chips that also supports negotiation to decide what transfer speeds to use depending on line quality and the capabilities of the other side.