An RTOS are very useful, since it is often very easy to make use of it, and it often gives simpler code to read and maintain.

The reason why RTOS are so very much avoided on 8051 processors is the lack of a "real" stack. With the very limited stack on a 8051, the way to expand the stack is to swap memory in and out - copying stack data to XDATA and restore another stack state.

With more general-purpose processors where all RAM are "the same", you can swap stacks (and decide what stack sizes the different tasks needs) by just modifying one or two registers - so you can perform the stack change at the same time that you switch the contents of the other registers.

The stack used by the different tasks is resources lost in the processor. But in many situations, you will find that the code space needed for the RTOS takes less space than the extra state-machine code you need to add to have a superloop try to time-slice between multiple, often quite complicated, tasks.

An 8051 program is normally very much defined by I/O operations. That means that for an 8051 program, you can normally create an "RTOS-like" program by having the interrupts for all the peripherials do the real-time tasks in interrupt time, while the superloop jogs around and handles any cleaning up needed. It's an efficient programming model, but a model that only works well for a subset of embedded problems. When this model does not work, the _best_ initial action is to look at a different processor, that is way better suited for RTOS. Or alternatively check if it is economical to split the problem into a partitioned solution with multiple 8051 chips that collaborates.

Given the name of this forum, it should be quite obvious that quite few people here will talk warmly about RTOS use. A single 8051 isn't so well suited to the task. And a multiple-chip solution doesn't need any RTOS since it separates computations from real-time I/O into different processors.

But an RTOS really is excellent to use, under the premises that the problem is of a suitable type, and you do have a processor that works well with the selected RTOS. You may perform round-robin time slicing of low-prio tasks, similar to the superloop but without the need for state machines to be able to return from the middle of a calculation and then be able to later figure out where to continue with the calculation. And you may use timers to force high-priority tasks to get activated at specific times for actions that have to happen at regular times with low jitter. And you can still use interrupts for data collection, and then send events to tasks to notify them that they have input data to process, or have them prepare more data for output.

In the end, the only resource you have to think twice about is the stack space. Is it worth it to consume a number of kB of stack space for your tasks. Or should you consume a number of kB of code space to try to write nested state machines, picking up tasks to do based on priority queues (or have a fast enough processor that you don't need to care about priority but just hammer along).

To this day, I have never used any RTOS with any 8-bit processor, just because I have always managed to do all real-time work in interrupt handlers. But in projects based on 32-bit ARM chips, I have not just needed real-time performances for a number of peripherial hardware, but also needed to perform huge amounts of time-critical computations, where it would be very complicated to write every single loop so it can check state flags and instantly jump out - or call a helper function with a higher-pri loop - when a time-critical action needs to be performed.

The easiest way to manage large complex code without an RTOS is to have every low-prio loop constantly call a helper function that checks current priorities and potentially calls a higher-priority function. But contrary to a "normal" super-loop, that means that you don't run the actions after each other, but that you write your code to nest operations. So you need to have a stack that can handle worst-case nesting. And you also get into troubles if the low-prio code makes RTL calls that you can't control. For example a printf() contains an internal loop you are not in control of. So if the printf() takes 500us, then that is a worst-case 500us jitter when you would have wanted to activate your higher-prio task. In some situations, you could have added complexity (in case the code can predict the time when a higher-prio nesting is needed) by first checking how much time you have, before calling printf(). But such code - besides being complicated and only working for some problems - can result in the printf() being called from a loop that never finds a 500us window. An RTOS could have called printf(), and still performed a task switch after 170us when a timer trigged the need for more critical work.

An important issue here is that the price of processors does not follow proportionally with number of kB of RAM or flash space, or width of the ALU. So it's best to really look at the problem before deciding what processor(s) to use. Some problems can be solved in a small fraction of the time and with way less life-time support costs, if selecting a suitable processor mated with a decent RTOS. But while busy implementing lamp timers, you will obviously never need an RTOS. A huge percentage of embedded tasks are simple to solve without an RTOS. And are simple to solve with almost any microcontroller in existence.

It's just important to not make a specific architecture, or development tool or RTOS etc into religion. Let time and cost constraints be the main factors when selecting a solution. Existing experience obviosly do affect these parameters.