If you have say 200 supported (valid) commands, each averaging 20 to 30 characters a validation test obviously must test these 200 alternatives.

But if a command looks like:
* <cmd_id> * <param1> * <param2> <special_key>

where <cmd_id>, <param1>, <param2> are sequences of 1 to several digits (but sometimes requiring a fixed number of digits and sometimes separate min and max numbers of digits) then validation tests, must also include the paths to error.

So ** is an error (no <cmd> entered)
So *<special key 1> is an error
So *<special key 2> is an error
...
So *<special key n> is an error
So *<digit> <special key1> is an error
...
So *<digit> <special keyn> is an error
So *<digit> <digit> <digit> <digit> is an error (if cmd is expected to be exactly 3 digits).
So * <digit> <digit> <digit> <special_key 1> is an error
...
So * <digit> <digit> <digit> <special_key n> si an error
So * <digit> <digit> <digit> * <special key 1> is an error
...

The number of routes you can enter an invalid key sequence may be many orders of magnitude larger than the number of valid sequences. The valid sequences forms a (very large) flow chart. The invalid sequences are all paths that leaves that flowchart.

The regularity of the supported commands affects the number of pivotal positions where the input may deviate from a known sequence. If a device only supports commands looking like * <digit> <digit> <digit> <endkey>, then the possible numbers of errors to check for are limited. But each time you concatenate a new requirement to a command, the number of error alternatives grows.

I am not talking about a theoretical case here. I have real products with real keypad handling. They do have lots of supported commands. They do have a very large set of error paths. If the "good" commands are often very long, then the error paths will also on average be quite long. And without a possibility to reset the state machine to an "in-the-middle" state, you need to get the ok/error message from one command before you then start from scratch with testing next sequence.

-----

You can't see why a state machine would include memory? Time to sit down and read code. Lots of code. There is a huge set of problem areas you haven't looked at yet. You never had a protocol decoder, where the states are affected by previous bytes received? Each byte received may add (in theory, but not practically) 2^0 to (definitely possible) 2^8 to the state machine, depending on protocol type and where in the packet you are.

-----

I would suggest the smoke is heavy in a room way closer to you, if you can't realize that a packet containing a 16-bit address, a 16-bit len, a 16-bit cmd, followed by an unknown number of parameters would form a packet larger than 16 bits where a CRC32 may be way better to use than a CRC16. After all, it's enough to get two consecutive bytes to play with in a memory block to be able to forge any CRC16 value.

It would be better for you if you spend time trying to figure out what I'm trying to say, instead of constantly argue against me. You may actually learn something by listening to others, instead of constantly debating in basically "output-only" mode. You jumb at me using the term CRC32 instead of figuring - how would a CRC32 field in a message affect the number of possible internal states of a state machine?

Worse than that - you think the number of states of a state machine is just the number of different values a variable named "state" can take when handled in a switch statement (or a jump-table concept in assembler).

Look at a state machine:

enum states {
    IDLE,
    PROCESS,
};
for (;;) {
    switch (state) {
        case IDLE:
            if (data_available) {
                crc = 0;
                sum = 0;
                state = PROCESS:
            }
            break;
        case PROCESS:
            if (data_available) {
                ch = get_char();
                if (ch == bad_ch) {
                    state = IDLE;
                    break;
                }
                if (ch == end_char) {
                    if (crc == zero) {
                        if (sum == 0) do_alt0();
                        if (sum == 1) do_alt1();
                        if (sum % 1) do_alt_odd();
                    }
                    state = IDLE;
                    break;
                }
                sum += ch;
                crc = crc32(crc,ch);
            }
            break;
        default:
            state = IDLE;
    }
}

The state machine only have 2 states in the switch statement, so it's trivially easy to think that it only have 2 states. But that is so very wrong. The CRC test adds state. The sum adds state. You can have an infinite number of characters sent to the state machine adding up to the sum before it's time to check if the sum was odd or even or the CRC32 of all received data ends as zero.

Another example (already mentioned in an earlier post) is if the state machine looks at a timer value, that is ticking once every second. If that timer is 32-bit long, it multiplies the state machine with 2^32. If such a state machine contains code that checks the second counter and reacts differently if the seconds part is zero, it will be very much harder for an external pattern generator to place it in the correct state to get it to follow the expected code path exactly when the seconds part are zero. If the timer ticks with ms resolution, then it will be exactly spot on once every 60000 ticks.

There are huge number of problems where a state machine really do have hidden state - either internal data getting updated asynchronously, or history information based on earlier iterations - that adds to the state. It adds to the exponent for the 2^n expression. But that represents a multiplication of the total number of states. Even if you haven't seen it, such state machines are not uncommon. What happens if a user with a lamp timer presses a button "light lamp" exactly when a lamp timer have ticked a programmed sequence exactly until it reaches "turn on lamp"? How do you get an external pattern generator to manage the key press at that specific time? Now, this time: Please think before arguing that it is irrelevant or doesn't happen or whatever. Show that you are a professional and either stay silent or respond with a comment that shows that you really do have spent time thinking about the problem.

-----


Why not ask me? I do work with signs, and display protocols too... Your anser was actually a "let's not consider this - let's use a teflon shield and bounce the discussion somewhere else since actually looking at the amount of state information in a display sign would show huge (!) amounts of memory handled by the state machine in form of if statements looking at different variables before deciding if the state machine may jump from state 1 to state 2. It's trivial to count the number of states the state variable can have. But externally, it takes huge amounts of test patterns to condition the internal variables so that you can test the if statement being taken. And the if statement not being taken. If a single state machine state contains 5 if statements, then that represents 2^5 = 32 different setups to try to get it to leave that state. If one of the if conditions is "have scrolled all text to the end before switching page" and the sign is long, it may take a couple of seconds (with frame rates of 100 or 200Hz) until the scroll is done. How would a fast external test vector generator get past that issue? A sw test harness can have a serial-port injector that does "set_hscroll_done!" making sure that the next turn through the state machine, that if statement will see no transitions ongoing on the panel, and will allow a switch to new text.

Don't you get it yet? Every if statement in source code represents a two-way branch. They are also part of the state machine. Not just the 20-state "switch". So 2^128 can be reached with just 128 if statements. That is reached in a puny program. Module testing splits the software so you test functions one at a time. Mathematically, you are factoring a huge number into the the indivudal numbers that gets multiplied from the individual modules in the software. Having an external test vector generator test a real program with full code coverage is harder than to crack an RSA crypto key with very large key size. This isn't just random noise. System testing with a test vector generator is a mathematical concept with easy math to show the complexity. Even 500 lines of code quickly gives you large numbers of test vectors. 5000 lines of code scales mathematically - if 500 lines requres n vectors and 5000 lines have same density of conditionals, you get n * n * n * n * n * n * n * n * n * n number of test vectors. Let's say n was 10. now you have 10*10*10...*10 = 10^10 test vectors. Just by growing the program from 500 to 5000 lines of code. Time to speed up that test vector generator, my friend...


And how many conditional statements do you have looking at these 256 bytes? Or looking at the SFR values? They are part of the state machine. But many of them are hard or impossible to set to known states by an external test vector generator.


A test vector can be serialized, so even a 8-bit UART can be used for an infinite number of unique patterns (if we ignore the amount of internal RAM to receive the test pattern).

Also remember that test vectors aren't strictly in the digital realm - signals being zero or one. You have analog signals (will the ADC measure 0x43 or 0x44 when you feed it a known signal? will the alarm processor see the ADC value as inside or outside the alarm range? Will it count down the filter time, or reset it? How will the test vector generator know?) And test vector data can be in the time domain, since some test vectors are only meaningful at specific times - such as when your code is inside a specific ISR while the main loop is inside a specific function - the only combination that may get a specific program to run into a stack overflow.


How do you do code coverage analysis? You just don't? The concept is as old as software engineering so it was probably talked about when you took your FORTRAN courses way back. Even an 8-bit microcontroller with 2048 bytes of flash can have a huge flowchart for full code coverage analysis. And there exists tools that can set up software test vectors to test code coverage. Hardware test vector generators are great - but invented at a time when electronics contained lots of discrete logic and when you could feed test vector stimuli into internal parts of a circuit in case the total logic contained too many flip-flops or other memory elements resulting in a too large test-vector set for full coverage from just externally accessible signals.