Wanted to post an update. I have a college buddy, Simon, (also an Mech E.) who has been helping me out with this and came up with a completely different solution to this problem than I did. I am currently moving forward with his design due to the simplicity, although I have not checked it against all cases at this time.
My solution -
Keyframes:
I attacked this problem by finding the minimum number of input memory states to reliably identify the pattern on the input channel "C" (this turned out to be 3). I then searched the entire input space for 9 bit combinations that uniquely identified the pattern (keyframes) and created parallel logic paths for these (34 individual 9-input AND gates). This was quite complex but definitely functional.
Simon's solution -
Anchor Point:
Simon came up with a totally different way to tackle this. He evaluated the values for inputs A and B to identify which of the 4 quadrature states the encoder was in (labeled T1 -> T4 in the circuit), and latched a flip flop high or low depending on if the C input was high or low for that state. The memory is always anchored in place at the rising edge of the A input and which memory state is being written to changes based on the states of A and B inputs.
The 6 output patterns each had a 3-input AND gate looking at the values of these 4 quadrature memory states. Seemingly by the original design of the system, the states are mutually exclusive and for all input combinations that actually appear on the encoder, one and only one pattern output is always on. This means I don't need to have the "radio-button" memory circuit discussed above.
With his design, direction changes don't change the latched states, so they can be ignored. Transitions between states happen seamlessly without the detected pattern/outputs glitching. I still need to evaluate direction changes across the pattern boundaries, but I am hopeful these work without issue.
Simon also identified one additional simplification. None of the 6 patterns have the input "C" appearing HIGH at the same time and A and B are both high. The only time this occurs is when the index signal is present (one single pulse per encoder mechanical revolution). This means the condition A and B and C = index pulse.
I also want to look into seeing if I can figure out the correct outputs right when the encoder is powered on and pre-load these into the circuit. This way the servo drives know how to commutate the motor immediately at power on. I read something at the beginning of this project that led me to believe it was possible to figure this out.
My Design: >33 SOIC chips (14 or 16 pin), I never finished the "radio button" memory, output logic, or synchronous flip flops. This could have easily added 5-10 or more ICs
Simon's Design: 11 SOIC chips (14 or 16 pin)
I applied clocked logic to Simon's logic design as some people had suggested and pointed me in the right direction. I'd love feedback if I did it correctly.
My Design (file attached in above post):
Simon's Design (File attached below):
Note that in actual operation, the clock on the left input buffering D-type flip flops would be tied to the clock used everywhere else. I left it manually activated so that I could punch in the inputs and then toggle the clock to introduce them into the circuit (click the clock input 4 times to cycle it in).
Would love to hear everyone's thoughts.