I'd still disconnect the main power coming into the machine and wire directly to the motor through a manual switch, you know to be in good working order, to test the motor and remove it from the equation as a potential host. That would squelch any concern regarding the 'jittering' behavior and remove both the motor and it's capacitors from the equation....if found to be in good working order. Considering the machine is wired at 120V, most likely your amperage is only going to be around 12-15 amps, depending on hp, so just ensure your switch that you use to test is rated for upwards of of 15-20 amps and you'll be fine. Bypassing this to test the motor isn't going to hurt anything.
And as others have stated, the relay is simply there so that you can have a safety margin built into the operation factor of the machine....namely the guard switch.
That said I believe your issue here is simply a broken neutral connection on the control circuit.
Once you've tested the motor-- I noticed in your latest schematic #36 that the past owner or someone has the control portion of the relay (A1 and A2) not wired conventionally. A1 is usually always the terminal for your hot and A2 is for your nuetral connection in a single phase situation. Someone has that wired backwards here and is using A1 as the nuetral connection and A2 for the hot. Considering this case, that loose wire coming from your guard switch would have to go to A1 at your relay in order to complete the control circuit. Namely, the neutral side of the circuit. At this point, there is no neutral connectivity there at the A1 terminal, in order to complete the neutral circuit.