The problem could be indirectly due to the nut. IIRC the pocket it sits in isn't machined (or not machined very well) so it could be rocking back & forth. That could result in a change in the effective amount of clearance between the feedscrew and nut, so the friction between them would vary under those conditions.
The X axis nut has two set screws so if the nut is rocking on a high point it may be possible to play around with them to adjust the alignment & reduce or eliminate the described problem. It really depends on where the high point is. It wouldn't hurt to play around with the set screws some -- loosen one then tighten the other one down, see what happens, then do the same with the other set screw.
Being new to it all lathes/machine work and picking up a used machine no previous owner seemed to know how to adjust/service it.
One thing I can say is , if you think its clean , clean it again.
I could not believe how much fine gunk was imbedded in the gibbs and ways. My carriage felt like it was moving through half dried cement. Clean it over and over now it darn near floats like skates on fresh ice.
I had all sorts of binding /tight loose movement
Cross feed nut just sits in a hole on a stub , it rocks all over. I shimmed the hole the best I can with what I had on hand it helped alot.
I also had dial contact on the housing , after cleaning out all the bearings, a complete mess.
Found a few of the balls have rough spots.
Now the gibbs, touchy touchy
So I took a good amount of time adjusting. Adjust with bed all the way in each direction and then 1/4 it then center and eventually I found a sweet spot . Oh I also to. Fine diamond stone to the gibbs and ways and took of a few very noticeable high nick spots.
I had to use shims to get the dial off the housing and provide enough thread travel to get the nut/bearings adjusted to no play.
At some point someone was in here and made a mess of the shaft.
Plus ran the machine with the bearings loose and cocked eyed, bush bearing was loose and spinning in the housing.
So keep at it, you will figure it out