- Joined
- Dec 18, 2019
- Messages
- 7,495
I think actually the error is +/- 0.0005" between the true distance and the rounded to 4 places value. I get a max error of +0.496 tenths and a min error of -0.496 tenths in the interval between 0 and 5.9998" inclusive. That's pretty good.This is true but the math conversion uses the micron value for displacement measurements, converting to inch at the UI. There is a bobble in the display but it will be limited to +/- 2.5 microns or .0001" in the reported position for the 5 micron scales.
My mill has 5 micron scales and reads accurately within +/-.0001" over a 6" distance, my calibration distance.
For the absolute geeks on HM, here is the python3 code I used. I cheated and used numpy because it uses array methods like matlab. More compact code.
Python:
#!/usr/bin/env python3
"""
Find the minimum and maximum error in tenths due to display rounding of metric
scale.
"""
from numpy import arange, around, amax, amin
scalediv = 5 # scale native resolution in um
aa = (scalediv * 0.001)/25.4 # 5um scale resolution converted to inch
rndidx = 4 # will round to this number of places
x = arange(30480) # 30480 = 6*25.4/0.005. x is an array of integers, eg. 0,1,2,... 30479
truedist = aa * x # true distance discrete values from 0 to 6 inches
rnddist = around( truedist, rndidx) # round every element of truedist to rndidx places
maxerr = amax( truedist-rnddist )*1e4 # find max error in tenths
minerr = amin( truedist-rnddist )*1e4 # find min error in tenths
print("Native scale division = ", scalediv, "um")
print("maximum error is = ", maxerr, "tenths")
print("minimum error is = ", minerr, "tenths")
Code:
In [42]: run scale_rounding_error.py
Native scale division = 5 um
maximum error is = 0.4960629921324511 tenths
minimum error is = -0.4960629921259632 tenths