Sorry to be so pedantic, but that's a "dial indicator" or "drop indicator" (DI). The term Dial Test Indicator (DTI) to me means those with a lever arm like this one:
DTI's are usually more accurate (and pricier), with smaller measurement ranges, but allow you to reach into small areas otherwise inaccessible with a DI.
As Rick points out, you'll likely find it easier to get repeatable results if you measure over a MUCH wider range than 0.100". A pair of 1-2-3 blocks, gage pins/blocks, or micrometer standards work well as he suggests.
As he also points out, mechanical standards and hard stops are far more precise than any adjustable measuring device (indicators, micrometers, calipers, etc.). But quality indicators should be extremely repeatable: they excel at +/- comparative measurements. For example: zero the indicator dial on one reference, then compare the reading to something else — if it doesn't also read zero then they are different sizes. Indicators are less useful for quantitative measurements (they are fine for coarse measurements, though).
So my preferred technique differs from Rick's only slightly: I use a pair of 1-2-3 blocks to create a 4", 5", or 6" reference, then mechanically attach a DTI to the spindle and zero the indicator on the top of the stack. The DTI eliminates any need for "feel". Remove the stack and lower the quill (or raise the knee) until the indicator reads zero again. You can be confident the indicator tip has moved exactly, say, 5.000" (if you've verified the precision of your 1-2-3 blocks and stacked then for a 5" reference).
Since you said your results aren't repeatable and are all over the place, it's probably an electrical or quality issue with the DRO. Mechanical issues should at least be repeatable (except for sticking or floppy mounts, which should be somewhat obvious).
I'm better with mechanical troubleshooting than I am with electrical troubleshooting, though, so here are some thoughts to at least verify the mechanical aspects:
- As above, use hard stops, physical standards, and comparative indicating rather than measuring anything whenever possible.
- Avoid any backlash effects. Ensure you do all of your measurements traveling in just one direction. E.g. always lower the quill until the dial reads zero and don't back up. If you overshoot, back way up, then travel back down until it just reads zero.
- My pet peeve: some of my offshore equipment dials effectively estimate 1" as 25mm (an error of 1.5% — one inch is 25.4mm). They use metric screws with metric pitches instead of actual inch screw threads with true turns per inch. I doubt this has anything to do with your DRO readings, but I wouldn't be surprised if your mill dials have this problem. Use a 1-2-3 block on its long axis, zero the indicator and your dial, remove the block, then turn your dials until you've supposedly moved exactly 3.000". Don't be surprised if the indicator doesn't zero on the table (or vise top if that's where you placed the block).
- Angular travel and poor tramming. In theory, lowering the quill should have the same effect as raising the knee (or lowering the head on a bench top mill). This is only true if the spindle is perfectly trammed, though. Tram the spindle to the table as accurately as possible, then use an indicator mounted in the spindle against the the vertical faces of a good square reference to ensure there is no movement in X or Y as you move in Z (a 1-2-3 block or a 2-4-6 block works well). Do the same with the indicator tip agains the side and face of your DRO scale instead of the square to ensure it's running perfectly parallel to the spindle axis.
- Verify rigidity. Some of my most difficult problems to track down boiled down to a loose screw or something flexing even a tiny amount without my noticing. It's incredible the amount of havoc one loose screw can create. Glued, brazed, or welded attachments are significantly more rigid than screws and nuts/bolts (screwed and glued gives you strength AND rigidity). Use glues that harden when cured, not stuff that remains tacky/soft. A few dots of CA glue under your DRO gage mounts won't hurt anything, and you can still remove them with a sharp tap. A broken bond will also make it obvious if something is flexing or moving. (Use acetone to clean up CA glue detritus.)
- Understand your dependencies: verify your metrology gear! Get even a cheap small surface plate and some basic metrology equipment (DTI, surface gage, and at least a few calibration standards at a minimum — I recommend acquiring at least a 0.500" and 1.000" gage pin, but I find a full set of gage pins up to 0.500" in 0.001" increments incredibly handy) . Learn how to inspect a pair of matched 1-2-3 blocks: Verify all surfaces are parallel to the plate surface and that the two blocks match in each dimension. Then use reversal methods to ensure the faces of your 1-2-3 blocks are square/perpendicular (glue a ball or even belt-ground round front to the base of your surface gage to perform comparative squareness checks). Ultimately, you should only need to depend on the flatness of your surface plates, the accuracy of your calibration standards (gages), and the repeatability of your indicator. You should be able to directly (or indirectly) verify everything else.