If your boring head cutter has the geometry to do the bottom cutting, then you could do those step cuts where you just feed over something like .1 and let the mill plunge the boring head into the work. Nice and easy, of course. Then just keep resetting and stepping over and repeating. That way your boring head is set to 1.5" and all you have to worry about is when you're going to hit your numbers on one axis.
Also, like others have said, you can use a hole saw. You could do both. hole saw with something like a 1-3/8" hole saw and then finish it up with the boring head set to 1/5" so it does not take so long and the finish is much better.