If you have a "flush-cut" bit, you can swap bearings on the bottom to smaller size, to pilot along the inner circle & use the overbite to chew off what you need. If you have a "reverse flush-cut bit" (one with the pilot bearing above the cutter) you can make a larger circle template out of some cheap material, to fasten (clamp, screw) over the hole & run the cutter down from the top to remove the extra material. This means cutting out a larger circle in the scrap to make it & would take a bit of time, but you can check the fit in the template & then know it will work properly when the cut is made.
That's the easiest way I know to get things right. I'm sure someone else will have a different/better idea.