Getting boxes to latch tight can kinda be a pain in the butt. I usually start 1/16th smaller than the inside diameter should be and start chiseling/sanding it down from there little by little till it just fits, and then give it the final sanding.