Talked to my buddy in Oregon today. He is having two Remington XR's rebuilt by Greg Tannel the Colorado gunsmith that we both use (
http://www.gretanrifles.com ) Tannel recommends shooting three shot groups. He says that three shots show the rifle's capability, while more show the shooter's ability. Okay, that sounds reasonable, as some have pointed out here they usually blow the last shot. I have pulled many a shot, but also experienced many an uncalled flyer - I even earned the Distinguished Flying Cross for errant shots.
Bullets are the most common cause of flyers. If a bullet is manufactured not quite right, it may fly like a curve ball. Drop a bullet on the floor and I guarantee it will leave the group. I was astounded when I began swaging my own jacketed bullets for competition. They were superbly accurate way beyond what I expected. Why? Because each was hand made, every jacket trimmed and measured, each core swagged to size and weight, cleaned by washing, and all operations carried out on the same set of dies. The finished bullets were washed clean, not tumbled, and were never packaged and shipped. Imagine a 7mm bullet that shoots a 100 yard .75" 10 shot group from a pistol, albeit a bolt action pistol, with iron sights! Hard to do with factory bullets, although they are much better these days.
Next is the barrel. A crappy barrel is a flyer generator. If the steel didn't stress relieve perfectly, or has a weird grain structure or inclusions, if it wanders as it heats, or if the chamber was cut a bit off, or the reamer wasn't right, nothing is going to fix it. Following that is any stress points on the action or bedding, if each shot doesn't exit the muzzle at the same point in the barrel's vibration node, it's going to a separate location. There is also a sweet spot in action screw tension, it's a trial and error proposition.
My same friend told me about his new Nosler Model 48 in .204 Ruger. First thing he noticed when cleaning before firing was that the barrel was the smoothest he ever felt as the patch passed through. Nosler uses Pac-Nor Super Match double lapped barrels. He took it to the range and found it capable of shooting groups down in .2's - with a sporter weight barrel! He also said the barrel throat was cut much deeper than the average, and that seems to enhance accuracy! Same thing I found with a Shilen Select Match barrel in .204 Ruger, bullets developed best accuracy seated well away from the lands, well down in the .2's (with Winchester brass no less). Same thing with a .223 Remington barrel I just reinstalled on a Savage action. It is a Pac-Nor Super Match 1:9' twist polygonal rifled barrel, it shot a sight-in group of .206" for 7 shots using Nosler 40 grain lead free BT's. This batch of ammo is in unfired full length sized cases - just primer pockets were uniformed and flash holes deburred, no neck turning. They were cranked out on a progressive press, go figure.
Had problems with one barrel. Checked everything twice. Finally checked bullet runout, it was less than .001". Crappy barrel. So if everything else checked out, it's probably a crappy barrel.