UPDATE
This got a bit long, but I *THINK* it makes sense towards the end.
So I am having a new theory that goes along with the other one.
I just measured the fired cases lengths, and there seems to be a variation or two.
In the target, we will concentrate on the left column of groupings using the 32BKs. I will call them #1. #2, #3, and #4 from the top down.
#2 all came in at 1.844, with one at 1.847. As you can see, 4 grouped up, and one shot in the target bullseye. ...Bare with me.
#4 (the group with the air show stunt plane) measure in at 1.853, 1.853, 1.850, 1.847, and 1.846. The target has 3 grouped right at the bullseye and two monster fliers. But look at the case lengths... Two, maybe 3 are really starting to pull away from the "starndard" size. I call it standard because #1 was also right in the midst of 1.845-1.849 for a length.
Now I am sure that is not the only reason, so here is the rest of my reasoning....
I had some rounds that felt different when closing the bolt. Not hard per say, but no where near smooth either. And if I recall, they were in groups #1, #3, and a couple in number #5 (with a few others in those 40gr groups too). Now why would that be? I am thinking my mixture of 204 brass from two rifles (60 rounds from a previously owned Ruger #1, and 140 rounds from the Cooper) might not be getting FL sized 100%.

I am not certain yet, but that is what I am going to be testing here. I have a bag of about 100 pieces of my fire 204 brass I have not loaded or sized yet. I am going to insert them into the Cooper. I will find a few that fit nicely, and I will (hopefully) find some that are a bit tight. I will size them and then attempt to chamber them again. It is my estimation that I just might have a few rounds that are slightly more difficult to chamber. If that is the case, then I think I will have solved my problem. I will need to screw down my FL die more until those slightly tighter pieces fit like the ones that are from the Cooper. Basically, I purpose that nothing is getting 100% FL sized, but since most are from the Cooper, it doesn't matter as much for most.
Of course some of this could be attributed to user error, but given what I have shot previously, I have to say the bulk is not.