Yeah, I hear what you are saying. Millbrae and Richmond are the most complicated endpoints on the system. Millbrae is sending trains to the airport, as well as down the mainline to San Fran. And they try to do as much as possible on platform 3 (next to Caltrain). So, unless every train is within 30 seconds, or so, somebody is in the way of somebody else.
Richmond has the added hassle of taking every train into the yard, which means everything runs through one crossover. Again, every 15 minutes, there are 8 movements over that piece of trackwork. Plus, there's the additional time factor, since the break room for the operators is a decent walk from the train. So, if an operator quickly offboards, walks to the bathroom, grabs a cup of coffee, and walks back, there's a total of 15 minutes. There's probably a "sweet spot" in the schedule, for getting from Richmond Yard to Richmond Station, and if an operator misses that, it can mean a late start. It's hard (but it can be done) to make up time to Macarthur, and the sequencing through the tube starts there (Macarthur). So, the train from Richmond trades places with another train (usually a train from Pittsburg), and is, at best, 4 minutes late. Some of that can be made up, but that's a lot to make up.
If everything works right, then a Richmond to Millbrae run can be on time, but there's a lot that has to be right.
> If anyone wants to sit down someday and record train times ...
A while ago (before SFO opened) I actually did this. During the boom, I often rode BART downtown, from east bay, and I usually planned meetings for either before 8 or after 11. This was based on parking availability at Dublin or Walnut Creek, which filled by 7 AM.
However, I had one meeting at 9, so I got to Embarcadero at 7 AM, and recorded about 45 minutes of arrivals. As I remember, only one train was about 5 minutes late, but all of the 18, or so trains were about 2 to 3 minutes late. The system was "normal" - nothing unusual - and that 96 % number looks about right.