Brewster was typically served by one of the local freights based in Danbury. Exactly which one varied over the years. Here's one example from 1955: NX-11 on Monday-Wednesday-Friday went in three directions out of Danbury (New Milford, Botsford, and Brewster). Brewster was listed as "as required." Those words typically suggest that business there is little and sporadic. In that period (mid 50s) I believe they served Danbury-Brewster Lumber Co. (later Lloyd's Lumber, and I do not know what it is today) on Rt 22, and the house track or team track further west where the NYC connection was. The same local also served Danbury Block & Supply Co. near Mill Plain, CT (Super Stop & Shop is on the site today) I don't recall any other business in Mill Plain in the 50s (Wickes Lumber came along in the 60s).
I would say the NYC connection was used little if ever in the 50s, which is probably why it was removed. Interestingly, it was designated as a passenger train detour route if the NH main was blocked. Trains in one directions to and from GCT would go that way, and the other direction would use the NH Norwalk-Danbury line if the blockage was east of Norwalk). To the best of my knowledge there was one occurrence of that, in the summer of 1958. I don't remember any details as I was a kid then.
I have never seen a photo of a New Haven train crossing the Croton River bridge in Brewster. Too bad --- I would love to see one! (In old time lore it's called the "Milk Factory Bridge" as there was once a Borden's plant underneath it)