Just saw this - it's 5 and 6 because of how the New Haven numbered tracks through old SS 3/Bungay/Oak Tower. Tracks 3, 4, 5 and 6 passed over the Hellgate bridge while 1 and 2 went to the NYW&B terminal/yard. 3&4 diverged at old Broadway Jct. to go to Harold and thence Penn, while 5&6 went straight to Fremont and the connection with the LIRR. So, no there were just 2 tracks on the portion that ran to Fremont, but the numbering system was kept into the PC/Conrail age. NHRTHA is a great source for this, look for posts from Henry Statkowski, who was a former NH operator who worked on that branch and was a huge source of info regarding its use.