Received the following from another source, thought I'd share it:
"Prior to completion of the Harlem River Branch, which was when? - about 1875? - the New Haven operated freight service right down the New York & Harlem line and Park Avenue into Manhattan. At first, in the 1850s, the trains went to a New Haven freight terminal located in the vicinity of East 23d St, I believe, that being the southern limit for steam power at the time. Carload feight was forwarded by teams of horses to a yard in the Canal Street area. Freight houses were located further downtown. (Ironically, New Haven operated carload freight service much further downtown than did the New York & Harlem.) Later, in 1871 after New Haven pulled back passenger service to 42d Street and the original Grand Central opened, much of the freight was handled at a yard located northeast of the terminal. By then, 42d St was the southernmost limit steam power could operate on the Park Avenue line (or 4th Avenue line, as it was then called). I have read stories in old New York City newspapers from the 1870s in which residents of Manhattan's East Side complained that the New Haven 'Night Freight' was assembled on one of two Harlem main tracks, frequently blocking local street crossings for prolonged periods. By inference the train was usually assembled once the evening rush hour ended and probably departed around 9 PM. The inbound freight usually arrived in the wee hours, I think, most news items I found involving an inbound New Haven freight took place in the predawn hours. But sometimes it arrived later. Columbia University was located just above Grand Central then and I found a story in which a student at the time recalled occasionally having his path blocked by a New Haven freight. Despite the danger, students would climb under and over the cars to get to school. "We didn't dare be late," this young man wrote.
"In the meantime, carload freight was still forwarded downtown to a freighthouse in Lower Manhattan called White Street, I believe. Below Grand Central Depot (as it was then called) the traffic was horsedrawn. One car at a time, with the - teamster? engineer? - sitting on the end of the running boards. This service lasted well into the 1880s, btw."
Ex-NYNH&H SS Opr