One other giant point to remember was that prior to coonstruction of any of the vehicular crossings (Lincoln, Holland tunnels, GWB) of the Hudson, everything had to be floated across. Even passenger took a ferry or the tubes.
Once those items could be trucked to/from the door of the facility to the railhead in NJ there was no need to double or triple handle the items. (i.e. truck from facility to pier in NYC, place on barge, place on pier in NJ, and then place on railcar.)
The other big item, coal for NYC, was also a small part of the operation by the 1950's having been replaced by other heating methods.
One of the largest buildings not in the picture is the Seaboard (cold storage) building that still exists a few blocks prior to the toll plaza and just north of the Erie tracks.
My wife's grandparents still live about two blocks south of the Seaboard terminal and the remains of the right of way. The entire area was very economically depressed when they bought the home in the early 1970's but now it is a thriving, affluent urban area. I often wish I had seen the area, especially the RRs in their heyday.
Chris
CV the Civil E