The port road is unusual piece of railroad. If trains are sitting at Perryville going east they are waiting on a slot on Amtrak and that can be mins. to hours. I have seen a train wait 5 hours with the crew still on board and then get to run. If the wait is to long or the crew is near their personal hours of service limits they are taken off and sent to a hotel (and occasionally driven back to home). If a train is waiting westbound, it is likely it took so long to get a slot on Amtrak that Perrryville was a far as they could make it before being taken off. Two things happen when a crew is taken off, if another crew is rested and Amtk reports the prospects are good they will run, a new crew will be put onboard. If prospects are not good, the train will be secured and it will await a new crew hours later.
The Port Road is unique in that Amtrak controls when the trains can move over the NECorridor. For the most part there are windows that start about dark and goes to the sun comes up. However the crude oil trains have rewritten the rules on these windows. But even with that said it is unusual for a NS train to get out at the height of the morning and evening rush hours.