The current Amtrak station (former REA building) is at the intersection of the ex-Southern (NS) and C&O (CSX) lines, so you will be able to see both lines from there. Amtrak's Crescent runs 7 days a week on the NS side with the Cardinal Sun-Wed-Fri on the CSX side. The adjacent two-story Union Station is now a restaurant, Wild Wings. The former C&O station downtown houses various offices. The concrete coaling tower also stands over the remains of the yard here.
Traffic on the NS side is heavier in the latter part of the week (We-Sa) and averages about 18 trains/day. Most of the northbounds are bunched in the morning with southbounds usually scattered more throughout the day. On the CSX side, traffic is westbound coal & grain empties with 2-6 per day. Locals call at Clifton Forge and Charlottesville to make 1-way trips three days a week. Generally, a non-Amtrak day (Mo, Tu, Th, Sa) on this side will give you a better chance to see freight over here.
If you haven't yet decided where to stay, the Hampton Inn on West Main Street (5-minute walk from the station) is quite close to the diamond and offers a view of both sets of track with certain rooms. And there are several restaurants in the area.
