I would definitely stick with Metro-North's Hudson Line. The river view is spectacular (and I went on a cloudy day in the winter, hills all capped with snow).
LIRR's Montauk Branch is worth the $29 RT. You'll pass by the Shinnecock Canal (actually caught me by surprise even though people on the LIRR forum told me about it : \ ) and see some beautiful ocean when the island "thins out" on the east. If you can, try to take Train #2706 (11:00a departure) so you can take #2709 back to NYC which departs at 2:51p. Bring lunch on this trip. Walking to town is far. #2709 should have a nice express ride down the Babylon Branch (which I found out it is a little boring but I liked the speed!)
Also if time permits, you should ride the A train down to Far Rockaway. By far the best views the NYC Subway has to offer, you'll overlook JFK Airport and Jamaica Bay. You'll also travel the longest distance between stops on this portion of the line. (almost easy to forget you are in the city
) You peak 45mph on the stretch.
If you ride the Flushing line 7 train going east after the Queensboro Plaza station, you'll hover over the big Sunnyside Yard which services Amtrak and NJ Transit trains. (Also can be more closely seen if you ride LIRR east out of Penn Station NY, i.e. when going to Montauk) The yard is usually full during the middays.
One tip, sit on the fireman's side (the left side) of the train on both legs of a railroad trip to get the most out of sightseeing the railroad.
That's all for now. Hope this helped.