If you ask the question, "why can't there be a waterfront intermodal terminal built in Portland," you first have to ask, "is there business potential for the railroad?" First off, container vessels are getting ever larger. Economics dictate centralized terminals to cycle the expensive ships as quickly as possible. There has been significant port consolidation over the years. Now, only a few ports are viable - Halifax, New York, Baltimore, Charleston, and Miami for containers on the East Coast. Other ports have their niches - Jacksonville, Norfolk, Philadelphia, Wilmington. Boston has been relegated to a feeder port as has been Portland. Which means that most of traffic going through Portland is locally originated or terminated. Which means, as Bud states, that it's a very short distance between customer and dock. No-one in their right mind would suggest putting a container on rail and railing up to Freeport, transloading and trucking to LL Bean.
As for domestic COFC or TOFC, Boston, Worcester and Ayer are right down the road, where centralized pools of containers, trailers and chassis provide operating efficiency. The business of hauling a trailer to an interchange point 150 miles away would be marginal at best. The trucking (drayage) cost to Worcester from Portland is probably about $250/tl. Subtract a discount for slower rail service, and you're not looking a lot of profit when terminal costs are factored in. CSX probably prefers a MA termination point: Economies of scale (Portland would siphon off some of these economies), fewer blocks to make up at distant terminals, fewer train starts, and the same revenue.
And if you're going to do it at all... do it in Rigby. More space, cheaper operating costs, and less eye-sore. (I remember when the Old Port wasn't artsy-fartsy- you wouldn't dare to go down there at night... and who would like to return to the days of Sappi-Westbrook stinking up Portland on a daily basis?)