What would anyone recomend as a good size for an HO bookshelf layout?
I think the simple answer is that it depends on the size of your bookshelf. this site has a large number of possible plans and concepts for very small layouts, and links to other sites with other ideas.
The problems are that, naturally, the smaller you go, the less scope you have. There's only so many tricks you can use to get around this. Also, I think the idea that a layout can be something you stick in a drawer or closet and only take out when you really want to use it has its flaws. You've got to find a way to store rolling stock in that case, and it's a pain in the neck to be constantly removing stuff from the layout to store it -- as well as the wear and tear that occurs when you have to handle the stuff that extra amount.
So I would probably look for the biggest permanent space I could locate for starters -- and that would include a place to put the power pack. Then I would start looking at the plans on the sites suggested above.