I've used a lot of different materials on my N scale urban layout that should work in HO. Styrene sheets cut and scribed with an Xacto knife, first roughed up with fine sandpaper (wear a mask) which gets rid of the plastic shine and gives the surface some "tooth" for paint. . Wash in soapy water, spray paint a weathered concrete color, then use washes to weather and bring out the cracks and joints. Bricks and cobblestones are fun, using commercial styrene sheets, sometimes brick or stone scraps from building kits. I use HO scale bricks for N scale cobblestones. You can make old style flagstone sidewalks with stone sheets or scribe your own. Then there's spackle, plaster, etc. The commercial stuff looks a lot better with the above treatments and I also save sidewalk pieces from kits.
Walk around and take some pictures, or look in books and magazines for the kind of city area you're modeling. Have fun!