it will indeed take a little while. it is not likely your age. it will probably take on the order of months to handle rougher terrains (depending on your definition of "handle"; maybe a year to get into the beginning of the gnarly range). and indeed you are too impatient: you are having fun!
smooth concrete basically feels like someone massaging your feet with warm velvet. which is fantastic. except if you ever want to leave the sidewalk.
in my experience, three things have to happen. first, your feet have to develop the fat pad to go between your tendons/ligaments/bones/muscles and the gravel. that will more or less happen by running on concrete as opposed to maybe sand or mud or grass. second, all the tendons/ligaments/bones/muscles have to strengthen and get used to what you are throwing at them. that comes by putting the miles in and throwing things at them. and third, the mind and the feet have to learn to work together to deal with the pokies.
around here, i am becoming the chip-n-seal apostle. running long distances on smooth concrete made my heart/lungs/legs happy and my feet weak. after that experience, i decided to go out of my way to run on chip-n-seal at least a token amount as often as i could to strengthen my feet (and hopefully keep everything else healthy, too). for me, it was a pride swallowing exercise (that is, an acceptance of reality since there wasn't/isn't much to be proud of anyway). i slowed down to basically a crawl and would then slowly speed up to whatever i could handle on the surface. but, by hitting the chip-n-seal (which doesn't move under your feet while still poking like gravel, so you get more poking and less useless "pain"), my feet (and knees, strangely) got much stronger, the skin thicker (which is important through the winter), and the nerves able to adjust their sensitivity appropriately. and the mind was able to learn the difference between a poke that really mattered and a poke that just happened to be poking a little bit. after a while, i started weaving all over the road searching for the roughest patches; and while the broom-finished concrete sidewalks felt more wonderful than ever, when i ran on them, i missed the rough pavement.
so yeah: it will take a while, but it will probably be worth it. your pace will slow waaayyy down and then slowly recover. and when you return to the smooth pavement you will really zoom.