I climbed in Rovinj in Istria last year. To give you some context, I was with the family including my 75 yo mother, a non-climbing complaining 16 yo and my 9 yo both of whom are more interested in shops. The town is beautiful, busy but no too busy.
The climbing is in an Venetian era coastal quarry. The climbing consists of pretty easy single pitch sports routes with great views. www.climb-europe.com gives a summary and Rovinj is covered in the main Croatia Guidebook. There is a guiding service but assuming that your partner and you are both climbers then it is not necessary. The climbing is ok to be honest but it was a compromise given the family circumstances
My wife and I headed out for a climb in the cool of the morning before going back to get the kids for a day of swimming and sightseeing.
You can ride a bike from town to the bottom of the crag and the base is pretty kid friendly. By infant I assume that the little tacker is non-ambulatory. If this was the case then you should be fine with a pram/stroller.
It worked for me but later on the trip I spent 10 days in Kalymnos where the climbing was infinitely better but my 16yo complained non-stop.