It's an interesting theory that you say we are smartest when we are babies. In the way that you state it, you may well be right. But why is this? I think it ties itself to a few things. For example, the rapid increase in brain development and growth during this time and the instinctive acquisition of language during this stage of life. I'm going to focus on the latter, as I find it most intriguing and thought provoking. This is all from memory so there are no references so don't ask.
Complex language has helped humans develop into the intellectual powerhouse among the animal kingdom. Did we develop complex language because of our increased brain size, specifically the frontal lobe, or did our use of language help increase brain size and structure. A bit of the chicken/egg conundrum. Personally I think they both influenced each other symbiotically.
Now how does this specifically affect children? In a few interesting ways. Young children have an instinctive fascination with learning how to talk and comprehend. They do it at an astonishing rate, much faster than an adult normally would. It is hardwired into their brains to acquire the langauge of those around them. It takes their motor skills some time to catch up so they can speak the words, but their comprehension of what we are saying begins very early. My son is almost 2 and he's been saying real words for maybe the last year, and he already understands almost everything we say and his vocabulary is astonishing. I think to when I took Russian or French classes and how frustrated I felt and how long it took me to leran what babies learn with ease.
Language helps anchor our thoughts and memories, helping retain and organize both in our brains better. Indeed there is a window of opportunity for a child to learn language. If no language is learned in this timeframe (it may be within 6 years of age but don't quote me, it might be odler) the child will almost never be able to grasp language in later life no matter how much they work at it, although there are exceptions. Interestingly these people also score significantly lower on intelligence scales, even when language is not a factor. These are mostly the rare instances of "feral children" as every other civilization on earth uses complex language.
Another interesting note is that developmentally, children who are deaf are at a higher risk of learning disablilities than those who are born blind simply because of their lack of auditory input. But parents who work dilligently with a deaf child to teach them well in sign language develop at a compareable rate to those who can hear. What's also kind of cool, is that if you work at it, children often learn simple sign language before they learn to talk becuase of how complex of a motor skill speech is compared to simple hand signs and gestures(think Meet the Fockers haha).
Well that's about all I have for now. Maybe I'll add more later but since no one will probably read all this, I may refrain
