Good point. I manage a site where we wanted some extra income from AdSense. It took people away from our site and sales diminished.
We found, however, that after we repositioned the ads, we have the best of both worlds. The ads are now on the bottom of the page and we get the AdSense income along with good sales from our primary products. Keep in mind that AdSense also pays for page *impressions*, not just clicks. So you get paid, albeit less, for the ads just being there.