Have you thought about Remington Reduced Recoil rounds. 150 gr. @ 2640 fps. That's a little lighter than a .30/06. If you reload, work up a load for the lightest bullet, that will give you the same trajectory as your normal round.
As a long time (old) handgunner, I would shoot .38 special wadcutters 95% of the time. For self defence, I would use .357 mag's. So when I bought a hunting rifle, I looked for a caliber that offered a lighter round, to get a dual use out of it.