Since deer are the main target with the possibility of elk something in the .270, 7-08, 280, 308, 30-06 will fit the bill nicely and not beat the heck out of you. Funny thing is when I started elk hunting hardly anyone I knew used anything bigger than a 30.06. A few guys toted .300 mags, but not many. Nowadays everyone seems to want to use the biggest cannon they can pack.
When and if you go after elk, use a stout, deep penetrating bullet and you won't have a problem killing elk with the cartridges listed above. I've seen many .277 150 grain Nosler Partitions exit on bull elk.