Right florida key to stay near tranquil beach and party spot

Hello! I'll be traveling from Kendall, Miami to the Keys for 2 nights with my 2 year old son, husband and my grandmother.

We want to see the typicall sights and landmarks of Key West, but I´m not sure if I want to stay there because I dont think my grandmother and son would be in party spirits all day.

All though we would like to go out for a drink during the night and leave our son with my grandmother.

Where would you recommend us staying for two nights in a middle ground where we can to get to know key west the first day, party one night and spend a day at the beach at a really beautiful place the second day, before leaving the next day?

Thank you!