First, make certain that the location is really the one you want for future vacations and/or retirement. Get to know it thoroughly. Many places that are nice to visit as a tourist (i.e., Cancun) are quite different to live in. Further, consider how the place will change over the next 30 years. Finally, consider how you will change: Will you still be surfing at 84?
Second, get a big enough house. No one-room cabin is large enough to even hold all the appliances that the average American regards as a necessity (or will come to be necessities by the time you retire). Remember that if you go up there for holidays, you will be tempted to invite the extended family. Also, your kids will get older, get married, have kids and come to visit you.
A good rule of thumb is: whatever size house (kitchen size and number of bedrooms and bathrooms) you need for your primary residence, you will need for your vacation home.
Third, realize that you may change your mind, and not want to have a vacation/retirement residence in that location. Therefore, you should always maintain the option of selling that house for at least as much as you put into it. Be cautious about buying in an area that does not bode well for long-term appreciation, or about building/modifying a home in a way that is so customized to your idiosyncrasies (i.e., a corral in the living room) that no one else would want it.
Fourth, make the decision a family affair, for it is the family that will share the joys (and negative experiences) of the second home.
\o7 Brink lives in Redlands and teaches psychology at Crafton Hills College. His second home is in Toluca, Mexico, west of Mexico City.