When Planets Wander too Close to a Star