It’s always a possibility, but this is partly why people settle. If you’re on a ship without a sail and you end up in some desert island, you’re gonna stay there because you’re scared of what may come(or not come) next. A desert island is better than another 2 months at sea with no land in sight.
On the other hand, a goal can have the same effect with the opposite idea, you may envision something amazing, and you give up a beautiful paradise island in pursuit if your goal, and in the end you need to stay in the desert island.
Basically, goal or no goal, you need a healthy amount of luck. But with a goal you have a standard to beat, and this means you’re less likely to settle.