Molecular orbitals explain why this type of cyclization is disfavoured.
Click the structures and reaction arrows in sequence to view the 3D models and animations respectively
The most important reaction of the endo-trig class is the disfavoured 5-endo–trig reaction. The one message that you should take away is that 5-endo-trig reactions are disfavoured. 5-endo-trig cyclizations are reactions that look perfectly fine on paper, and at first sight seems quite surprising that they won’t work. But this reaction doesn’t happen, and instead the amine attacks the carbonyl group in a (favoured) 5-exo-trig cyclization.
K. Gilmore and I. V. Alabugin, Chem. Rev., 2011, 111, 6513–6556.