Borrowers taking out smaller loans are having their applications denied at higher rates than those taking out bigger mortgages.