While a vast system of new rules governing bank behavior has arisen, some regulators wonder if enough has been done, and some analysts say the largest banks may shrink.