Concern about the health of smaller banks focused on the venture capital and startup communities is prompting regulators to consider extraordinary measures.