Depends. If your primary group has strong variety and is ably prepared for all threats, you can get away with it. However, if there's a glaring weakness in that formation then you can bet a scenario will arise that will wipe you out.
Mainly I try and vary it up to avoid the risk of squandering exp, it really frustrates me to see a unit reach level 20, not have an item to upgrade their class, then see them continue to demolish enemies with no gain. That exp could have gone to someone else.
Tell you what, the characterisation in this game is superlative, even better than previous Fire Emblem's I've played (not many, admittedly). Within minutes I've developed a strong fondness for Sully, and her interactions with Kellam are adorable.