Switching banks by itself will not damage your credit. However, if you choose to switch banks at the wrong time, your credit score may at least be temporarily affected. For example, you should never close an account while it’s overdrawn or has a negative balance.
If you’re about to a apply for a loan, don’t switch banks. Having two credit checks so close together is bad for your credit. You should also not switch banks very often, as this too can damage your credit.