Should a startup have business insurance?

The only business insurance you really need in the UK is employers' liability insurance. It's illegal not to have it. Besides EL the most common covers are Public liability insurance and Professional indemnity insurance. It depends on your trade / profession what cover you need.

Insurance is a matter of trust. A simple PL insurance policy doesn't have to be expensive but will give your customer trust in your business.

Public liability insurance is important for your business because it
will pay out in the event that a mistake is made which causes an injury
to a member of the public or customer, or which damages their property.

You should consider taking out Professional indemnity insurance if your
business gives advice or offers a professional service to other
businesses. It covers you if you are negligent or make a mistake which
causes a client’s business to suffer financial loss.

If you tell me what kind of business you are in and if you are based in the UK, I can check what you need.