Yes, you can refuse unless you’re legally required to provide your Social Security number (SSN).
However, there’s also no law preventing businesses from asking for it or refusing you service if you don’t provide it. If a business requests your SSN, you can always speak with a manager or ask if there’s an alternate ID you can provide (such as a driver’s license number).