Am I justified in my fear of African Americans?

No worries. Anyone can get a little irritated by an accidental comment. You just happened to accidentally irritate a bunch of black dudes in one day. Life goes on.