I’m sure this is a banal observation but: what does it mean to call something offensive? It’s not to say that onesself is offended, necessarily. It doesn’t simply mean that someone somewhere would be offended. It could be like that a reasonably large number of people somewhere would be offended, but that doesn’t ring true to me.
Rather, I suspect it means that someone somewhere is normatively entitled to be offended. So something is offensive even if no one nowhere would be offended, if they could rightly be offended. Likewise, something may not be offensive even if many people are offended, if they are wrong to be offended (ex: “Sarah Palin is not qualified to be VP”).
If this is true, though, the normative theory that determines the content of “offensive” is somewhat opaque. Arguably we should be more open and specific when using “offensive” in this sense. It usually means something like, it unfairly denigrates some group (NB “unfairly” stands in for a moral theory); compare to the contemporary usage of “to discriminate.”