Am I the Asshole? Bus Stop Edition

I need to get something off my chest.

I drive a city bus. Often in the course of my work day, I have to "deadhead" between the bus yard and either the beginning or end of my routes. Deadheading means driving an empty bus which is not in service. The header signs on the front, side and back of the bus indicate this with various phrases like "Not In Service", or "Garage", and certainly no route number showing.

Some of our deadhead routes follow established bus routes, so it often happens that you pass people waiting at bus stops. Most people see the header sign and accept their fate that my bus is not for them. However, there are always a few people who, for various reasons, don't get that information from the signs, and become visibly distressed. This is usually demonstrated with upraised arms and a gob-smacked expression on their faces.

Now, I understand that some people have barriers to being able to read the header signs. It might be poor vision, not knowing English, or possibly illiteracy. I'm not trying to sound ableist or ethno-centric here. It's also possible they simply didn't bother to read the header sign. That's on them.

However, even allowing for those possible barriers, why does it seem that the first assumption by the people being passed up is that the driver (me) is either an idiot, or an asshole, or both? Have they never encountered an out of service bus before? Possibly, but you would think they could conceptualize it. Should I stop and try to explain the butt-hurt away? No.

When I pass someone while deadheading, I usually try to make eye contact. Like I said, most people get it, but I like to give them a friendly little wave, if for no other reason than to let them know I see them and acknowledge them. If they still look concerned, I try to make the wave look sympathetic. I don't know how successful I am at that.

But when the person gets too assertive about it, and begins uplifting their arms in incredulity and adopting that slack-jawed expression, I find it difficult not to mock them by raising my own arms (if safe to do so) and staring open-mouthed back at them. Sorry. They made it personal. I could probably get in trouble if this got back to my superiors, but I'm going on the assumption that if my victim can't figure out that a bus is out of service, they're probably not going to be able to figure out how to report the incident, or if they do call, they probably won't be able to successfully identify what bus, what bearded, Caucasian driver (out of several hundred), and so forth.