The "No one really knows what you do anymore" is a murmur that follows senior leadership role all the time. And while it CAN mean that you're not doing anything, in this situation I think it was a case of some people struggling to assess the value of all the activities you do through their own lens. More likely than not, you were making valuable contributions to the company; much more so than your employees realized.

So when they weighed their perception of your contribution against their negative feelings towards you, the fulcrum was positioned against your favor.

Departure like this can be a turning point for the company for worse, especially at a small company like this. I hope this isn't the case.