Evolutionary, and probably not even a unique idea. My post does go into the intelligent sharding a bit. Essentially sharding, in the case of &bang, on teams with a simple lookup to find which server a given team is on. Allowing the stateless app layer (as you rightfully call it, other than keeping track of subscription event routing) to connect to multiple Redis servers as needed. You could be on two teams on separate servers and it would work fine.
I also agree that people have been using Redis for messaging, however, what we're doing is a step past that. Again, probably others are doing this. We have atomic verbs for dealing with higher level objects in Redis that use Redis-PubSub for communicating the changes to the objects. This way, whenever an object (feed) is changed (publish/edit/delete/reposition), it can bubble up to the user or processes that care. These feeds are broken down by not only topic but subscribe-able units of interest as well. In this way, processes, users, etc only get updates on data that is relevant to them.
I'm working on converting these verbs to Redis-Lua scripts as my tests have shown it decreases CPU time and reduces atomicity code (especially in Node.js where watch->multi->exec callback stacks can be interrupted by other events). I also expect it to make supporting Thoonk in multiple languages easier as the core code will be shared.
The gossiping is currently being discussed in the Redis mailing list and may end up taking a similar approach to Hadoop's intelligent clients. Antirez would like to provide some tools to make this easier as others have rolled their own Redis gossiping. I'll be researching that more after implementing the team sharding. HA, while related to scaling, is a bit off topic, and I admit to hand-waving here. For now we're focusing more on not losing/corrupting user data.
Perhaps Henrik's title is hyperbole. I doubt the approaches that we're taking are revolutionary, and I imagine similar things been done before. However, we believe that it is a good approach, and we believe in the direction we're taking enough to share it.
Cool, thanks for taking the time to go a bit more in depth on this. This is a topic I'm particularly interested in, so I'm looking forward to the follow-up.
I also agree that people have been using Redis for messaging, however, what we're doing is a step past that. Again, probably others are doing this. We have atomic verbs for dealing with higher level objects in Redis that use Redis-PubSub for communicating the changes to the objects. This way, whenever an object (feed) is changed (publish/edit/delete/reposition), it can bubble up to the user or processes that care. These feeds are broken down by not only topic but subscribe-able units of interest as well. In this way, processes, users, etc only get updates on data that is relevant to them.
I'm working on converting these verbs to Redis-Lua scripts as my tests have shown it decreases CPU time and reduces atomicity code (especially in Node.js where watch->multi->exec callback stacks can be interrupted by other events). I also expect it to make supporting Thoonk in multiple languages easier as the core code will be shared.
The gossiping is currently being discussed in the Redis mailing list and may end up taking a similar approach to Hadoop's intelligent clients. Antirez would like to provide some tools to make this easier as others have rolled their own Redis gossiping. I'll be researching that more after implementing the team sharding. HA, while related to scaling, is a bit off topic, and I admit to hand-waving here. For now we're focusing more on not losing/corrupting user data.
Perhaps Henrik's title is hyperbole. I doubt the approaches that we're taking are revolutionary, and I imagine similar things been done before. However, we believe that it is a good approach, and we believe in the direction we're taking enough to share it.
Thanks for the intelligent discussion.