Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Why might I want to use this or pgbouncer instead of, say, the connection pooling built into ORMs like SQLAlchemy?


Because the way you scale up a Python web app is by adding more servers, and ORMs like SQLAlchemy can't share(aka pool) connections across processes let alone servers. Scaling a web app means thousands of open connections to Postgres, and the only way to fix that is server side connection pooling. (Server as in Postgres, because here your app server is the client)


You might want to have many app backends, each with it's own Alchemy pool. This setup would create a lot of Postgres connections. When the network flaps you need to reinstall lots of TLS connections. Maybe 100ms of cpu for each handshake. Also each connection may cost you a fork with lots of following CoW, also up to 100ms of cpu. And also Postgres gives you maximum TPS throughput when you run some hundreds of connections, not thousands. And you might want to isolate one microservice from another (in terms of DB throughput utilization). Poolers can help here too.


You would only see the real value in these poolers in dire times. Screwing up apps or deploys leaking connections is really easy. If that happens you will bring down the whole DB. With these poolers you will not only be able to limit that but also enforce per user (service based if every service has it's own creds) limits. These tools are useful beyond certain scale and you will only realize there true value once you are working in distributed micro-service world specially when they are connecting to same DB.


The ORM connection pooler is client-side - so each instance of your application using SQLAlchemy will have its own connection pool. This may become unwieldy as you scale up the number of app servers/containers/processes.

In contrast, connection poolers like Odyssey or pgbouncer live external to your applications, meaning that:

1. You can connect to them from any application written in any language (not just Python) 2. They provide a global pool for all of the applications connecting and help to not overload Postgres.


Why not python? Works on my machine.


I believe you read "just not Python", when the parent poster wrote "not just Python", meaning that a connection pooler can be used also from other languages besides Python, which don't have SQLAlchemy specifically.


Oof. Yep. And the timer is over so I can't edit or delete :(




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: