Rumble was founded in the summer of 2011 by an all-star team of game industry veterans from Zynga, Activision, BioWare, Blizzard, Playdom, Electronic Arts, Turbine, FooMojo and RockYou, and backed by top investors. Check out our team: http://www.rumblegames.com/team/
We are building both high-quality titles and a publishing platform for independent developers. By combining the best of AAA game design with free-to-play accessibility, we want to change the way people play games.
Redwood Shores, CA (SF Bay Area) - Engineers and more
EA2D is hiring! We're a new, autonomous studio within EA building cross-platform social games for gamers. We've just launched our first game, Dragon Age Legends:
* New features for Dragon Age Legends (big ones: real-time, mobile, etc.)
* New games for big-name IP (we have access to the full EA library)
* An epic social gaming platform (for both internal and external teams)
We're small (30 people) and scrappy. And we're growing fast: 0-200 servers in the past 6 months. Tech stack is primarily AS3/Java backed by MongoDB, but we also use Python, JavaScript, and Ruby. We have a highly automated infrastructure running on AWS (EC2 w/ELBs and ASGs, EMR, SQS, etc., plus more than a few super-secret/unreleased Amazon features). Buzzword potpourri includes Chef, node.js, Google App Engine, Hive, Graphite, Tomcat, GitHub, Pingdom, Loggly, PagerDuty, and continuous deployment. <3 DevOps.
We need platform, game, systems, and mobile engineers. We need data people and a Director of Technology. We need producers, marketing, and designers. Some of our job listings are posted here: http://www.ea2d.com/jobs/, but we have positions we haven't even finished writing descriptions for.
If you're an A player, drop me a line: mikeb@ea2d.com
Designing for fault tolerance and scaling out instead of individual node durability and scaling up is, in most cases, a sounder approach. Since the vast majority of applications map well to this case, it makes sense for Amazon to focus on features that assume/assist scale-out.
It's easier to design for scale-out than one might think; a great resource to get started is http://highscalability.com.
Regarding EBS, I haven't seen the hangup issue you've described. Any data?
Overall, I think Amazon is making a lot of inroads in areas with specific hardware demands. They just launched GPU compute options, and I expect we'll even see SSDs soon.
Scaling out vs. scaling up is the false dilemma, the answer is both at the same time solving the bottlenecks you have using the time and resources you have. The "vast majority" of applications do not map well to an n-wide map-reduce-esque design, the "vast majority" of applications bottleneck on an authoritative datastore's I/O capacity.
I'm the sysadmin (not the dba) for a website that does 30K http req/s at the edge, and our mysql cluster does about 3000 req/s during mid-day with most queries in the 1 - 5ms range.
Highscalability is a terrible place to get started, its a great place to share notes but if you get started there you will waste insane amounts of time on architecture astronaut nonsense.
On the spectrum of scaling an application AWS has definitely leaned hard towards scaling out. (where the other end is scaling up) This means it caters a little more to denormalized databases and load balanced web servers than it does to one giant mysql database server or a cluster of a few large database servers.
I agree with nethergoat in that if you need low latency communication between ec2 instances that you should be using their hpc instances with faster/optomized ethernet.
I however disagree with nethergoat and agree with cagenut and think that the vast-majority of applications still currently depend more on a scale up paradigm. That is slowly moving towards scale out as people/businesses see its benefits and AWS is catering to this trend.
Can you share some example of how you are using them together. I am currently using Python scripts with boto to bring up the instances and doing all kinds of configuration stuff on them. Just wanted to see if there is any better way of doing this.
(Note that this assumes you have a Chef server or are using the Opscode Platform. There's also a server-less option called "chef-solo" that you can use)
1. Create an appropriate first-boot.json to be used by the node(s). This will probably just be a "run_list" with entries like "role[app_server]" (function) and "role[production]" (environment).
2. Replace line 42 of the bootstrap script with your first-boot.json. Since this bootstrap script is actually a template, make sure to replace all other .erb stuff ("<%= %>") as appropriate.
4. Build your CF template, placing the JSON-encoded script in the "user data" section of your instance/auto-scaling group(s).
5. Instantiate your CF stack
If you'd like to use Chef for the entire lifecycle of your instances, just include a Chef recipe for running chef-client via cron every, say, 5 minutes. This allows changes to be propagated automatically in a pull style.
Done right, the end result is a complete, Chef-managed environment provisioned from bare metal using just one command.
Cool, thanks a lot for sharing. Your post has given me the motivation to start learning Chef. Maybe you can blog about it somewhere to help wanna be cloud programmers like me.
You can also bootstrap using Puppet. Basically you generate a template that you can use to upload to CF and it will just run the bootstrap script from the UserData section when the instance is started. See the wiki here: http://projects.puppetlabs.com/projects/puppet/wiki/Bootstra...
We're a small, autonomous studio within EA building social games for gamers. Our first monetized title, Dragon Age Legends, is about to enter closed beta and has been generating a lot of buzz (http://pc.ign.com/articles/114/1146553p1.html).
Our Platform team is looking for mid to (very) senior engineers. You would be responsible for building the platform on which all EA2D games will run, from the pages that serve the HTML to the suite of game servers supporting them. Analytics, social network integration, shared services (REST), scaling, real-time stateful and stateless game servers -- this would all be your domain.
Our stack is primarily Java with MongoDB on the backend, but we're becoming increasingly polyglot (Python, Ruby) in the middle tier. We're hosted on EC2 (w/auto-scaling and on-demand environments) and practice continuous deployment. A sampling of the tech we use: Chef, git (GitHub), Hudson (now Jenkins?), ant, ivy, SQS, S3, EMR, Loggly, PagerDuty, Mixpanel, Kontagent, NginX, Tomcat, Hive, and Google App Engine. We contribute code back to the open source projects we use, and we've even started open sourcing our own (https://github.com/EA2D).
The Platform team is small and young -we're still building out the initial implementations of many services- so you would have a tremendous impact on our architecture and direction.
We're a tight-knit team with lots of whiteboards, so you'd need to be on-site. On the plus side, EA's campus is fantastic - we have a stellar gym, cafeteria, theater, shuttle service to Caltrain, soccer field, volleyball and basketball courts, tournament-quality foosball tables, and much more.
Drop me a line if you're interested, mikeb@ea2d.com
We've already hired two people from previous HN "Who's Hiring?" threads, so you'd be in good company!
Note that it's not 1000 servers every week, it's up to 1000 servers in a given week. This was likely done to support the launch of a new game, and may factor in server turnover (they're on EC2).
That being said, launching 1000 nodes for non-batch jobs is still pretty remarkable.
Zynga can have absolutely crazy growth on launch. FarmVille was growing few million users a week.
Also keep in mind interactive multiplayer games with persistent state are different beasts compared to usual web applications. Games are very write heavy and do not cache well.
This is precisely the approach they take. Section 3.4 ("The Model") begins, "The heart of the model is to treat incoming events as a Poisson process..."
Note that we're very new (and hiring!), so much of this is still being built out.
Software lifecycle: git for SCM (GitHub as central repo); ant for builds; ivy for dependency management; Bamboo for build automation; Crucible for code review; JIRA for issue tracking; Greenhopper for sprint planning; Vagrant for containerized development environments
Systems infrastructure: AWS (EC2 w/ELB+ASG and S3 for now, more services to come); nginx-fronted Tomcat for JVM; Cassandra for persistent DB; chef for configuration management (currently chef-solo, but
evaluating Opscode Platform); PoolParty+Fabric for one-click deploy server fleets; Nagios + Pingdom for monitoring; Mixpanel + Kontagent for analytics; Dynect (likely) for DNS, possibly for GSLB; Akamai (likely) for CDN, possibly for GSLB
Misc: PagerDuty for on-call stuff; Google Apps for email, shared docs, etc; Confluence for wiki; OpenFire (Jabber) for chat
I definitely wouldn't consider EA2D a startup, but fortunately the OP left it open ("startup/company").
That being said, I do think we (EA2D) have a lot in common with later-stage startups. Three big factors are our autonomy (near-complete), self-sufficiency (product, dev, marketing, ops, etc. is internal to our studio) and size (just 27 people across all disciplines). I bet our stack more closely resembles that of a startup than that of an enterprise.
We're a new division of EA focused on bringing big-name titles to the browser. Think Halo on Facebook, but with EA titles. We work in small, cross-functional teams, so each engineer has a huge impact on game and systems design. We're autonomous of EA, so we can use the tools we want without all the red tape (GitHub, AWS, choice of Mac/Linux/Windows, etc.), and the stack that fits our model best (Flash+Java+Cassandra+EC2 on Facebook to start). And we're a new division, so you can help define our studio culture and processes.
Of course, being part of a large company has its advantages: nice hardware, excellent comp., onsite gym+soccer+basketball+volleyball, great food, free games, and more.
Engineers, artists, designers, more - http://www.rumblegames.com/careers/
Rumble was founded in the summer of 2011 by an all-star team of game industry veterans from Zynga, Activision, BioWare, Blizzard, Playdom, Electronic Arts, Turbine, FooMojo and RockYou, and backed by top investors. Check out our team: http://www.rumblegames.com/team/
We are building both high-quality titles and a publishing platform for independent developers. By combining the best of AAA game design with free-to-play accessibility, we want to change the way people play games.
Drop me a line: nethergoat@gmail.com