So I'm heading a shop with some serious legacy deployment woes. I've got plans for everything but scaling. Now, we're likely to hire out Ops eventually, but for now I'd like to learn devops (backend eng currently) and cut my teeth by getting the ball rolling on a slightly more robust setup.
With that said, what do you recommend? We're a small shop, and our auto scaling needs a very very minimal. We're non-public facing, so our user base is pretty predictable. Yet, I'd like to migrate us to a more modern and mature system. Naturally I was looking towards K8s because of the overwhelming support I see for K8s on HN/etc, but your comment makes me question that.
Honestly, it's the vast options that really screw with my head. There are just so many tools. And where they lack is not immediately evident from the outside. It's difficult to pick a direction. At the moment I'm reviewing Nomad, as it sounds interesting to work with our existing by-hand infrastructure.
I'd suggest going in baby steps. Start with the absolute basics - standalone docker containers + docker-compose. Work in those containers, deploy those same containers. Build your deployment pipeline. That'll get you familiar with the workings of containers in the docker context, and you'll be forced to iron out issues you may find with this way of working.
Then slowly start to spread further - you could start with docker swarm, which is a smaller next-step for devs. Again it's to help you figure out what kind of deployment pipelines you would create and what constraints you need to introduce. The reason I call it a smaller next-step is because the docker compose file used for development is the same one you can use for deployment to standalone or swarm.
By the time you start hitting the limitations of swarm, you should be ready to move to k8s and look at the various options available to you.
Containers are still fairly portable and I've worked at a shop where we moved from CoreOS+Consul to Marathon/DCOS and there were almost no app changes.
That is one of the big benefits of Docker. We had some stuff just running on nodes with Docker/docker-compose and even that stuff slowly got migrated over to the big platform.
Agreed, this method sounds nice. The only thing I'd like to find on top of it is some software to manage apps across container resources and physical resources. Ie, I've got some non-cloud old windows boxes that I want to deploy to, so ideally I need a mechanism further abstracts that. I'm currently looking at Nomad, though i'm very early in research.
Currently on this path almost exactly. Problem description and steps all the way up to limitations of swarm. Our swarm currently handles 1M/requests per day we haven't really hit any limitations yet. Good advice though, really
You almost certainly don't need containers. Just use a PAAS (platform as a service). Heroku is the "go to" default, and will be worth much more than they charge. If you're worried about scaling, I've run off the shelve PAAS providers past 5000+ TPS without any issues.
If for some reason you can't use Heroku, you could try a hosted Cloudfoundry (https://run.pivotal.io). If you want something that hybridises PAAS-like behaviour with the big IAAS services, Amazon's Elastic Beanstalk isn't entirely awful. All these setups largely honour the PAAS haiku of "Here is my source code / Run it on the cloud for me / I do not care how" (https://twitter.com/sramji/status/599255828772716544?lang=en).
Don't waste your time on getting container orchestration going, or even containerising your app yet. For internal, low scaling, app-focused scenarios, you'd be mad to go down this road _unless_ your code has been broken into many discrete deployable artefacts _and_ you need to run them all on the same machine (either locally or in production).
You have an app. You want to run it. PAASes do _exactly_ this.
Edit: I see you're worried about costs. PAAS are pricier than the equivalent AWS instance when you consider raw compute, sure. However, factor in your time in creating the same level of redundancy, deployability and monitoring, I'd be shocked if Heroku wasn't an order of magnitude cheaper. Note that PWS Cloudfoundry will host you any 12 factor app for $21.60/GbRAM/month with no bandwidth limit you'll ever hit.
I should have made it more clear, but one need of mine (as mentioned in another comment) is a deployment system that supports the ability to manage very old, archaic, wonky systems. WinXP, Win95, etc. Do you recommend any tooling to support this?
PAASs aren't going to help me much in managing our resources like physical legacy machines. Thoughts?
Depends on what you want to do with these machines, but for multi-OS setup you need VMs. I guess you could check out Vagrant, maybe with some AutoIT to help you avtomate tasks inside Windows. It won't be pretty, but it will work.
K8S won't support them either. K8S doesn't support Windows at all at the moment. And even when it does, it likely won't support WinXP.
To add to that, the container story for Windows is pretty dismal right now, even on Azure. I was at Microsoft Build this year, and most of the PMs gave me the impression that Windows containers were an afterthought on Azure due to lack of use -- Linux support were priority. I also did some PoCs on Windows containers both on-prem and on Azure and they were unwieldy and buggy. The microsoft/windowsservercore Docker image is also a very heavy image at 10+ GB. The microsoft/nanoserver image is smaller, but has certain limitations.
tldr; if you want to use containers, go with Linux.
If you really need XP support, VMs are your only option. Tools like Hashicorp Packer [1] may be able to help with managing machine images.
i think kubevirt is being rolled in soon. It adds a VM type to the set of things k8s can manage. i'm not sure what's required inside the vm to play nice. I haven't used kubevirt at all. But it might be worth checking out.
Yea it containers definitely won't. Though something like Nomad _(as I'm researching now)_ that tries to manage both physical machines and container fleets might fit nicely.
To be honest it sounds like you probably shouldn't. There is a risk to researching "the best" technology, it can be a full time job and you still wouldn't get any closer to your goal. You are likely to have a lot of low hanging fruit in a legacy systems shop. Start with those and move yourself forward from there.
Without any specific knowledge of your situation it's hard to be anything but generic advice. But first you need eyes and ears. Under no circumstances should customers know of problems before you do. Are you monitoring the right things?
If there is a product, take a long hard look at the build process. When those mostly work without that special someone that always does them because no one else can, start mapping out all the infrastructure, document it and place it under version control. Implement a configuration management system. No exceptions.
With repeatable builds and repeatable infrastructure you are in a much better place to see what your real requirements are.
What you are going through is why “Just use Heroku” is still great advice from wise people.
There are so many options out there that until you actually hit the point where Heroku is too expensive or no longer makes sense...Heroku is your best option (if AWS is an option).
Dokku is extremely flexible Heroku alternative, and a good introduction to dev-ops. You can start-out treating it like a "dumb" Heroku host, but install plugins and customise to your heart's content. You can easily migrate from apps/buildpacks to Dockerfile's one app at a time etc.
Whilst it's great for getting your apps up and running, and learning basic dev-ops at your own pace. It's not really designed for scale/reliability. So if your small business/app/service starts to take off, you'll quite likely need to migrate to K8s, Docker Swarm, Amazon ECS, if not for scale, for redundancy/reliability.
This is something I've been hearing. I have a small dev team that deploys many small (but important) apps internally.
We feel we need to hire more ops folks to help us with maintenance and troubleshooting, but people keep telling me that instead of increasing headcount, to just outsource my ops to PaaS like Heroku. After reading the comments here, I may look into this more.
As somebody who was used to running my own servers when I was first forced to use Heroku on a project...I'll go ahead and tell you there is an adjustment period.
But after using then going back to the alternatives, there's a lot more that I miss about it than the other way around. You take for granted how much stuff you no longer have to deal with, until you have to again. There will be a moment when you'll be irritated about some very low-level knob that you want to turn that you can't get access to but when that happens, don't take for granted all the other ones you're not dealing with.
And it simplifies life for developers tremendously.
I'm hoping to keep it cheap though, and cloud deployment services scare me a bit on price. We're a ~15y old company (of which I'm recent) and in a domain where margins are fairly thin. I'd rather spend the sweat learning this (or hiring for it) than paying Heroku, fwiw.
Oh, Heroku doesn’t lock you in. It’s just running your code on containers. There’s even an open source library called Dokku that uses their open source build packs to give run the same thing in Docker containers.
The minute you price out the time and training investment to factor into your deployment solution, Heroku starts winning in a landslide.
Existing scale certainly changes things, but the fact that it forces you to go ahead and build for 12 factor makes moving elsewhere pretty simple.
On that note, we have some odd ball legacy machines that are dictated/required by our customers. Things like WinXP. So I'm surveying Nomad now as a potential to support non-traditional OSs (though I wouldn't be surprised in Nomad doesn't support WinXP). Do you recommend anything for odd deployment needs like this?
I imagine we'll write custom plugins to handle the odd ones, since who the hell releases builds for WinXP or Win95 these days lol. Nevertheless, I figured I'd ask. Especially since broader solutions like Heroku are not going to help our WindowsX deployment woes. (note, they're physical machines currently, though virtualization might be possible)
Heroku locks you in in the sense that Heroku has this huge ecosystem of "add-ons" that often can't be replicated 1:1 outside of the Heroku environment, or if they can, require 100x the configuration management that Heroku does.
Much of the task of moving a project from Heroku to e.g. AWS is taking all those "add-ons" and going through the whole process of converting the Heroku-bound account (where everything is managed through your Heroku dashboard) to a separate, full-fledged account with the same service provider. If that add-on is a data storage provider, that also often entails a data migration.
With that said, what do you recommend? We're a small shop, and our auto scaling needs a very very minimal. We're non-public facing, so our user base is pretty predictable. Yet, I'd like to migrate us to a more modern and mature system. Naturally I was looking towards K8s because of the overwhelming support I see for K8s on HN/etc, but your comment makes me question that.
Honestly, it's the vast options that really screw with my head. There are just so many tools. And where they lack is not immediately evident from the outside. It's difficult to pick a direction. At the moment I'm reviewing Nomad, as it sounds interesting to work with our existing by-hand infrastructure.