Software Delivery Platforms 2019

Fernando
Fernando

We are starting a new "SaaS" project.

After working for 5 years on a project with a rusty "enterprise" deployment solution (Weblogic!, 🙄... not my fault), I'm now evaluating options to build a modern "software delivery platform". It's almost 2020 and "devops" evolved a lot in the last few years.

This post is a disorganized dump of my thoughts and findings.

Here's a initial list of what I want to "solve" and integrate into a usable platform:

  • Source control
  • Provisioning (infrastructure as code)
  • Continuous Integration
  • Building pipelines
  • Continuous Deployment
  • Databases (provisioning, backups, etc)
  • Availability (clustering, scaling)
  • Centralized logging
  • Monitoring
  • Automation of public DNS and HTTPS certs
  • Secrets handling
  • Identity management
  • Standardize packaging of any app so they can all be deployed and managed the same way
  • Standardize "Microservices" (deployment, protocols, monitoring, circuit breakers/backoff)
  • ...etc...

Before getting too fancy, I know we could simply use Heroku: We launched two successful startups with it before. You can get up & running quickly and get pretty far with it, but you have limited control for more sophisticated architectures (say, microservices). The same goes for Google App Engine and Amazons's Elastic Beanstalk.

This time we wanted something more flexible.

Infrastructure as code

In the past we've used tools like Puppet and Pallet to automate the configuration of virtualized servers (on AWS EC2, Linode, Digital Ocean).

That's old.

Instead, we now want "immutable" infrastructures: automate the creation of new VM images and provision new servers on every release.

Oh, wait... that's already old too.

Nowadays, with Docker containers taking over the world, the cool way is to "containerize" your apps and services, and deploy them to a "container platform". And to make a long story short: Kubernetes (aka: K8s) is now the de-facto container orchestration platform. Most of the "infrastructure as code" is now replaced by Kubernetes configuration.

So, I've spent a few days researching and getting my hands dirty with Kubernetes. I must say it's really cool and seems to live up to its hype. It makes a lot of sense. If you are interested, I recommend the book "Cloud Native DevOps with Kubernetes" By John Arundel and Justin Domingus.

Getting your Kubernetes

Alright. The decision is made. Kubernetes it is. Our solutions will now be all "cloud native", so we'll be fully buzzword compliant again :-).

I cannot justify configuring and maintaining our own Kubernetes cluster. It's not a trivial undertaking. I'd be a great learning exercise, but I cannot justify the time to do it myself. We'll go with one of the many providers of managed Kubernetes clusters. Here are the ones I'm evaluating:

  • Google's GKE: Probably the most sophisticated managed Kubernetes offering. Not surprising given that Kubernetes came out of Google. Lots of features: sophisticated dashboard and monitoring, full integration with the rest of Google Cloud, explicit support for Helm "charts" (the standard for "cloud-native" app packages), world-wide data centers, etc. New accounts current get a $300 credit, which is great to get started. You don't get charged for the master nodes: You pay for your compute engine nodes only (plus additional storage, load balancers, etc).
  • Digital Ocean's Kubernetes: Easy to use. Nice UI with some dashboards. It's inexpensive too: you only get charged for your worker nodes (plus additional storage/load balancers, etc), and Digital Oceans pricing of nodes is very transparent. One drawback for us: they don't have data centers in South America.
  • Amazon Elastic Kubernetes: Minimal Kubernetes on EC2 nodes. No dashboard at all. Pro: You get to leverage the whole AWS cloud platform, which is the market leader. Cons: you have to pay $0.20 per hour per cluster (that's $144/month !) not cheap. It seems to me that Amazon did the minimum they could do to tick the Kubernetes checkbox. I don't doubt its technical robustness and all, but given the feature-set and pricing, they are clearly not interested in promoting a standard clustering solution which is a threat to their leadership on the cloud market.
  • Azure: I haven't yet tried Microsoft's offering, but I've read good things about it. Like Google and Digital Ocean, they don't charge for cluster management, only for worker nodes. I'll give it a try soon.
  • Linode: The very first "cloud" provider I've used since year 2000 or so :-) real pioneers of the "cloud"... even before the virtualization era (was based on User-Mode Linux !). Their K8s offering is still in Beta... I've registered and still waiting.

Now, Kubernetes doesn't magically cover all the items I listed at the beginning. You still need a container image repository, git hosting, a Continuous Integration system, Deployment pipelines, secrets handling, handling of multiple environments, etc, etc. However, Kubernetes does solve some of them and makes the others easier.

Before re-inventing the wheel

Alright. Now I have my clustering solution... now I need to develop apps in a "cloud-native" way, need a container image repository, CI/CD system integrated with it and deploying to Kubernetes, etc. Still lots of work to do, tools to evaluate and decisions to make...

Isn't there a platform with all those items integrated already? Say, a "software company in a box"? Every new software startup has to solve the same set of problems of software delivery... there should be a one-stop solution by now, right? Here are some options I found:

  • Redhat OpenShift: From their promotional videos and websites, this might the closest to what I had in mind: clustering infrastructure, deployment pipelines, team management, and more. Or so I hoped. Unfortunately, they seem to target "enterprises": big companies wanting to move their internal infrastructure to the cloud. Their pricing is not transparent at all. You are actually encouraged to "contact sales" in order to get a quote. Still... I decided to give it a shot. Looks interesting... except that 80% of the time all I got was these kind of pages: OpenShift Console I've signed up for a "Pro" account, just in case. Same thing. It simply doesn't work for me. Canceled.

  • Fabric8: It's an open-source solution that integrates Jenkins and lots of other components into a cohesive solution. From their site: "fabric8 is an end to end development platform spanning ideation to production for the creation of cloud native applications and microservices. You can build, test and deploy your applications via Continuous Delivery pipelines then run and manage them with Continuous Improvement and ChatOps". Perfect! Except... the project seems dead :-\

  • Jenkins-X: Alright... looks like I'm getting into something here. First, ignore the name: it has little to do with Jenkins. I'd define Jenkins-X as a "set of components" for software delivery on Kubernetes, and a command-line utility to make it all happen. Its jx tool automates the set-up of a full platform bringing together Github, CI (with Jenkins or Tekton), Prow, Sonatype Nexus, a Docker registry, ChartMuseum, Monocular, etc. The result is a "Gitops"-driven platform running on your Kubernetes cluster, with automated pipelines, handling multiple environments, promotion strategies, and more. After a few hours of trial-and-error, I finally got it running on Google cloud, creating a new cluster with all Jenkins X components and Github integration. Quite impressive. I still need to use it and learn it more, but overall Jenkins X seems like a good option. Some doubts/questions:

    • It's really new and bleeding edge. Documentation is a bit messy and some issues I run into have been reported by others only 3 days ago!. The fully-automated setup failed for me multiple times, and I had to delete all left-over artifacts before starting all over.
    • It's very opinionated. I'm not criticizing it here. It's an understandable trade-off: if you want many problems solved for you, the solution will undoubtly come with decisions made by somebody else.
    • Maybe I'm missing something here: looks like it uses Kubernetes namespaces to separate "staging" from "production". Namespaces are just a logical naming, with no physical separation... which means Production pods may run alongside staging pods. Maybe one could use Kubernetes' "node affinity" feature for this? I would have liked to be able to use a different cluster for the production environment.

  • Gitlab: I knew Gitlab leapfrogged its older brother Github in many aspects, but I've never used it before. It includes CI/CD capabilities and now integrates with Kubernetes, so I thought I'd give it a try now. Wow! Why didn't I try it earlier?!. It's a very nice and fully-integrated solution. I was able to setup a simple Java project with a non-trivial pipeline, deploying to a Staging cluster, and then promote it to a Production cluster. I'm very impressed. Looks like it does everything I could think of and more, including:

    • grouping projects in hierarchies
    • assigning one or more Kubernetes clusters to each group or project
    • includes a docker image registry
    • npm/maven repository
    • CI/CD with flexible pipelines, auto-deploy to staging/prod/whatever
    • dynamic environments
    • "automatic containerization" of standard apps

Enough analysis for now

You may easily fall into a "Choice paralysis" trap here. So, I think I'm done with my "discovery and research" phase for starting a modern "software platform" by the end of 2019. We'll go with Gitlab for now, with GKE and Digital Ocean. I'll get deeper with Gitlab setting up other users, automating SSL certs management and deploying more realistic applications.

Some options we didn't try but might be good:

  • Atomist: "Take Control of Your Cloud Native Development". I'm curious, but starting at $833/month is a bit too much
  • Circle CI: CI/CD service now with K8s integration
  • Codefresh: Another CI/CD service with K8s support
  • Drone CI: Newer open source CI/CD
  • Logz apollo: Open source CI/CD for K8s
  • Weaveworks: Automate Enterprise Kubernetes the GitOps way
  • Puppet Labs' Nebula: Simple workflow automation to continuously deploy cloud native applications and infrastructure
  • Platform9: "Run Cloud-Native Infrastructure Without the Pain"
  • ... etc. etc. The list is infinite

I hope this helps somebody. If you know of other ideas or alternatives, I'd be happy to know about them!.