DevOps CI/CD with Kubernetes

please welcome the stage staff developer advocate at google kelsey hightower all right so i was supposed to give a project i love you too probably don’t quite know you yet so we’re not quite there yet so twitter friends are not your real friends if you’re new to kubernetes it’s going to set you free but first it’s going to piss you off while it’s born to many of us there’s still a lot of challenges for it but one thing i want you to understand is that kubernetes is not the end game okay the goal is i see people do this all the time they get kubernetes installed everywhere and they go out and giving everyone coop cto right who is that is that you in the crowd i’m just going to give everyone our back roles and everyone just use coop cto coop ctl is the new ssh okay do you remember virtualization right if you had to ask the station to the server to do deployments or to do the majority of your job you knew something was wrong i think coupe ctl has now reached that point if you’re using coupe cto deploy from your laptop to production you’re missing a few steps so i wrote this guy called kubernetes the hard way so that everyone could learn how to do kubernetes okay there is a big difference from installing kubernetes and using kubernetes chances are you’re not on both sides and we’ve come a long way raise your hand if you believe installing kubernetes is easy what that’s like seven hands here’s my first demo i’m hoping it works i think it’s gotten to the point where it’s so easy that you can ask for a cluster okay so we’re going to try to see if we can just ask for a cluster and see if it will appear fingers crossed say one for the demo gods so we’re gonna try this no one sent me a dm or anything to interrupt the flow let’s try this okay google talk to kubernetes engine sure here’s the test version of kubernetes engine good day create a kubernetes cluster how many nodes shall i provision eight creating a eight node kubernetes cluster in the us west one region that’s what i call kubernetes the easy way what do you mean all right so after today it’s not hard anymore what’s hard is that collection of hardware that you have on prim okay that’s not our fault you got to figure that one out so the next thing we want to talk about is once you have a kubernetes cluster again our goal is not to use coop ctl for everything that we want to do so what i like to see people do and people ask me all the time kelsey what is the best practice for running kubernetes in production the truth is there are some known patterns but i’m not quite sure everything is super mature and set in stone where we can go off and just have a complete list of best practices you do this and you’re good so what i like to think about is separate clusters based on your organization structure if you work in the enterprise and you have to open a ticket to get a deployment you probably need to have separate clusters for each of your environments trust me it will be easier that way do not go down the r back rabbit hole if you don’t have to but as a developer how many people are developers in the crowd are you looking forward to getting your nice cube ctl installed on your laptop no you see that ops people no one cares as a developer you have probably one flow in mind everything else is noise okay so i have this registry where they have this app it’s called hello world complex i know and as a developer this is how you envision the world okay and it’s all of its perfection you want to clone a repo not install kubernetes is not your first step that comes to mind i’m going to clone the repo that i’ll be developing against we’ll do this here we’ll go to my laptop and i’m going to do git clone amazing internet works all right and then as a developer you want your actions to trigger something else to happen ideally if i make a code change all i want is a url to tell me where it’s running you get bonus points if you give me metrics to tell me how well it’s running so the next thing we want to do here is create a branch so we’ll say git uh check out we’ll create a branch called change message so we’re going to change it from hello world to something else so i’m going to come here and we’re just going to edit this app go is the best programming language in the world i’m not biased at all yes you can clap for that hell yeah all the php people like no it’s not it’s okay all right so we’re gonna change the message and we’ll look at the diff here all right looks good to me now the contract between the developer is how to build this application and what the dependencies are that’s what the developer owns and that’s okay contract to provide and that’s where the docker file comes in sweet those are the only things i care about my source code my docker file everything else is implementation detail or should be so once we have this we’ll commit it so git commit always use helpful messages like change message oh yeah who does this one do you hate when people do this it’s like what’d you add so you got your message and now we’re going to push it so remember we have this branch get push or gin and then we’re going to push this branch okay so at this point what do you expect to happen you know switch to coupe ctl write some yaml lots of people like that sweet yaml nobody wants to write yaml what you expect is when this gets checked into a branch that it should be somewhere ready for me to test in a staging environment so we go to i’m going to just go here so we can see some of the implementation details if i hit let’s look at our build pipeline how many people have end to end continuous delivery bill pipelines damn we should have had a continuous delivery track because i think once you get to kubernetes this is the next holy grail that you have to do at some point this is going to be table stakes i think it already is but this is where you really should be focusing your time big time so here’s this build pipeline and i have a few rules based on a developer’s intentions one is if i push to any branches set for master based on this regular expression i want to deploy to the staging environment if i tag it then we want to go to qa and we’ll build a container image based on that tag and propagate it to production that is the goal now some of you you’re like where’s the ammo files where’s the deployment descriptor implementation detail folks no one cares or shouldn’t so now we go here and we look at our build history so i pushed to this branch and we see just now a bill completed and succeeded it built an image with this particular commit so this is the commit of the repository so the developer can chase it back to see how it was built and then if we go to our pods and i hit refresh we see we have our app in the staging environment now you don’t actually have to show this to anyone ideally you should have a url that maps to this running application there’s no need to use coupe ctl to figure out what the ip is anyone ever heard of this thing called dns oh it’s sweet if you’ve never used it people reinvent dns all the time i’m like what are you doing that’s dns like no it’s not it’s a distributed service discovery dns that’s what you invented again this week so we see our changes live so what happened here let’s not worry about it quite yet so once it’s in staging the next thing i want to do is maybe it looks right to me i’m going to merge this i’m going to tag a release hopefully you’re tagging releases and not building docker containers on your laptop if you are be in shame and then change your behavior all right so the next thing to do as a developer is i’m going to check out master and i’m going to merge in the change message branch everything looks good here and now i’m going to push it or gen master this is the experience when you adopt all these tools that you should be after do not leak your implementation details into the developer workflow you don’t have to once i push this change nothing should happen because i haven’t done the thing to trigger everyone at once what do i need to do to trigger the next step tag or release so we’re going to tag this release git tag 1.0 git push origin tags so now that i’ve tagged this release my workflow i assume at some point if the bill passes and the test pass that it should end up wherever you tell me qa is i don’t care where it is give me a url if you give me a url we’ll come here and we know we need to go through our build pipeline so we’ll take a peek at that so if we go to images really quickly and we’ll see down here in the hello world we see that right now our build is still probably going and when it’s done what we should see is 1.0 tagged here and if that’s successful we should see that deployed to the next environment i’m inpatient so i’m going to hit refresh as if it’s going to go faster it probably won’t let’s look at the bill history so let’s see what it’s doing we’ll take a peek into the build pipeline so what i’m doing now is checking out the repository grabbing the tag and then i’m actually using kubectl to actually check out my yaml file from a separate git repository that actually holds my kubernetes deployment structure for that entire environment now the nice thing about this is that you can actually have multiple teams work on those kubernetes configs independently of the pipeline and what i’m doing in this case is i’m checking out the infrastructure repository and i’m using the kubectl patch command to patch the one container image that i’m concerned with and i’ll issue a commit to the infrastructure repository i want a history for all of these things so if this works out well i can go to my qa infrastructure repository and see now we have a new commit if i look at that new commit what we’ll see here is the change that i expect all right and again no one needs to see this repository but it’s a good place to actually track all the changes that you do whether it’s a human doing the change or ci cd system doing the change so once this is in place where do you now expect the deployment to be let’s try with just the url without looking at anything else so we come here and we’ll go back to dns and then we’ll just hit qa so it looks like it’s working and we’ll verify the deployment here hit refresh and it’s there now there’s a missing piece here if you don’t want people asking for a cube ctl again then you got to give them visibility they can’t just be flying blind here that’s when people start asking for access again so you got to give them more tools now anyone notice anything hidden in my deployments i’m going to show you something here coop ctl get pods actually i’m going to show you the functionality that i get first so i’m going to go here and i have graffana running over a proxy and i’m going to show you what value looks like right we get cop so much in the tools that sometimes we forget about the actual value that they bring what you want to do is give people visibility one way of doing that is to give them a curated dashboard that gives them insight into what’s actually happening so this is my qa dashboard and ideally if someone were to hit the qa url maybe they’re doing some testing you got to put those sleeps oh we need curl actually okay so now we’re hitting the actual application maybe you’re running some integration tests so ideally if you give people visibility they will stop asking for tools like coop ctl to do their job because now they can actually just kind of observe what’s actually happening in the cluster so in addition to metrics you probably want logs http traces the more visibility or tools that you give to people the less they’re going to probably want to touch the infrastructure right and that frees you up to actually upgrade it move it around without people getting attached to where it’s actually running so once everything looks good in production what do we expect to happen now not everyone can deploy straight to production this is why i don’t like to show these end-to-end pipelines where you go straight to production anyone ever had an outage in production what does the developer say it was a small change that everything is down not my fault man so you might want to put that manual check in there just to say what’s actually happening so let’s take a look and see what’s going on so in my case i need to trust but verify so what i’m doing in the last stage of my pipeline i’m just going to issue a pull request from the bill system the bill system will also propose changes to the infrastructure so that the team can understand what’s actually happening until we get a little bit more comfortable with the system so in this case i’m proposing a commit message by this system that i want to change this particular container image to this tag the same one that’s used in qa a lot of people get confused about this one do not rebuild the container image between environments that defeats the whole point of testing in qa but maybe between stage and as you’re kind of prototyping and figuring things out review the change looks good 1.0 and then now what i want to do here is review it looks good to me rebase and merge confirm and you guessed it if i do that what do you expect it to be you expect it to be in production so we’ll look at the last part of the bill pipeline so we’ll come back here and as soon as i merge into the master branch whether i was doing it as a human maybe i want to change the entire structure of the deployment config maybe i want to update the service type that’s underneath there again all that is extracted away from the developer because what i’m doing now is i’m taking all of my infrastructure stuff and i put it in this one directory structure and what you can do is cube c tail apply recursively and everything underneath the kubernetes directory it will just apply again including the changes that you made from the ci cd system that are just concerning the app so that means your services ingress all your deployments if they’re all combined and that way you can actually stop messing with this thing once you start to get to a stable point where you’re not changing your configs as much anymore now what tools should you use to manage your configs go to some sessions there’s ton of them tons of them there’s helm there’s case on it there are many of these tools that will make it much easier than coupe cto patch to manage these things so once you all have this up and running the last thing that you think about is how do i actually do other changes without going to the pipeline you need an easy way to break the glass right you can’t have everyone there may be emergencies that you need to sidestep the process at some point you’ll notice one thing here as i wrap up is that in our infrastructure directory when you look at my deployment config you’ll notice one thing missing i don’t include the replica account if i don’t hardcode the replica account here i’m free to use an auto scaler okay side by side or you can use some other system within the infrastructure to do this so we’re going to see how we can actually integrate a last system to see what this looks like once we get everything working so we’re going to do is look at the number of pods in production and see what it looks like when another system comes up on top so here we’ll say while true get the pods context production so if you’re new to kubernetes you can actually use coop ctl when you have to and if you do it actually does support multiple clusters go away siri alright we see we have one pod running and since the replica count is not included in my config we’ll do our last bit to see if we can actually have multiple systems interact with the cluster because kubernetes maintains the cluster state so we have multiple actors interacting with these configs at the same time so here we go we’re going to try this last thing demo guys have been on my side so far okay google talk to kubernetes engine okay let’s get the test version of kubernetes engine greetings scale the hello world deployment how many replicas would you like you crazy you’re gonna break everything 10. scaling the hello world deployment to 10 replicas thank you i gotta admit that was pretty dope and with that i would like to end the presentation thank you .