What are we going to do exactly?

Are you are tired for using kubectl whenever you want to issue a simple command like viewing the pods in some namespace, listing the services, the ingresses…etc.? Do you use Slack in your workplace (and who doesn't!)? Please read along.

In this post, I will show you how you can create your very own Slack bot. It will listen to you and, happily, transform your words into kubectl commands. Don't worry, it's not a big deal as you may've thought, and it does not require a lot of code.

What do you need to make this happen?

  • A Kuberbetes cluster that you have access to (at least you should be able to create pods). * Docker and access to a Docker registry (you can use the default Docker Hub or any public/private registry of your choice). * A Slack account. * Some spare time.

How does it work?

OK, let's have a very brief idea about how this workflow is possible. We'll use a very well-known service called Hubot (it's pronounced hew-bot). It contains a list of boilerplate scripts written in CofeeScript that enable you to easily create bots that interact with multiple chat applications like Slack, HipChat and others.

In our example, we must first register an application on Slack, associate it with our Slack account and obtain the authentication token. Using this token, our Hubot script can hook itself to Slack and listen for chat. Whenever you or one of your team mentions the bot name (we'll call it kubot), whatever is sent in this chat is transmitted to the bot. It's then up to the CoffeeScript that you placed there to respond to the text that it receives.

kubot workflow

Create the Slack app

This is probably the easiest part. Open your Slack application, and click on “+Add apps” at the left hand side of the interface as shown:

Then, in the search box, type “hubot” and click “install”

Click on the green button, “Add configuration” to configure our bot.

Now, try to be creative here! choose a name for your bot, an icon, a description of what it does, the channels that it should be member in (#general is selected by default) and so on.

The most important thing to notice is the API token because this is the bot's ticket to access your Slack workspace and listen to your commands. Make sure you copy this token and save it in a text file for now.

Once you save this configuration, you should be able to talk to your bot on the #general channel. Just type hello @kubot, where kubot is the name that you chose for your bot. You can also chat privately with it just like you do with any other normal user.

Write the CoffeeScript script

If you are coming to Kubernetes and DevOps from the operations domain (like myself) rather than the development one, you may feel a little intimidated. JavaScript was never meant to be a systems language like bash, Perl or even Python. Let alone having to learn a wrapper language that compiles to JavaScript (Duh!). But don't be. It's fairly easy to write and understand. If you've used Ruby before, you'll notice a lot of similarities. Now, enough talking and let's get our hands dirty. I intentionally made this script very basic, it only accepts one command and sends its output back Slack, but feel free to add more levels of complexity and features to it once you get the grasp of the language.

Create a new file called “worker.coffee” and add the following to it:

module.exports = (robot) ->
    robot.hear /List pods/i, (res) ->
        @exec = require('child_process').exec
        command = "kubectl get pods"
        res.send("Getting your pods")
        @exec command, (error, stdout, stderror) ->
            if stdout
                res.send "```\n" + stdout + "\n```"
            else
                res.send ("Sorry that didn't work")

As you can see, the magic happens where @exec gets invoked. It executes the command that was defined earlier in the command variable. We pass in the error, stdout (standard out), and stderror (standard error) as parameters to the function. They will get populated by whatever return value is brought from executing the command.

It's your job now to decide what to do with the output (standard or error). You can parse it for important keywords, use it to issue more commands, apply some filtering to it, or just send it as it is to Slack. The sky is the limit for your imagination here.

To keep things simple, I'm just sending back the results of the command to Slack and, if for any reason there was an error, I'm responding with the friendly message.

Let's build our image

Now that we have our token, and our script, let's build the image that will host all that. The image that will do this task must contain node as well as npm so we'll be using the official NodeJS Docker image. Next, we install the Hubot scripts that will make the magic happen.

It also need kubectl command line tool to be installed so that it can interact with the Kubernetes cluster.

Create a Dockerfile and add the following:

FROM node:latest
RUN apt-get update && \
    apt-get install -y apt-transport-https && \
    curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - && \
    echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" >> /etc/apt/sources.list.d/kubernetes.list && \
    apt-get update && \
    apt-get install -y kubectl
RUN npm install -g yo generator-hubot coffeescript && mkdir /kubot && chown node:node /kubot
RUN chown node:node /home/node/.kube/config
USER node
WORKDIR /kubot
RUN npm install hubot-scripts && npm install hubot-slack --save && yo hubot --owner="Ahmed El-Fakkarany <abohmeed@gmail.com>" --name="kubot" --description="Manage your cluster through Slack" --adapter="slack" --defaults
COPY worker.coffee /kubot/scripts/
CMD HUBOT_SLACK_TOKEN=${HUBOT_SLACK_TOKEN} ./bin/hubot --adapter slack

The Dockerfile starts by setting the base image, node. Then, it installs the kubectl command-line tool. If you've ever worked with Kubernetes, this line should look very familiar to you.

Then it uses npm to install yeoman, coffeescript and the necessary Hubot generator package. It creates a directory that will host the application, /kubot, and sets the required ownership so that node can walk free inside it.

Next, and using the node user, it installs the Hubot scripts that makes Slack interaction possible. Once done, it uses yeoman to initialize the project. You can add the owner, the app name and description as environment variables and have them called when the container runs. But I'll leave that to you as an assignment.

Our CoffeeScript file is copied along to the scripts directory under /kubot. That's where Hubot will search and execute scripts.

Finally, we specify the command that will be used to keep this container running and listening for Slack events. Notice that we're passing the Slack token as an environment variable here.

You can (and should) try this image locally on your machine before deploying it to a hosting provider so that you can quickly spot and react to any errors. Let's do that:

docker build -t afakharany/kubot .

Notice how I tagged the image with my Docker Hub username so that I can later push it to the registry.

With the image built, let's run the container locally:

docker run --name kubot -d -e HUBOT_SLACK_TOKEN=xoxb-455666822692-603012545189-mYQnqXeTGIH3bQscGgPi2gKV afakharany/kubot

Then you can have a look at the logs to ensure that the container has successfully authenticated with your channel by running docker logs kubot. You should see a message like the following:

The bot is actually connected to your Slack workspace now and listening to you. But since it is not yet connected to Kubernetes, any kubectl commands will fail. However, you can try and see the friendly error message if you want a proof that everything is working as expected. Just type “@kubot list pods” in the #general channel and hit Enter.

The final step in this stage is to push the image to Docker hub. If you're using another container registry like GCR, feel free to push it there. Just remember to tag the image accordingly. In my case I will push to Docker using the following commands:

docker login
# Enter your credentials
docker push

The process may take some time depending on your internet connection speed.

Where should we deploy our bot?

There are many ways you can host your Hubot bots. The only thing that you need is that the machine on which the bot runs is always internet-connected so that it can respond to the events sent by Slack. So, a virtual machine on any cloud provider, Heroku, or your own self-hosted server is perfectly fine. Some people may even take this one step further and go serverless by creating a Lambda function on AWS with an API gateway (other cloud providers have similar functionality as well).

However, in our case, it is not feasible to deploy our bot on any publicly-exposed service. Remember, we are using it to receive and execute commands from Slack. So, you don't want someone exploiting this and injecting malicious instructions to your cluster (which can get very ugly).

So, that being said, the most suitable way of deploying a bot that talks to Kubernetes is inside Kubernetes as a pod! This will ensure that it is not publicly exposed, will not need credentials stored on the container as it can make use of the service account (more on that later). This adds an extra layer of security. So let's do that.

Creating a service account for our pod

Before going ahead with our pod deployment, we need to create a service account that will be used by this pod to access the cluster and issue commands against the ApiServer. One way of doing this is copying your kubectl config file along when building the image. This is a very bad idea. Specially if you are hosting your image publicly on Docker hub. You're not only exposing your credentials, but you're also giving the service the same access level as yours. That's double impact!

Instead, we'll create a service account on our cluster. Pods can use this account to perform whatever actions they need on the ApiServer, as long as the account authorization level permits it.

In order to make this happen, you'll need to add yourself to the cluster-admins group. Create a new file called admin.yaml and add the following:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cluster-admins
subjects:
- kind: User
  name: "YOUR USERNAME"
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: ""

Apply this configuration by running kubectl apply -f admin.yaml

Next, we need to create our service account and grant it the appropriate permissions. Create a new file called kubot-role.yaml and add the following:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubot
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: kubot-role
  namespace: default
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kubot-default
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubot-role
subjects:
- kind: ServiceAccount
  name: kubot
  namespace: default

Here, I'm granting the account access to the get and list verbs on the pods resource only. I think you get the idea by now. You can fine tune the access level that you want to grant to your account.

Apply this configuration using kubectl apply -f kubot-role.yaml

Creating a deployment file for our pod

Now that we have a service account at our disposal, let's create a deployment that used this account to grant the required access to the pod. Create a new file called kubot-deployment.yaml and add the following:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubot-deployment
  labels:
    app: kubot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubot
  template:
    metadata:
      labels:
        app: kubot
    spec:
      serviceAccountName: kubot
      containers:
      - name: kubot
        image: afakharany/kubot

A very straightforward deployment. I could've even created the pod directly without using a deployment wrapper. But I'm just making room for future enhancements. Notice the serviceAccountName in the spec stanza where we are mentioning the service account created earlier.

Go ahead and apply this configuration by running kubectl apply -f kubot-deployment.yaml

Monitor the state of the pod by running kubectl get pods | grep kubot. I'm assuming that you're running the pod against the default namespace.

Once it is in the running space, you can double check that kubectl will work properly inside the pod. Open a bash shell with the pod using a command like the following:

kubectl exec -it kubot-deployment-8476487fd-7wr8q bash

Now, run kubectl get pods and you should get a list of the currently running pods on the default namespace.

Time to test your work

Here comes the moment that we've been all waiting for. Open you Slack application and go to the #general channel. Type @kubot list pods. You should kubot replying to you with the output of the command.

Where to go from here?

This is far from perfect. Even far from complete as well. Lots of enhancements can be made to this model. As I mentioned earlier, the sky is the limit to what you can do. For example:

  1. Configure a phrase that will contact your version control system, pull the deployment code, apply it to the cluster and inform you about the results. 2. Perform backups 3. Kill pods that match a regex. 4. And many more

Thank you for reading this article. If you enjoyed it, you may want to visit my Udemy instructor page where I teach DevOps, AWS, and Linux courses.