Thoughts on Automation
Steve Jobs called the computer a bicycle for our minds, which sums it up perfectly. A computer's ability to process data is unmatched in humans. It requires no breaks, makes no mistakes, and does so at a speed that makes it appear magical. A chess-playing computer beat the world chess champion in 1997. But an average chess player working with a computer today can outplay computer alone.
The ever-creative infosec community has started to embrace this hybrid approach as well with things like Human-in-the-Loop tooling.
Humans and computers work so well together because our abilities are complementary. A computer is the master of math, repetition, speed, and consistency. Humans live in the world of nuance, judgement, morals, and reason. A computer, then, is ideal for automation, while a human is best equipped to decide when to automate.
My relationship with automation started with a dead run in the opposite direction. The release of Office 365 rendered years of experience administering Microsoft Exchange obsolete. It was faster, better, and cheaper than anything I could offer.
I left IT for a career in information security on the premise that no matter what we automated, humans would be the ones implementing it. The presence of humans means inconsistency, and when inconsistency exposes a weakness we call it a vulnerability.
I still get uneasy whenever I'm doing anything a computer would do better. But I also embrace the awesome power at my fingertips. You'd be hard pressed to find another industry where smart use of automation can speed up your work by several orders of magnitude. These type of gains just do not exist outside of tech. And certainly not ones that you can realize with a few hours' worth of work.
I've been thinking about automation for years, so I decided I would take some time to flesh out my thoughts and write them down.
Why Automate
The best answer to "why automate?" is because you can. Time is the only thing we cannot get more of. A minute saved by automation is a minute you can spend on a task you cannot automate. Tasks you cannot automate are where you should be spending as much of your professional life as possible. This has been true in tech for quite some time, but is starting to become true elsewhere, too.
Humans are the most expensive part of any solution and if they can be removed, they eventually will be.
What to Automate
What makes a task a good candidate for automation? The first and most obvious answer is that it can be automated. Tasks that involve human interaction, good judgement, or nuance are out. Need to do the exact same thing every day at 4:03 PM exactly? We've got a winner.
Second, the benefits of automation should outweigh the cost of automating the task. Spending 80 hours to save five minutes is a waste of time. By the time you realize your gains, chances are you will no longer be doing that task. That said, if you hate the task with a fiery burning passion, knock yourself out.
An obvious corollary is to avoid automating tasks you enjoy. Although, if you love doing a boring, repetitive, simple task that would be better done by a computer, you should probably make sure that's not a significant part of your job, since it may end up being automated for you.
When to Automate
Automating a task encapsulates the knowledge required to complete that task manually. Automate a task after you've mastered the manual steps. Writing a tool is teaching a computer how to execute a task on its own.
Some characteristics of a good candidate:
- Repetition - The task must be completed regularly.
- Consistency - The task must be done at the same time, or in the same way, each time.
- Speed - The task needs to be done as quickly as possible.
How to Automate
"When a process is automated the automation encapsulates learning thus far, permitting new people to perform the task without having to experience that learning. This stunts or prevents future learning." Automation Should Be Like Iron Man, Not Ultron
One of my favourite things about infosec is that we are a tribe of contributors. A love of open source seems to be one of our defining characteristics and we are all better for this. We are so spoiled with high-quality open source tooling that you could probably get through a good chunk of your career without ever writing a line of code.
But you absolutely should.
Automation is best implemented by the people doing the work. People who have mastered the manual task, and then encapsulate that knowledge in a tool. Using tools written by someone else trades knowledge for expediency. There are only so many hours in a day, so you this is a necessary choice to make, but you should always look for excuses not to.
"Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it?"
When building tools, aim low. The simpler the better. Embrace the Unix philosophy of "do one thing, and do it well". Write tools that take input on stdin and write output to stdout. Embrace the incredible power of the Bash pipeline. Don't overthink things.
You can start with stupid simple things like the commands you type every time you start a specific task. Let's look at how you can automate your recon workflow:
|
|
It took me about 16s to find the above command in my wiki. It took about 5s to create that shell script. I pay for the cost of automation in my first use of run-nmap.sh. This is the ground floor of automation. Add ~/scripts to your PATH and you don't even need to remember where they are.
Repeat this with a few of your other recon tools and now you can do things like this:
|
|
Now you've got a decent recon script that saves you time every time you run it. And guess what, you can now schedule this to run regularly. Maybe then you can diff the results to let you know if anything changes. And notify you in Slack when this happens. And so on.
Congratulations, you've now got nerd superpowers.
Getting Started
Bash is a great way to start because you are likely already working in Bash for a good chunk of your tasks. Any command you type can be run by a shell script. But don't stop there. You've got options! Bash is great for quick and dirty. Go is amazing for concurrency and simple command-line utilities. Need to share something with neophyte programmers or make things easy to tweak? Reading and writing Python is about as frictionless as programming can get.
Whatever language you choose, here are some things worth knowing:
- Reading files - Useful for reading configuration files.
- Writing files - Logging, saving your results.
- Parsing data - Structured (JSON, CSV, XML) and unstructured (Regex, filesystem contents).
- HTTP Requests/Reponses - Send HTTP requests. Extract data from the responses.