Using AI to Write Deterministic Code | by Teri Radichel | Cloud Security

Avoid blanket statements — use the right tool for the job

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

⚙️ Security Automation (Pre-AI). The Code.

⚙️ AI Automation. The Code.

🔒 Related Stories: Cybersecurity | Penetration Tests

💻 Free Content on Jobs in Cybersecurity | ✉️ Sign up for the Email List

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Using the right tool for the job

In the last post I explained how to use a tool I wrote in part with AI.

How much AI did I use to write the code in that repository?

The fact that I was using AI helped me hunt down some of the APIs I didn’t know existed that I could use to build this. In fact, this is what I was trying to do from the start in this blog series, but I couldn’t find an API like this at the time. I ended up scraping the resource names out of the documentation in one of these posts:

I mostly used Q in the AWS console which I explained how to do and how to work around some issues with that in prior posts.

I also used Google’s AI Mode (https://google.com/aimode) in some cases because it was handy when I was not logged into the AWS console, or I was working on a long thread in Q and I just wanted a quick answer on something.

For the most part either one worked for simple code and gave me roughly the same answers. But on some tricky bugs, both got stuck and in a couple of cases, AWS provided a better answer to my bash queries. Google’s AI Mode is surprisingly pretty good at AWS commands and CloudFormation. Most recently Google’s AI mode keeps renaming my variables which is not very helpful when you are fixing a function or script used in conjunction wtih a larger code base.

Google’s UI is very nice. I find Amazon Q in the AWS console to be a bit clunky. It’s hard when you are trying to look at these three screens at once and sometimes the slider to change window sizes gets stuck:

Note that I missed the button at top right of CloudShell where you can maximize the screen. That is helpful.

I think AWS should create a separate website for Amazon Q (and I put in a feature request) so you can have that window open while you’re doing other things in AWS. Having it in a separate window on a separate domain would probably also be more secure. A vulnerability in Q wouldn’t be able to somehow get into some other open window. Same for CloudShell actually. then I could easily switch between tabs. I actually think that the BuilderID site would be a good place to link to different websites with different tools like: AWS Console, CloudShell, or Amazon Q.

For the record, I did test Grok in an earlier post but it’s code was way off at some points. I don’t really want to use OpenAI giving the pending lawsuit either. I had an issue in Amazon Bedrock with foundational models that sent me in an eternal loop between two bugs. I sent some feedback and noticed that in a recent similar situation Q recognized that an earlier but had been re-introduced so maybe I’ll go back and test it again. I dabbled with O3 but still have more comparison testing in general to do. Didn’t try Claude yet (I will once I get more into using the AWS CLI and Bedrock). But I didn’t need anything better for the most part these two worked for me up to the point it didn’t make sense anymore.

Here’s what AI was good at:

Getting started. Generating an initial concept.
I wrote the code line by line using single prompts. By providing very specific instructions, adding the lines of code without error was great — in the very beginning.
Fixing bugs. I did hit some bugs that neither Google’s AI Mode or Q (in could solve. But for the most part one or the other could fix the problem. I would say that Q beat Google’s AI Mode in fixing some tricky bash bugs, but Google’s AI Mode is nice when I think of something and not logged into the AWS console. I can flesh out an idea quickly.
A simple prototype that ran — with some issues, and nowhere near the final code structure.
Small blocks of code. Breaking the code into small functions helped. I can ask a about a single function — but unfortunately I can’t paste that much code into Q. So has to be pretty small or I had to load up the function in chunks as I explained how to do in a prior AI post.
Writing useless text that does not solve the problem. Q tends to write things like “OK I’ll help you do xyz” like annoying customer service reps on the phone that repeat everything you say. That is wasted and unnecessary compute. I wish AI bots would skip unnecessary words like please, thank you, I’m sorry, blah blah blah. Get real. It’s not a human. Don’t waste compute power on that stuff. It takes time to write it to the screen, delays results, and how much power are you wasting writing those words for all the requests processed in a day? I wonder what the impact would be if the useless noise was eliminated.
Writing excessive comments and overly verbose code. If it is obvious what the line of code does do I really need a comment? Can I create a global prompt that says don’t write a comment for every single line of code? Perhaps you want this if you are learning to code but in a lengthy program I don’t need the extra noise. I write comments for things that are complicated or tricky, not every line of code. And I definitely don’t want a comment that says “changed this…”

Here’s an example of excessive comments:

Prompt

Response (plus a bunch of extraneous prose):

#!/bin/bash# Initialize the number
n=0
# Start the loop
while true; do
# Display the current value of n
echo "Current value of n: $n"
# Ask the user if they want to continue
read -p "Do you want to continue? (y/n): " answer
# Check the user's response
if [[ $answer == "y" || $answer == "Y" ]]; then
# Increment n if the user wants to continue
((n++))
else
# Exit the loop if the user doesn't want to continue
echo "Exiting. Final value of n: $n"
break
fi
done

And by the way when I asked:

Q gave me one option. Google’s AI Mode gave me four options.

Here’s what I’d like to have gotten back, and I realize this is personal preference, but all these extra characters are wasted compute in my opinion. There are other options besides a case statement which can run on one line such as using regex. If someone wants to know what a line does they can copy the line and say “Explain this line.”

#!/bin/bashn=0
while true; do
read -p "n=$n. Do you want to continue? (y): " answer
case ${answer} in [yY]) ((n++)) ;; *) break ;; esac
done
echo "n: $n"

Here’s what AI is NOT good at (and I wouldn’t trust it even in an IDE):

Restructuring the code into separate files for better organization.
The overall architecture of the program.
Recursion. It doesn’t recognize when a recursive loop will solve the whole problem at once and how to properly minimize code using recursion.
Anything that starts getting complex. It tends to start swaying wildly like something that is rocking a little bit and then just keeps rocking further and further to each side instead of stabilizing. At some points I just started over or gave up on a particular thread.
Keeping things consistent. It will randomly change a variable name in a small function I’m working on and of course that doesn’t work with the rest of the code I’ve written if I haven’t provided it all the code.
It makes assumptions all the time. I have to explicitly tell it not to make assumptions and look up the correct data — like the AWS CloudFormation schema for a particular resource instead of guessing what it might be.
Sticking to the request. It keeps adding things I didn’t request. Like adding random blocks of code into my functions that I have elsewhere in my code. If I’m calling other functions it decides it needs to create all those functions for me when they already exist. I had to reprimand it the bots a bit and tell them to stay on task and not add things I didn’t ask it to add. When trying to only load up some code I told Q not to analyze the code just record it.
The size limit in Amazon Q chat window is a problem for me. I can’t hardly fit anything in that. Google is better in that regard. When I tried to use Amazon Bedrock, as noted, I had issues I didn’t have in Q. In additionally, I had problems getting other models working but will revisit that shortly. And no, I don’t want to use an IDE and I am going to use q chat but I wish it existed in CloudShell for what I was testing. Now that I have this script working, I plan to spin up some EC2 instances with it and will expand my horizons with the q CLI and see how that works out.

Bottom line is: AI is very useful, but not reliable. It’s not good at architecture or the structure of code. But it’s very good at syntax and generating initial ideas. In the end, I revised and wrote most of the structure myself, but AI was super helpful in getting the syntax right on the first try or finding dumb bugs like a missing quote in a block of code.

Knowing when to use AI vs. an application using deterministic logic

This particular repository demonstrates exactly why deterministic code is better than AI for some use cases.

When I deploy resources on AWS, I want them to be exactly how I want them configured every time. I don’t want some random property value changed or insertion of a Lambda function in my CloudFormation as was the case in a prior post (something I am absolutely opposed to for security reasons and because it is not necessary). You cannot trust AI to write your code consistently every time.

But if you use a deterministic program that uses deterministic logic it will produce the correct results — every time. That doesn’t mean it won’t have bugs or things that haven’t been completed yet. It just means you won’t get random changes. If there’s a bug it will be a bug every time the thing happens that causes the bug and the results will be consistent. If you fix the bug then you know your fix will execute the same way every time after that point. So I want deterministic code and templates to deploy my resources.

However, while writing the code I can have the AI help me produce it and check it to make sure it is right. I can ask AI to quickly generate some code that might give me a different idea about how to implement something. I can ask the AI to tell me how to fix a bug the same as if I were pasting an error message into Google, except that the AI might actually read all those pages for me and give me some sample working code. By the way, I find that Google’s AI Mode is significantly faster than Amazon Q in the console for those types of questions. I still need to test the CLI.

It’s not alway right and sometimes AI can’t figure out how to fix a bug and I have to do it the old-fashioned way. But really, it’s just another form of search but it reads and summarizes the results for you and that can save a lot of time. Just finding the APIs I used for this has helped me write this tool in about a week — the same tool I was going to try to write and didn’t get even close in three years in my prior post. I was basically looking for or going to try to reverse-engineer the data model I now have (which is also deterministic, not coming from AI, so it is accurate.)

I was pondering how to create a template for every resource more quickly in those prior posts, and that problem is mostly solved. But there are still countless security and governance issues covered in my prior posts that are not handled with this code. I’m thinking about how to merge the two ideas. This code incorporates some of the things I was doing in those posts but not all, because I as trying to get this proof of concept off the ground quickly.

The important bit:

You need deterministic code for security policies and governance — not a predictive model.

The other point is that just that trying to produce working templates for various resources took me hours upon hours in the past blog series — tweaking code that has a missing quote or curly brace or an incorrect indentation. I was covering a lot of different topics but I also was trying to generate working CloudFormation templates for the resources I was describing and working with.

I only got a tiny fraction of all the possible templates written for AWS resources within the past three years.

And now, with the help of

1.) a proper data model on which to base deterministic code

and

2.) AI to help me quickly generate a prototype, fix syntax errors, and quickly generate a line of code to complete a specific objective

I was able to produce a way to deploy any AWS resource (with a few problems yet to solve done shortly) in about a week.

If a company wanted to create templates with particular rules, they could start with these generated templates, save a ton of time, and then tweak them to hard code or follow various policies.

You can likely also use this code to deploy AWS Service Control Policies.

But.

Speaking of policy code — that’s one thing that’s another thing that’s not done here. The policy document properties are just a single blob of JSON or yaml. I made a case for turning that into a first class CloudFormation citizen to the AWS CloudFormation team for exactly this reason — if the policies themselves were true CloudFormation types — I’d be done with those right now. They would be handled by the code I’ve written. The same is true for step function JSON blob properties and anything else where the team did not take the time to implement proper CloudFormation but just has you set a blob of JSON for a property.

So that and a few other things are still on my to do list. But check it out! And follow the git repo or my Email List to be notified of updates.

Follow for updates

About Teri Radichel:
~~~~~~~~~~~~~~~~~~~~
⭐️ Author: Cybersecurity Books
⭐️ Presentations: Presentations by Teri Radichel
⭐️ Recognition: SANS Award, AWS Security Hero Former SANS, IANS Faculty
⭐️ Certifications: SANS ~ GSE 240
⭐️ Education: BA Business, Master of Software Engineering, Master of Infosec
⭐️ Company: Penetration Tests & Security Research ~ 2nd Sight Lab

Cloud, SAAS, and Application Penetration Testing
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
🔒 Request a penetration test

Follow for more stories like this:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
❤️ Sign Up my Medium Email List
❤️ X, Threads, Blusky, Instagram, AWS BuildID: @teriradichel
❤️ LinkedIn: https://www.linkedin.com/in/teriradichel
❤️ Mastodon: @teriradichel@infosec.exchange
❤️ Facebook: 2nd Sight Lab