When I learned a safety lesson - and did a little bit better

· June 19, 2021

Programming   Life of a consultant

I like to automate repetitive and boring things I do in my daily work. This lead to that me write bash scripts from time to time, but the problem is that I’m a newbie scripter. I’m learning as I go.

And the other day I did a cardinal sin in scripting. Not only that - I decided to show the world. Now that is a point to feel ashamed or to learn. I did both.

In this post, I will tell you about how I made my curl-script a little better and a lot safer, but using an old way that’s been in curl for ages.

The background

I was waiting for the summer part of Salt to start and had an hour over. The weather was amazing and I found a nice tree to sit under. Of course, I flipped up the computer and did some scripting.

In fact, I thought that sight was pretty nerdy so I tweeted about it. When I did I added this picture to the tweet.

Coding in the sun

Do you see the problem? Well, it took about 2 minutes before consider the human informed me that I just exposed my GitHub keys.

It was not that bad since this was actually an OAuth token that just has permission to list repositories on GitHub, but that is bad enough.

The script

The script is something I’m very proud of since it saves us from A LOT of work in the instructor’s team. I wrote about it before but very shortly it verifies the integrity of all our labs, tests, and material. It’s about 100 repositories so it’s quite cumbersome to do manually.

The script starts with me getting a list of all the repositories using curl and the GitHub API. Since I want to read the private repositories in our organization I have to use credentials that have those permissions.

This can be done by:

  1. Creating a personal OAuth token for your account
  2. Give it the correct scope
  3. Save the private key

The error

You can then call the GitHub API like this:

curl -u {username}:{oauth toke} -H "Accept: application/vnd.github.v3+json" -s "https://api.github.com/orgs/appliedtechnology/repos?sort=full_name&per_page=100&page=${i}" | jq -r ".[] | select(.archived==false) | .name"

Sad trombone moment, because I just exposed my private key to anyone that uses the script.

The better way

I started to think about letting the user log on before using the script, but that will not use the OAuth token, and I just want the user to have the very limited privileges that I’ve given them.

But curl is amazing and of course, you can pass credentials in other ways. You could set them in environment variables, if you wanted or, the way that I like; use netrc as described here

So, let’s do that again:

  1. Go to your Developer Settings tokens
  2. Generate a new token
  3. The this token the full repo access, check the top-level scope for repos. Nothing else.
  4. Remember to copy the OAuth token

  5. Create a new file on your computer at ~/.netrc with the following content

     machine api.github.com
 login [your login]
 password [your generated OAuth token]
  6. Nothing special in that file, no quotes, etc.

Then update the curl-command to this:

curl -n -H "Accept: application/vnd.github.v3+json" -s "https://api.github.com/orgs/appliedtechnology/repos?sort=full_name&per_page=100&page=${i}" | jq -r ".[] | select(.archived==false) | .name"

curl -n (or curl --netrc) will use the credentials from the ~/.netrc , making your command clean AND safe to check in.

The summary

So, that’s a short little description of how I took someone’s (Thank you consider the human ) advice, as just that, advice and tried to do better.

And I also realized how much greatness has been invested in those all commands that have been around forever.

Twitter, Facebook