Last week I ended up spending more time than I expected on debugging. I spent most of the time figuring out why a library did not accept my freshly generated RSA private key. I was supposed to provide the RSA private key as an environment variable RSA_PRIVATE_KEY. I wanted to store the RSA private key in a file private-key.pem so my first attempt became RSA_PRIVATE_KEY=\"$(cat private-key.pem)\" docker-compose up -d, which failed due to decoding issues by the library. It would appear that the library was very strict on the format of the private key. After a lot of debugging, I managed to figure out how I was supposed to provide the private key and I will try to share my learnings here.

During the debugging, I learned a ton about a few UNIX tools. I realized that we have a great toolbox in front of us all day which we should try to get to know a little better. This toolbox is part of Unix Development Tools and comes pre-installed on most UNIX systems (including MacOS). I will go through a subset of these tools and explain how they helped me debug the decoding issue with the private key.

The private-key.pem referenced in the following command was generated by openssl genrsa -out private-key.pem 3072

File

The File command classifies the specified file. The command determines the file type, and by providing the -I flag, the tool displays the mime file type. The mime type corresponds to the media type and can help you verify that the file is of the type/format you expect it to be.

» file private-key.pem
private-key.pem: PEM RSA private key
» file -I private-key.pem
private-key.pem: text/plain; charset=us-ascii

The above output from File was reasonable, it shows that the file is of the type that we expect. If file private-key.pem would have returned private-key.pem: ASCII text, it would have been a red flag indicating that the file is not of the expected type.

Sed

By viewing the content of private-key.pem, I could see that it contained linebreaks, and I was afraid that those linebreaks caused the issue. I wanted to replace the linebreaks with the newline character \n, and I used sed from the toolbox to achieve that.

Sed reads the specified input stream and modifies it according to the given commands or regex.

» sed 's/$/\\n/' private-key.pem
-----BEGIN RSA PRIVATE KEY-----\n
MIIG4wIBAAKCAYEA4CN/W5vGK8qbYN7zL7s2g5jhI5afW3kfXbITCDe19Dwt4/p7\n
/k8+cEFFxnC+yJQpcgBm7nbWkMRkx53RpR2UHlMpcKhCImPnl5IN0Zv6L9xsQvuL\n
...
prO6CylgkdImkNN4BN/sSDwTFTS7Aqed+YpdeNTUIFy4HQVYYUBw\n
-----END RSA PRIVATE KEY-----\n

Above, the sed command s (substitute) was used. This command changes all occurrences of something to something else. The syntax of the sed s command is: sed s/before/after, meaning that s/$/\\n/ substitutes the end of the line (indicated by $) with with actual value \n. Unfortunately, this substitution did not solve my problem.

Hexdump

The next tool to grab from the toolbox was Hexdump which displays the file content in different formats. One of the formats is hexadecimal. Hexdump is very helpful in displaying hidden characters that are difficult to identify manually, for instance, linebreaks, newlines, tabs, and spaces.

» hexdump -C private-key.pem
00000000  2d 2d 2d 2d 2d 42 45 47  49 4e 20 52 53 41 20 50  |-----BEGIN RSA P|
00000010  52 49 56 41 54 45 20 4b  45 59 2d 2d 2d 2d 2d 0a  |RIVATE KEY-----.|
00000020  4d 49 49 47 34 77 49 42  41 41 4b 43 41 59 45 41  |MIIG4wIBAAKCAYEA|
...
00000970  48 51 56 59 59 55 42 77  0a 2d 2d 2d 2d 2d 45 4e  |HQVYYUBw.-----EN|
00000980  44 20 52 53 41 20 50 52  49 56 41 54 45 20 4b 45  |D RSA PRIVATE KE|
00000990  59 2d 2d 2d 2d 2d 0a                              |Y-----.|

The -C flag sets the output format to hexadecimal. Each row contains 16 characters, each displayed as their hexadecimal representation. The first value is 2d, and by looking up 2d in an ascii-table, I could see that 2d corresponds to the - character, meaning that we have no unwanted characters at the beginning of the file. The last binary value was 0a which represents LF which is the character that indicates the end of the line. It is not the same as newline \n or carriage return \r. These characters can be tricky to handle, but I had already tried to replace them with newlines using sed without any luck.

Diff

Running out of ideas for a potential next step I decided to test with a copy of the private-key-pem, named private-key2.pem. I copied the content of private-key.pem by running pbcopy < private-key.pem, and I opened my editor and pasted the content into private-key2.pem. To my surprise, the application started when I ran it with private-key2.pem, the COPY of private-key-pem.

To understand why this was possible, I grabbed a new tool from the toolbox, Diff. This tool compares files line by line.

» diff private-key.pem private-key2.pem
39a40
>

The output of Diff above shows a minor difference between the files. 39a40 represents the change: starting at row 39, add line(s) until reaching row40.

Running the Diff command in reversed order shows the same thing but from the perspective of private-key.pem

» diff private-key2.pem private-key.pem
40d39
<

40d39 represents the change: starting at row 40, delete line(s) until row 39. The output from Diff can be seen as instructions for making the files identical.

Conclusion

To conclude, all that was missing was a \n at the end of the file. The key generation command excluded the necessary new line, but my editor decided to add it. A lot of work for just a missing newline, but I realized that the included toolbox is quite powerful. We should all try to get familiar with the tools, you never know when you may need them in the future.

Honorable mentions

There are a few more tools I want to mention before I move on.

pipe |

Allows one to use the output from one command as input to another command.

» sed 's/$/\\n/' private-key.pem | hexdump

0000000 2d2d 2d2d 422d 4745 4e49 5220 4153 5020
0000010 4952 4156 4554 4b20 5945 2d2d 2d2d 5c2d
0000020 0a6e 494d 4749 4135 4249 4141 434b 5941

grep

Search for information in files.

» cat private-key.pem | grep RSA
-----BEGIN RSA PRIVATE KEY-----
-----END RSA PRIVATE KEY-----

man

Show the manual and instructions of the given command.

» man grep

GREP(1)                      General Commands Manual                     GREP(1)

NAME
 grep, egrep, fgrep, rgrep, bzgrep, bzegrep, bzfgrep, zgrep, zegrep, zfgrep
 – file pattern searcher

SYNOPSIS
 grep [-abcdDEFGHhIiJLlMmnOopqRSsUVvwXxZz] [-A num] [-B num] [-C[num]]
      [-e pattern] [-f file] [--binary-files=value] [--color[=when]]
      [--colour[=when]] [--context[=num]] [--label] [--line-buffered]
      [--null] [pattern] [file ...]

DESCRIPTION
 The grep utility searches any given input files, selecting lines that match
 one or more patterns.  By default, a pattern matches an input line if the
 regular expression (RE) in the pattern matches the input line without its
 trailing new line.  An empty expression matches every line.  Each input line
 that matches at least one of the patterns are written to the standard
 output.

• Unix