How to escape binary data for inclusion into bash script

Question

I want to create a bash script that reads a "payload" of binary data from an external file and spit out another bash script with this data escaped and encapsulated inside a string variable. Example:

• mydata.bin - The source data: a binary data file to be encapsulated
• myencoder.sh - The main perpetrator: a bash script that converts the binary data to a script with string variable
• mypayload.sh - The end result: a bash script generated by myencoder.sh that contains the encoded data as a string variable.

To use this I would run myscript.sh mydata.bin mypayload.sh and myscript.sh would convert/escape/wrap/whatever the mydata.bin file into mypayload.sh

After running this command, the mypayload.sh file would look something like this:

# Generated by myencoder.sh with data from mydata.bin
encoded_data="[...]ugly escaped string representation of the binary data found in mydata.bin[...]"

The problem I am facing that I amunsure how to solve is how the data would be properly encoded. I read that printf "%q" could be used to escape strings, but how to invoke it on data fetched from an external binary file eluded me completely.

So please, any stabs at this and any tips are welcome!

PS: I don't want to introduce any dependencies outside of bash if possible. Depending on bash 4.x features is OK.

PPS: The encoding should favour small size and encoding/decoding performance.

Answer 1

It's very hard to handle zero bytes from bash. You can output a zero byte by

printf %c

but you can't store it in a variable.

It's far easier to use external tools:

xxd < mydata.bin > encoded
xxd -r < encoded > binary

Answer 2

This is not a complete solution to your problem; as others have mentioned dealing with NULLs in your input file is difficult in bash, and I'm sure potential of unicode input files could cause even more headaches.

But I put a little time into thinking how you might use printf %q in bash to do something similar to your suggestion and came up with this quick hack:

echo -n 'myvar="'
while read -r; do
  if [ ! -z "$REPLY" ]; then
    printf %q "$REPLY"
  fi
  echo -n '\n'
done
echo '"'

I'm sure it could break in many ways but maybe it might satisfy some part of your curiosity.

Answer 3

This reply does not answer the question directly because apparently containing 0 bytes in bash variables is not possible and apparently there is no way to escape them into strings that can be safely kept inside bash variables without relying on external tools like xxd or uuencode that are not available by default on the platforms where my script will run.

However this was the solution that I ended up pursuing because it solved my problem closest, namely creating a script that holds arbitrary binary data as a payload that can be handled by the very script that holds it.

I used this article as my source. Basically the approach uses a separator string between the script and the binary data and never lets the bash interpreter reach the binary part at the end.