Bashscript upload files to hdfs

Question

I am trying to create a bashscript to upload files from the local edge node filesystem to hdfs. I was wondering a good way to add the timestamp in the file. Having some problems with getting timestamp to work.

#!/bin/bash
echo Running upload script to hdfs...
timestamp(){date +"%T"}

hdfs dfs -put /home/myname/folder1/* /user/myname/example_1_$(timestamp).txt
hdfs dfs -put /home/myname/folder2/* /user/myname/example_2_$(timestamp).txt

Answer 1

Using date +%T is not possible as the command result would contain : characters in it like 11:12:45, and creating filenames with : character is not possible in HDFS. See Hadoop-3275.

Try this command in the script,

hdfs dfs -put /home/myname/folder1/* /user/myname/example_1_`date +%H%M%S`.txt

This will create filename like /user/myname/example_1_111245.txt.