Hi everyone, author here, I put this small utility together while exploring the idea of using existing tools to import macroeconomic data from the BLS vs a pure homemade solution with a full-on Clojure project.
With that project I ended up settling on using wget, my tsv2csv tool, and pgfutter for streaming data into postgres and GNU parallel to create and import all tables at once.
I couldn't resist creating this tool at the time because, honestly, I write LISP faster than I adjust to conventional tooling like sed and awk.
It's still a goal to learn those better, but when it came to full on scrubbing of rows and columns, I already had code like tsv2csv in my homegrown BLS importer and I knew how simple it was.
It's made to participate in streaming so it processes line by line from stdin to stdout.
With that project I ended up settling on using wget, my tsv2csv tool, and pgfutter for streaming data into postgres and GNU parallel to create and import all tables at once.
I couldn't resist creating this tool at the time because, honestly, I write LISP faster than I adjust to conventional tooling like sed and awk.
It's still a goal to learn those better, but when it came to full on scrubbing of rows and columns, I already had code like tsv2csv in my homegrown BLS importer and I knew how simple it was.
It's made to participate in streaming so it processes line by line from stdin to stdout.
Thanks!