(note: this may not work if the file is named .CSV, according to threads online. I haven't tested that because this works and I'm sick of dealing with it)
Azure Data Factory (ADF) allows you to write files directly to Azure Blob Storage, which is nice. However, there have been problems over the years getting it to write without double-quotes. I use pipe-separated, which with my data set won't ever have double-quotes. But getting them removed can be a pain, possibly involving moving from Copy Task to Data Flow Task (which spins up a Spark cluster!) or using an Azure Function, which can also be a pain to set up.
However, this code works (at least if the file is named .txt). In my variables, I replace the ColumnDelimiter with a pipe (|) symbol. You can probably just do that in this definition, but mine works and I'm Not Touching It. The real magic is not assigning anything to QuoteChar - any value means that it doesn't count as a NULL, so it's not handled right. I've tried to parameterize as much as possible - but you can't seem to parameterize the dataset itself. :-(
Hope this helps.
No comments:
Post a Comment