Spark quick commands – Scala

By | April 27, 2017
  1. Save file to HDFS with custom delimiter in Spark:
    import spark.sql
    
    val df = sql("""
    select * from test_db.test_table1
    """)
     
    df.write.format("csv").partitionBy("year", "month").mode('overwrite').option("delimiter", "|").save("/user/cloudera/project/workspace/test/test_table1")
    
    
    

Leave a Reply

Your email address will not be published. Required fields are marked *