【ESP32 物聯網實戰 10】樞紐：資料匯出與大數據銜接，打造完整端到端架構

今天，我們將把物聯網與我們上一個系列【PySpark 大數據處理】完美結合，示範如何將雲端收集到的感測日誌匯出，打通從「硬體採集」到「大數據分析」的完整端到端 (End-to-End) 藍圖！

2. 對接 PySpark：用大數據引擎提煉價值

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, to_timestamp, date_format, round, avg

spark = SparkSession.builder.appName("IoT_Data_Analysis").getOrCreate()

df_raw = spark.read.csv("feeds.csv", header=True, inferSchema=True)

df_clean = df_raw.withColumnRenamed("field1", "Temperature") \
                 .withColumnRenamed("field2", "Humidity") \
                 .withColumn("Record_Time", to_timestamp(col("created_at"), "yyyy-MM-dd HH:mm:ss")) \
                 .withColumn("Date", date_format(col("Record_Time"), "yyyy-MM-dd")) \
                 .select("Date", "Record_Time", "Temperature", "Humidity") \
                 .dropna()

df_clean.createOrReplaceTempView("iot_sensor_data")

insight_sql = """
    SELECT 
        Date,
        ROUND(AVG(Temperature), 2) AS Avg_Temp,
        ROUND(AVG(Humidity), 2) AS Avg_Humi,
        MAX(Temperature) AS Max_Temp_Recorded
    FROM iot_sensor_data
    GROUP BY Date
    ORDER BY Date ASC
"""

df_report = spark.sql(insight_sql)
print("=== 本週溫室環境每日監控報表 ===")
df_report.show()

3. 結語：萬物互聯，數據為王

【ESP32 物聯網實戰】10 篇連載在此正式完結！希望這套系列教學，能為資管系的學生或是渴望跨領域的初學者，點亮一盞通往全端資料工程師的明燈。我們，下個系列見！