Wildfire PySpark — Academic machine-learning study

Case snapshot

Problem

Predict final wildfire-size classes from discovery-time information under extreme class imbalance.

My role

Individual university-project author responsible for preprocessing, feature engineering, distributed training, and evaluation.

What I changed

Removed leakage fields and engineered temporal and spatial features.
Compared scaling and PCA effects across distributed model runs.
Measured the precision cost of weighting rare wildfire classes.

Result

Class weighting raised rarest-class recall from 0 to 0.7518, but overall accuracy fell to 32% and rarest-class precision to 0.0099.

The assignment tested scale and imbalance together

The 1.88 million-record dataset made Spark appropriate, while the seven wildfire-size classes created a much harder evaluation problem than overall accuracy suggested.

Leakage and class frequency could invalidate the result

Containment date and final fire size were removed because they would not be known at discovery. Temporal and geographic features were engineered before the shared Spark ML pipeline.

Each experiment isolated a trade-off

Random forest, decision tree, logistic regression, and gradient-boosted trees were compared. Scaling reduced one random-forest training run from 52.55 to 27.14 seconds, while PCA reduced one GBT run from 154.69 to 47.09 seconds with near-equivalent reported accuracy.

Rare-event recall exposed the model’s weakness

The baseline random forest had zero recall for classes C through G. Weighting raised class G recall to 0.7518, but precision fell to 0.0099 and overall accuracy to 32%.

The honest result is a failed production trade-off

The study demonstrates distributed experimentation and careful evaluation, not a deployable wildfire predictor. Improving a rare metric is not useful when false positives become overwhelming.