[All work](/work?type=work)

# Wildfire PySpark

I worked with 1.88 million US wildfire records for a university project, and got into how a model could look accurate while missing the larger fires.

University project, done solo: preprocessing, features, training and evaluation.

Recorded random-forest results: class G recall rose from zero to 0.7518 with weighting; class G precision was 0.0099 and overall accuracy was 32 percent.

PySpark experiment

1.88M records

- Class G recall

  **0 → 0.7518** With class weighting

- Weighted class G precision

  **0.0099**

- Weighted overall accuracy

  **32%**

Recorded random-forest results. Class weighting finds more rare fires, with many more false positives.

## 1.88 million records, seven very uneven classes

The 1.88 million-record dataset made Spark appropriate, while the seven wildfire-size classes created a much harder evaluation problem than overall accuracy suggested.

## Removing the leakage

I took out containment date and final fire size, as neither would be known when a fire was discovered. Then I built the temporal and geographic features into the shared Spark ML pipeline.

## The model comparison

Random forest, decision tree, logistic regression, and gradient-boosted trees were compared. Scaling reduced one random-forest training run from 52.55 to 27.14 seconds, while PCA reduced one GBT run from 154.69 to 47.09 seconds with near-equivalent reported accuracy.

## Where it fell over

The baseline random forest had zero recall for classes C through G. Weighting raised class G recall to 0.7518, but precision fell to 0.0099 and overall accuracy to 32%.

## The trade-off does not work

The weighted model found more of the rare fires, though the false positives were too high for me to treat it as something anyone should deploy.

## Outcome

In the weighted random-forest run, class G recall rose from 0 to 0.7518. Class G precision was 0.0099 and overall accuracy fell to 32%, so it was finding more of the rare fires while getting a lot more predictions wrong.

## From this project

- [Wildfire Analysis with PySpark](/reports/wildfire-analysis-with-pyspark)

[All work](/work?type=work)