I'm pleased to share our research published at the 2022 IEEE Bombay Section Signature Conference (IBSSC), co-authored with Riya Adsul and Saumya Pailwan at NMIMS University.
DOI: 10.1109/IBSSC56953.2022.10037478
Abstract
Lung cancer is one of the most common types of cancer and the leading cause of cancer-related death in humans. Early diagnosis is critical — survival rates improve dramatically when the disease is caught before symptoms manifest. This research examines lung cancer symptoms and risk factors using Machine Learning algorithms to identify lung cancer patients from healthy individuals, and further distinguishes the three pathological subtypes of Non-Small Cell Lung Cancer (NSCLC). The optimal data mining strategy is determined by comparing classifier performance across two independent datasets. For the binary detection task, SVM achieves the best results; for NSCLC subtype classification, XGBoost performs best.
Introduction
Lung carcinoma causes uncontrolled cell growth in lung tissue and accounts for disproportionate cancer mortality worldwide — particularly in North America, Europe, and East Asia. Long-term tobacco use accounts for approximately 85% of cases, with radon gas, asbestos, and air pollution also contributing.
Lung cancer is classified into two primary types:
- NSCLC (Non-Small Cell Lung Cancer): The more common and slower-growing type, with three subtypes — Adenocarcinoma, Squamous Cell Carcinoma, and Large Cell Carcinoma.
- SCLC (Small Cell Lung Cancer): Rarer but more aggressive, typically originating near the bronchi.
The goal of this paper is two-fold: (1) detect the presence of lung cancer from symptom and risk factor data, and (2) classify the type of NSCLC if cancer is detected. This sequential pipeline addresses a gap in the literature — most prior work handles only one of these tasks.
Datasets
Dataset 1 — Binary Detection (data.world / UCI)
Used to distinguish lung cancer patients from healthy individuals.
| Attribute | Values |
|---|---|
| Gender | M / F |
| Age | Numeric |
| Smoking | YES=2, NO=1 |
| Yellow Fingers | YES=2, NO=1 |
| Anxiety | YES=2, NO=1 |
| Wheezing | YES=2, NO=1 |
| Alcohol Consumption | YES=2, NO=1 |
| Coughing | YES=2, NO=1 |
| Shortness of Breath | YES=2, NO=1 |
| Swallowing Difficulty | YES=2, NO=1 |
| Chest Pain | YES=2, NO=1 |
| Lung Cancer (target) | YES / NO |
- Instances: 284 | Attributes: 16 (15 predictive + 1 class)
- Preprocessing: Class imbalance corrected via minority upsampling (random_state=4, n_samples=269)
- Top correlating features: Allergy, Alcohol Consumption, Swallowing Difficulty
Dataset 2 — NSCLC Subtype Classification (UCI)
Used to classify the three pathological subtypes: Lung Adenocarcinoma, Squamous Cell Carcinoma, and Large Cell Lung Cancer.
- Instances: 170 | Attributes: 57 (56 predictive + 1 class)
- Data type: Nominal integer values (0–3), including tumor marker levels and TNM staging
- Preprocessing: Dataset is balanced — no resampling required
Methodology
Classifiers Evaluated
Twelve classifiers were trained and compared on both datasets:
| # | Algorithm | Key Hyperparameters |
|---|---|---|
| 0 | Linear Discriminant Analysis (LDA) | Default |
| 1 | Logistic Regression | solver=lbfgs, max_iter=200 |
| 2 | AdaBoost | Default |
| 3 | SVC | C=1, gamma=1, kernel=rbf |
| 4 | Random Forest | max_depth=5 |
| 5 | Decision Tree | criterion=entropy |
| 6 | Gaussian Naïve Bayes | Default |
| 7 | K-Nearest Neighbors | n_neighbors=7 |
| 8 | Gradient Boosting | learning_rate=0.01, random_state=1 |
| 9 | XGBoost | learning_rate=0.01, random_state=1 |
| 10 | Voting Classifier | Decision Tree + SVC |
| 11 | MLP Classifier | hidden_layer_sizes=800, random_state=50 |
Evaluation Protocol
- Train/test split: 80/20, with
StandardScalerfor scale-free features - Metrics: Test accuracy, training accuracy, precision, recall, F1 score
- Cross-validation: 4-fold cross-validation on the best classifier (SVC) with
decision_function_shape='ovr'
Results
Dataset 1 — Binary Lung Cancer Detection
| Algorithm | Test Accuracy | Train Accuracy |
|---|---|---|
| LDA | 87.96% | 91.6% |
| Logistic Regression | 87.04% | 91.4% |
| AdaBoost | 92.59% | 96.5% |
| SVC | 79.81% (raw) → 98.14% (CV) | 99.7% |
| Random Forest | 91.67% | 94.4% |
| Decision Tree | 95.37% | 99.7% |
| Gaussian NB | 88.89% | 86.1% |
| KNN | 85.19% | 88.3% |
| Gradient Boosting | 86.11% | 90.7% |
| XGBoost | 86.11% | 89.7% |
| Voting Classifier | 95.37% | 99.7% |
| MLP Classifier | 91.67% | 90.9% |
Winner: SVC — 98.14% test accuracy after 4-fold cross-validation (precision 96.3%, recall 100%, F1 98.1%). Cross-validation score: 0.986.
Dataset 2 — NSCLC Subtype Classification
| Algorithm | Test Accuracy | Train Accuracy |
|---|---|---|
| LDA | 71.43% | 87.5% |
| Logistic Regression | 57.14% | 100% |
| AdaBoost | 42.86% | 70.8% |
| SVC | 57.14% | 95.83% |
| Random Forest | 57.14% | 100% |
| Decision Tree | 57.14% | 100% |
| Gaussian NB | 28.57% | 91.66% |
| KNN | 71.43% | 66.6% |
| Gradient Boosting | 71.43% | 100% |
| XGBoost | 85.71% | 95.8% |
| Voting Classifier | 71.43% | 100% |
| MLP Classifier | 71.43% | 100% |
Winner: XGBoost — 85.71% test accuracy, outperforming all other classifiers on NSCLC subtype classification.
Discussion
The results validate the sequential detection-then-classification pipeline:
Why SVM wins on Dataset 1: SVM's decision boundary complexity is determined by support vectors rather than data dimensionality. For symptom-based binary classification where samples are linearly separable in higher-dimensional space (via the RBF kernel), SVM consistently outperforms ensemble methods on this dataset size.
Why XGBoost wins on Dataset 2: The NSCLC subtype dataset has 57 attributes and only 170 instances — a high-dimensional, sparse regime. XGBoost handles missing values natively, is robust to unscaled data and outliers, and its gradient-boosted decision trees capture complex feature interactions between tumor marker levels and TNM staging that simpler linear models miss.
Key tradeoffs across classifiers:
- Naïve Bayes is fast but assumes feature independence — a poor fit for correlated clinical symptoms
- Multilayer networks handle complex numerical patterns but are the most computationally expensive and require careful tuning
- Random Forest addresses decision tree overfitting via bagging but plateaued at ~57% on Dataset 2
- Gradient Boosting and Voting Classifier achieve competitive accuracy but are outperformed by XGBoost on both datasets when hyperparameters are tuned
Conclusion
This study demonstrates that cancer can be detected and subtyped from tabular symptom and risk factor data alone — without CT scans. The SVM-then-XGBoost pipeline achieves clinically meaningful accuracy (98.14% for detection, 85.71% for subtyping) that could support pre-diagnosis screening, reducing unnecessary and costly imaging procedures.
Future directions include integrating CT scan imagery with symptom data via multimodal deep learning, applying text mining to unstructured clinical notes, and evaluating the pipeline prospectively on held-out clinical cohorts.
Published at the 2022 IEEE Bombay Section Signature Conference (IBSSC). Co-authored with Riya Adsul and Saumya Pailwan, NMIMS University, Mumbai.