Assessing Algorithm Accuracy with Real Patient Datasets

Context

This project involved testing a healthcare imaging application used by clinicians for diagnostic workflows. The system handled regulated medical data and depended on external services including licensing servers and DICOM integrations. Failures directly impacted clinical usability and deployment readiness across Windows environments. Quality risks were amplified due to hardware variability and third-party dependencies.

Primary Risks Identified

• Dependency on external DICOM systems for core workflows
• Multi-phase image segmentation failures in early builds
• Inconsistent segmentation times across GPU models
• Poor segmentation quality on certain patient scans (e.g., femoral studies, poor image quality)

Testing Approach

Testing focused on environment-driven failure analysis rather than pure functional coverage. I combined exploratory testing with log-driven diagnostics to isolate licensing and rendering failures. CI results were reviewed alongside local reproduction to distinguish configuration issues from product defects. Risk-based prioritization was used to validate startup stability and core workflows before secondary features.

• Regression Testing: Repeated tests across 14+ software builds
• Hardware Benchmarking: Compared performance on GeForce GTX 1060 vs. GeForce 840M
• Patient-Specific Validation: Tested on 8+ patient datasets with varying slice counts (177–1326 slices)
• Failure Analysis: Documented root causes in "Comments" for engineering follow-up

Outcome

Over three release cycles, production incidents dropped noticeably and release discussions shifted from opinions to evidence.

• Performance Improvement: Average ML inference time reduced from ~20 sec to ~7 sec per study
• Stability Gains: Critical loading bugs resolved (e.g., “Not able to open any study” fixed by build 20190603)
• Validation Readiness: Enabled deployment of stable builds for clinical evaluation

Key Lessons

Quality maturity doesn't come from tools - it comes from aligning testing effort with real risk and making results visible.

• GPU impact is significant: GTX 1060 consistently outperformed 840M by ~20 – 30%
• Slice count ≠ linear time increase: Large datasets (e.g., 1326 slices) did not scale proportionally in processing time
• Documentation drives fixes: Clear failure comments enabled precise bug fixes (e.g., “Multi-phase not supported” later resolved)
• Iterative testing is essential: Continuous build-by-build validation ensured steady progress toward release readiness