🏥 Healthcare AIHW ETL & Visualisation
July 2025 · Web Scraping · Streamlit · Data Visualisation
🔍 Project Overview
This project focused on automating the extraction, transformation, and visualisation of healthcare-related statistics published by the Australian Institute of Health and Welfare (AIHW). The goal was to enable quick, dynamic insights into key health metrics using an interactive Streamlit app.
⚙️ Methodology
- Web Scraping: Used
requests and BeautifulSoup to scrape Excel data from the AIHW site, filtering out outdated or broken links.
- Data Transformation: Parsed and standardised health indicators from Excel sheets using
pandas, handling non-tabular formats and inconsistent headers.
- Data Cleaning: Merged time-series datasets across various categories (hospital, mental health, chronic disease), cleaned nulls, and renamed confusing column labels.
- Visualisation: Created a Streamlit app with dynamic dropdowns, trend charts (line/bar), and data export capabilities. Implemented responsive layout with
st.columns().
💡 Features
- Automatic scraping of latest health reports from AIHW
- Consolidation of multiple Excel sources into unified tables
- Interactive data exploration by year, topic, or region
- Built-in download option for cleaned data
📉 Errors & Fixes
- Encountered inconsistent Excel layouts – fixed by detecting header rows dynamically with
openpyxl fallbacks.
- Streamlit caching bugs with file download – resolved using
@st.cache_data and unique filenames for downloads.
- Broken URLs due to website update – added regex validation and fallback logic to skip invalid links.
📊 Key Takeaways
- Automating data sourcing can drastically speed up reporting workflows.
- Handling real-world healthcare data requires flexible and robust preprocessing logic.
- Streamlit enables rapid visual insight development without frontend overhead.
🔗 Repository
GitHub Repo: Healthcare AIHW ETL
← Back to Blog