[{"title":"Engineering Resilient Data Pipelines - Lessons from Healthcare ETL","slug":"resilient-data-pipelines","excerpt":"Healthcare data pipelines can't afford to fail. When patient care and regulatory compliance depend on data availability, resilience isn't optional - it's fundamental. We share practical patterns and hard-won lessons from building a fault-tolerant ETL system that processes critical healthcare data from multiple sources while maintaining 99.9% uptime.","coverImageThumb":"/images/projects/resilient-pipelines-thumb.png","date":"2025-01-21 12:00 +0000","categories":["data-engineering","reliability","docker","healthcare","etl","dagster","data-pipeline","python"],"path":"/src/content/posts/resilient-data-pipelines.md","weight":100},{"title":"Privacy-First Development - Using Synthetic Data in Healthcare ETL Testing","slug":"privacy-first-etl-testing","excerpt":"Testing healthcare ETL pipelines presents a fundamental dilemma: how do you ensure your data transformations work correctly without exposing sensitive patient information? With GDPR fines reaching €20 million and patient trust at stake, we developed a synthetic data generation approach that enables comprehensive testing while maintaining complete isolation from real patient data.","coverImageThumb":"/images/projects/privacy-etl-thumb.png","date":"2025-01-21 11:00 +0000","categories":["data-privacy","privacy","gdpr","testing","healthcare","etl","dagster","data-pipeline","python"],"path":"/src/content/posts/privacy-first-etl-testing.md","weight":100},{"title":"Building Modern ETL Pipelines with Dagster - An Asset-Centric Approach","slug":"modern-etl-dagster","excerpt":"Healthcare data processing presents unique challenges - multiple data sources, stringent compliance requirements, and the critical nature of accuracy. We recently tackled these challenges by building a modern ETL pipeline using Dagster's asset-centric approach, moving beyond traditional task-based orchestration to create a more maintainable and observable data platform.","coverImageThumb":"/images/projects/dagster-etl-thumb.png","date":"2025-01-21 10:00 +0000","categories":["data-engineering","dagster","etl","healthcare","data-pipeline","python"],"path":"/src/content/posts/modern-etl-dagster.md","weight":100},{"title":"A WebGL 3D, VR/AR Data-cluster component","slug":"data-clusters-titanic","excerpt":"This is the first prototype of a 3D dataviz component that allows dataset (rows and columns) to be displayed and explored with seamless visual transitions through various presentations defined by the data field types.","coverImageThumb":"/images/projects/data-clusters-titanic-xyz.png","date":"2024-09-23 18:45 +0200","categories":["svelte","charts","dataviz","layercake","d3","WebGL","threejs","aframe","3D"],"path":"/src/content/posts/data-clusters-titanic.svx","weight":96},{"title":"Visualizing an Election","slug":"data-clusters-ukelection","excerpt":"We extend our 3D (VR) dataviz component to visualize an election datasets, showing a geographic hex-map for the UK constituencies and a radial chart showing the party sizes.","coverImageThumb":"/images/projects/data-clusters-ukelection.png","date":"2024-09-23 18:45 +0200","categories":["svelte","charts","dataviz","layercake","d3","WebGL","threejs","aframe","3D"],"path":"/src/content/posts/data-clusters-ukelection.svx","weight":96},{"title":"Visualizing the UK 2024 Election","slug":"data-clusters-ukelections","excerpt":"We used our 3D (VR) dataviz component to visualize the UK 2024 election, comparing it to the 2019 election, which shows a dramatic change of fortune for the two main UK parties.","coverImageThumb":"/images/projects/data-clusters-ukelections.png","date":"2024-09-23 18:45 +0200","categories":["svelte","charts","dataviz","layercake","d3","WebGL","threejs","aframe","3D"],"path":"/src/content/posts/data-clusters-ukelections.svx","weight":96},{"title":"Svelte QMUL Project Management App","slug":"svelte-qmul","excerpt":"We used D3 and Svelte (with Layercake) to build an in-house project management app for Queen Mary's College London (QMUL). Using a pseudo-physics force-directed network allows one to find correlations in the project dataset.","coverImageThumb":"/images/projects/qmul-thumb.png","date":"2024-04-29 19:47 +0100","categories":["svelte","charts","dataviz","layercake","d3"],"path":"/src/content/posts/svelte-qmul.svx","weight":97},{"title":"Line to Stacked Area Chart with Svelte Transitions","slug":"line-to-stacked-area-chart-with-svelte-transitions","excerpt":"This chart transitions from a conventional multi-line chart to a stacked area chart, using Svelte transitions. LayerCake and D3 add ballast to the foundations. The y axis can be switched from fixed to dynamic.","coverImageThumb":"/images/projects/line-area-chart-thumb.png","date":"2022-12-17 19:55 +0000","categories":["svelte","charts","dataviz","layercake","d3"],"path":"/src/content/posts/line-to-stacked-area-chart-with-svelte-transitions.svx","weight":96},{"title":"Circular Bar Chart Component with Svelte, LayerCake and D3","slug":"circular-bar-chart-component-with-svelte-layercake-and-d3","excerpt":"This is a little demo of a circular bar chart, built using Svelte, LayerCake and a little D3.","coverImageThumb":"/images/projects/circular-bar-chart-thumb.png","date":"2022-11-28 18:45 +0200","categories":["svelte","charts","dataviz","layercake","d3"],"path":"/src/content/posts/circular-bar-chart-component-with-svelte-layercake-and-d3.svx","weight":96},{"title":"A Binary Tree with Svelte, D3 and Layercake","slug":"svelte-layercake-binary-tree","excerpt":"A client wanted a recreation of the rather superb binary tree demo by R2D3. Without appropriating the specific aesthetics but building a configurable component that could be used to demonstrate binary trees in action and teach machine-learning.","coverImageThumb":"/images/projects/binary_tree-thumb.png","date":"2022-06-27 19:46 +0100","categories":["svelte","charts","dataviz","layercake","d3"],"path":"/src/content/posts/svelte-layercake-binary-tree.svx","weight":96},{"title":"Svelte Scatterplot Component with Gapminder-esque Demo","slug":"svelte-gapminder-demo","excerpt":"We used Svelte (with Layercake) to build a little scatterplot component with selectable axes and toggled log-plots. This makes for an easy Gapminder-alike demo with the classic income/population/life-expectancy dataset for countries.","coverImageThumb":"/images/projects/gapminder-thumb.png","date":"2022-06-18 19:47 +0100","categories":["gapminder","svelte","charts","dataviz","layercake"],"path":"/src/content/posts/svelte-gapminder-demo.svx","weight":97},{"title":"Svelte Pollen Sketches","slug":"svelte-pollen-sketches","excerpt":"We used Svelte (with Layercake) to build a minimal app as a prelimary for an in-house tool for analyzing large pollen-count datasets. With customized dataviz the principal features and correlations of the dataset can be brought into sharp focus.","coverImageThumb":"/images/projects/pollenapp-thumb.png","date":"2022-06-18 19:47 +0100","categories":["svelte","charts","dataviz","layercake"],"path":"/src/content/posts/svelte-pollen-sketches.svx","weight":97},{"title":"Bespoke Charts with Svelte, D3 and LayerCake","slug":"bespoke-charts-with-svelte-d3-and-layercake","excerpt":"In order to present their data on discrimination against Asian-Americans in the United States, Dataface required some bespoke interactive charts, based on some attractive inhouse designs. They wanted the charts done with Svelte and suggested using the LayerCake library, which provides a clever Svelte based scaffold for D3 etc. dataviz.","coverImageThumb":"/images/projects/svelte-d3-laaunch-thumb.png","date":"2021-10-13 17:18 +0100","categories":["d3","svelte","dataviz"],"path":"/src/content/posts/bespoke-charts-with-svelte-d3-and-layercake.svx","weight":100},{"title":"All That Jazz, Patterns in the Keys","slug":"d3-jazz","excerpt":"This visualisation uses a force-directed layout, courtesy of D3, to view approximately 1200 classic Jazz standards by key, date and composer. Some interesting patterns emerge.","coverImageThumb":"/images/projects/d3-jazz-thumb.png","date":"2020-05-06T13:49:58.000Z","categories":["d3","javascript","dataviz"],"path":"/src/content/posts/d3-jazz.svx","weight":92},{"title":"UK Demographics Explorer (Verify GOV.UK)","slug":"d3-gov-demographics","excerpt":"A data visualisation for the UK government's Verify project, allowing exploration of Verify''s demographic coverage using a dataset provided by the Office of National Statistics","coverImage":"/images/projects/d3-gov-demographics.png","coverImageThumb":"/images/projects/d3-gov-demographics-thumb.png","date":"2019-01-08T13:49:58.000Z","categories":["d3","javascript","dataviz"],"path":"/src/content/posts/d3-gov-demographics.svx","weight":-1}]

Data Diviners

Build custom data processing pipelines to extract, transform and load (ETL) your data. Visualize that data with custom data-visualizations, built from the ground up to bring the stories in your data to life.