Big Data Visualization using DeckGL
“deck.gl is a WebGL-powered framework for visual exploratory data analysis of large datasets.”
In this tutorial, we will learn how to visualize large geospatial datasets using deck.gl through an example. We will take uk accidents data sets and will visualize it using deck.gl.
In this tutorial, we have considered the UK accidents dataset from 2005 to 2104 which has around 1.5 Million points of occurred accidents for that period. This dataset have location coordinates that will be used in deck.gl for visualization.
Deck.gl is ReactJS friendly. It can handle efficient WebGL rendering under heavy data load.
To get full source code for this tutorial, click here.
To check the live demo for this tutorial, click here.
We will divide this tutorial in three parts:
- Preparing the UK accidents dataset.
- Setting ReactJS project.
- Visualizing data points using deck.gl.
1. Preparing the UK accidents dataset
We will use python to validate and clean up the UK accidents data. We will only validate location coordinates. We will remove invalid coordinates from dataset. The reason for validating location coordinates is because invalid coordinates may cause problem during data visualization in deck.gl.
The following code in python will validate the UK accidents location data and it will save the final validated data in JSON format which will be used in deck.gl for visuaization.
Python related code for data processing is in deckgl_app/data_processing_using_python path in github repo.
- We are using pandas to read UK accidents dataset.
- Then, at line number 13 and 14, we are converting the data type of longitude and latitude into number type.
- Then, at line number 17, we are using geopandas to convert the longitude-latitude columns into geodataframe. Using geopandasframe, we can validate invalid and empty coordinates or geometries.
- At line number 21 and 22, we are removing invalid and empty longitude-latitude coordinates from dataset. This makes dataset cleaner and ready to use.
- And then finally, we are storing the validated the longitude-latitude coordinates in JSON format.
- We will use this JSON file in deck.gl app for visualization. This file has around 1.5 Million points.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
import pandas as pd import geopandas as gpd import json #Read datasets uk_accidents_1 = pd.read_csv('data/uk_accidents/uk_accidents_2005_to_2007.tar.xz', low_memory=False) uk_accidents_2 = pd.read_csv('data/uk_accidents/uk_accidents_2009_to_2011.tar.xz', low_memory=False) uk_accidents_3 = pd.read_csv('data/uk_accidents/uk_accidents_2012_to_2014.tar.xz', low_memory=False) uk_accidents = pd.concat([uk_accidents_1, uk_accidents_2, uk_accidents_3]) # Convert Long Lat into numeric type uk_accidents['Longitude'] = pd.to_numeric(uk_accidents['Longitude']) uk_accidents['Latitude'] = pd.to_numeric(uk_accidents['Latitude']) # Convert Longitude Latitude into Point Geometry uk_accidents = gpd.GeoDataFrame(geometry = gpd.points_from_xy(x=uk_accidents['Longitude'], y=uk_accidents['Latitude'])) uk_accidents = uk_accidents.set_crs('EPSG:4326') # Remove invalid geometries if any uk_accidents = uk_accidents[uk_accidents.is_valid] uk_accidents = uk_accidents[~uk_accidents.is_empty] # Get x, y coordinates uk_accidents['x'] = uk_accidents.geometry.x uk_accidents['y'] = uk_accidents.geometry.y df = uk_accidents[['x', 'y']] #Save data points in json format coords = [] for x, y in zip(df.x, df.y): coords.append({ "COORDINATES": [x, y] }) with open('data/uk_accidents.json', 'w') as filehandle: json.dump(coords, filehandle) |
JSON file we created above looks like below sample:
|
1 2 3 4 5 6 7 8 9 10 |
[ {"COORDINATES": [-0.19117, 51.489096]}, {"COORDINATES": [-0.211708, 51.520075]}, {"COORDINATES": [-0.206458, 51.525301]} . . . . {"COORDINATES": [-0.206458, 51.489096]} ] |
2. Setting ReactJS project
DeckGL is react friendly. “deck.gl APIs are designed to reflect the reactive programming paradigm. Whether using Vanilla JS or the React interface, it can handle efficient WebGL rendering under heavy data load”. Because of that reason, we will set ReactJS project and install required dependencies.
Use the following commands to create ReactJS project and to install required dependencies:
|
1 2 3 4 5 |
(spatial-dev.guru) geoknight@pop-os:~$ npx create-react-app deckgl_app (spatial-dev.guru) geoknight@pop-os:~$ cd deckgl_app (spatial-dev.guru) geoknight@pop-os:~/deckgl_app$ npm install deck.gl (spatial-dev.guru) geoknight@pop-os:~/deckgl_app$ npm install h3-js (spatial-dev.guru) geoknight@pop-os:~/deckgl_app$ npm install react-map-gl |
Our default app structure looks like following:
3. Visualizing data points using deck.gl
Our final step is to visualize the point data we have created in first step of this tutorial using python. Our coordinate data for UK accidents is ready.
Now we will write deck.gl code in ReactJS app we created in step 2 of this tutorial to visualize the data points.
1. First, we will copy JSON file into reactjs app. We will copy this file under app->src->jsons directory with uk_accidents.json name.
2. Then, we will overwrite app.js file with our deckgl code that will visualize the data points on map.
Lets talks about deck.gl specific code that we wrote to visualize large data points.
- First, we are importing required dependencies like React, DeckGL, TileLayer, HexagonLayer etc.
- We are importing accident JSON file as module at line number 8 and giving alias to it as accidentss_jsons.
- We will use React module to create react component.
- We will use DeckGL module to render map on browser.
- We will use TileLayer module to create OpenStreetMap layer and will be displaying it as basemap.
- We will use HexagonLayer module to create 3D hexagon layer which will take JSON file as input and render it on map as 3D hex bars.
How the Visualization is happening:
We are using HexagonLayer module for visualization. This HexagonLayer layer takes accidents JSON file as input. Now we have to define radius value for hex bins. See line number 127, we are defining hex radius of 1000 meter. This means that hex bins of 1000 meter radius will be created on area of interests and their height will be based on number of data points(accidents JSON file) intersecting with each of hex bin. More the height of hex bin, more occurrences of accidents will be there.
This hexagon visualization will give clear indication at which location more accidents have happened based on hex bin height.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 |
// app.js import React from 'react'; import DeckGL from '@deck.gl/react'; import { TileLayer} from '@deck.gl/geo-layers'; import {HexagonLayer} from '@deck.gl/aggregation-layers'; import {BitmapLayer, PathLayer} from '@deck.gl/layers'; import {AmbientLight, PointLight, LightingEffect} from '@deck.gl/core'; import accidents_jsons from "./jsons/uk_accidents.json"; const ambientLight = new AmbientLight({ color: [255, 255, 255], intensity: 1.0 }); const pointLight1 = new PointLight({ color: [255, 255, 255], intensity: 0.8, position: [-0.144528, 49.739968, 80000] }); const pointLight2 = new PointLight({ color: [255, 255, 255], intensity: 0.8, position: [-3.807751, 54.104682, 8000] }); const lightingEffect = new LightingEffect({ambientLight, pointLight1, pointLight2}); // Initial View State const INITIAL_VIEW_STATE = { longitude: -1.415727, latitude: 52.232395, zoom: 6.6, // minZoom: 5, // maxZoom: 15, pitch: 40.5, bearing: -27 }; function getTooltip({object}) { if (!object) { return null; } const lat = object.position[1]; const lng = object.position[0]; const count = object.points.length; return `\ latitude: ${Number.isFinite(lat) ? lat.toFixed(6) : ''} longitude: ${Number.isFinite(lng) ? lng.toFixed(6) : ''} ${count} Accidents`; } // DeckGL react component function App({showBorder = false, onTilesLoad = null}) { const layers = [new TileLayer({ data: [ 'https://a.tile.openstreetmap.org/{z}/{x}/{y}.png', 'https://b.tile.openstreetmap.org/{z}/{x}/{y}.png', 'https://c.tile.openstreetmap.org/{z}/{x}/{y}.png' ], // Since these OSM tiles support HTTP/2, we can make many concurrent requests // and we aren't limited by the browser to a certain number per domain. maxRequests: 20, pickable: true, onViewportLoad: onTilesLoad, autoHighlight: showBorder, highlightColor: [60, 60, 60, 40], minZoom: 0, maxZoom: 19, tileSize: 256, zoomOffset: devicePixelRatio === 1 ? -1 : 0, renderSubLayers: props => { const { bbox: {west, south, east, north} } = props.tile; return [ new BitmapLayer(props, { data: null, image: props.data, bounds: [west, south, east, north] }), showBorder && new PathLayer({ id: `${props.id}-border`, visible: props.visible, data: [ [ [west, north], [west, south], [east, south], [east, north], [west, north] ] ], getPath: d => d, getColor: [255, 0, 0], widthMinPixels: 4 }) ]; } }), new HexagonLayer({ id: 'HexagonLayer', data: accidents_jsons, "coverage": 0.8, "pickable": true, "autoHighlight": true, "elevationRange": [ 0, 3000 ], "elevationScale": 50, "extruded": true, "getPosition": d => d.COORDINATES, "radius": 1000, "upperPercentile": 100, "colorRange": [ [1, 152, 189], [73, 227, 206], [216, 254, 181], [254, 237, 177], [254, 173, 84], [209, 55, 78] ], material: { ambient: 0.64, diffuse: 0.6, shininess: 32, specularColor: [51, 51, 51] }, transitions: { elevationScale: 3000 } }) ]; console.log(layers); return <DeckGL effects={[lightingEffect]} mapStyle='https://basemaps.cartocdn.com/gl/dark-matter-nolabels-gl-style/style.json' initialViewState={INITIAL_VIEW_STATE} controller={true} layers={layers} getTooltip={getTooltip} > </DeckGL> ; } export default App; |
Our new project directory structure looks like below:
I hope this tutorial will create a good foundation for you. If you want tutorials on another GIS topic or you have any queries, please send an email at contact@spatial-dev.guru