XLDB_BigEarthData
展开查看详情
1.Big Earth Data today: Challenges, Approaches, and Standards XLDB-Europe, Clermont-Ferrand, 2017-oct-11 Peter Baumann Jacobs University | rasdaman GmbH [gamingfeeds.com] Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
2.Datacube Research @ Jacobs U Large-Scale Scientific Information Systems research group • focus: large-scale n-D array services & beyond • www.jacobs-university.de/lsis Spin-off: rasdaman GmbH Main visible results: • pioneer Array DBMS, rasdaman • Editor of OGC & ISO „datacube“ standards Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
3.Overview Introduction Coverage data & service model Implementing Coverages Through Array Databases Coverages in Practice: EarthServer Summary Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann 3
4.Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
5. Big Science Data [OGC Ocean Science Interoperability Experiment; image source: Mbari] Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
6.Data Integration & Consumption sensor feeds Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann 6
7.Coverages & WCS in Orchestrations HPC atmospheric cascading data WCS WCS Sentinel cascading EO data WCS WCS SWE in-situ data WCS Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
8.Analysis-Ready Data Before: 10x EO „flat“ scenes Now: few space/time datacubes Users responsible for „Big Picture“ Data providers responsible for harmonization to „Big Picture“ [EarthServer] [EOX] Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
9.Overview Introduction Coverage data & service model Implementing Coverages Through Array Databases Coverages in Practice: EarthServer Summary Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann 9
10.Features & Coverages Open Geospatial Consortium (OGC) - Collaborating with ISO, OASIS, OMG, W3C, ... The basis of all: geographic feature Special kind of feature: coverage - aka space-time varying phenomenon - regular & irregular grids, point clouds, meshes • Usually, coverages are Big Geo Data Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
11.Coverages - abstract: OGC Abstract Topic 6 = ISO 19123 - concrete, interoperable: Coverage Implementation Schema ISO 19123-2 • (CIS, aka GMLCOV) «FeatureType» Abstract Coverage MultiSolid Grid Coverage Coverage CIS 1.1 Rectified MultiSurface Coverage General Grid Coverage GridCoverage MultiCurve Referenceable Coverage GridCoverage MultiPoint Coverage Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
12. Coverage Structure «Feature Type» GML::Feature «Feature Type» Coverage 0..1 metadata domainSet rangeType rangeSet «Data Type» «Data Type» «Data Type» DomainSet SWE Common::DataRecord RangeSet [OGC 09-146r6] Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
13.Grid Types Axis Types CIS 1.0 (from GML): coverage classification by grid type - GridCoverage, RectifiedGridCoverage, ReferenceableGridCoverage - non-intuitive, hard to describe, missing cases CIS 1.1 approach: axis types - Index axis: not georeferenced, Index CRS - Regular axis: georeferenced, constant spacing - Irregular axis: georeferenced, variable spacing - Displacement axis nest: georeferenced, arbitrary grid point locations - Algorithmic coordinates: such as sensor model - All combinations possible Integrating GML 3.3 and SensorML grids Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
14.A Simple Coverage, in GML Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
15.A Simple Coverage, in JSON { "type": "CoverageByDomainAndRangeType", "domainSet":{ "type": "DomainSetType", "generalGrid":{ "type": "GeneralGridCoverageType", "srsName": "http://www.opengis.net/def/crs/OGC/0/Index2D", "axisLabels": ["i", "j"], "axis": [{ "type": "IndexAxisType“, "axisLabel": "i", "lowerBound": 0, "upperBound": 2 },{ "type": "IndexAxisType“, "axisLabel": "j", "lowerBound": 0, "upperBound": 2 }] } }, "rangeSet": { "type": "RangeSetType", "dataBlock": { "type": "VDataBlockType", "values": [1,2,3,4,5,6,7,8,9] } }, "rangeType": { "type": "DataRecordType", "field":[{ "type": "QuantityType", "definition": "ogcType:unsignedInt", "uom": { "type": "UnitReference", "code": "10^0“ } }] } } Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
16.A Simple Coverage, in RDF <http://www.opengis.net/cis/1.1/examples/CIS_05_2D> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.opengis.net/cis/1.1/CoverageByDomainAndRangeType> . <http://www.opengis.net/cis/1.1/examples/CIS_05_2D> <http://www.opengis.net/cis/1.1/domainSet> <http://www.opengis.net/cis/1.1/examples/CIS_DS_05_2D> . <http://www.opengis.net/cis/1.1/examples/CIS_DS_05_2D> <http://www.opengis.net/cis/1.1/generalGrid> <http://www.opengis.net/cis/1.1/examples/CIS_DS_GG_05_2D> . <http://www.opengis.net/cis/1.1/examples/CIS_DS_05_2D> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.opengis.net/cis/1.1/DomainSetType> . <http://www.opengis.net/cis/1.1/examples/CIS_DS_GG_05_2D> <http://www.opengis.net/cis/1.1/axis> <http://www.opengis.net/cis/1.1/examples/CIS_DS_GG_I_05_2D> . <http://www.opengis.net/cis/1.1/examples/CIS_DS_GG_05_2D> <http://www.opengis.net/cis/1.1/axis> <http://www.opengis.net/cis/1.1/examples/CIS_DS_GG_J_05_2D> . <http://www.opengis.net/cis/1.1/examples/CIS_DS_GG_05_2D> <http://www.opengis.net/cis/1.1/axisLabels> <http://www.opengis.net/cis/1.1/axisLabels0> . <http://www.opengis.net/cis/1.1/axisLabels0> <http://www.w3.org/1999/02/22-rdf-syntax-ns#first> "i" . <http://www.opengis.net/cis/1.1/axisLabels0> <http://www.w3.org/1999/02/22-rdf-syntax-ns#rest> <http://www.opengis.net/cis/1.1/axisLa <http://www.opengis.net/cis/1.1/axisLabels1> <http://www.w3.org/1999/02/22-rdf-syntax-ns#first> "j" . <http://www.opengis.net/cis/1.1/axisLabels1> <http://www.w3.org/1999/02/22-rdf-syntax-ns#rest> <http://www.w3.org/1999/02/22-rdf-syn Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
17.Encoding Coverages Coverage Domain set Range type Single file encoding: Range set - Informationally complete: GML, JSON, RDF, … App Metadata - Further formats: GeoTIFF, NetCDF, JPEG2000, GRIB, … Multipart: container( “header” + file1 + file2 + … ) Coverage - Multipart/MIME, zip, GMLJP2, SAFE, GeoPackage, ... Domain set - Collections, tiling, streaming Range type xlink App Metadata NetCDF Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
18.OGC Web Coverage Service (WCS) WCS Core: access to spatio-temporal coverages & subsets - Encoding on the fly - subset = trim | slice Large, growing implementation basis: rasdaman, GDAL, QGIS, OpenLayers, OPeNDAP, MapServer, GeoServer, GMU, NASA WorldWind, EOx- Server; Pyxis, ERDAS, ArcGIS, ... WCS Extensions: optional functionality facets - Extraction, transformation, …up to flexible analytics (WCPS) WCS Application Profiles: domain-oriented bundling Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
19.Server-Side Processing: Performance xxx Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
20.WCPS: Datacube Analytics Web Coverage Processing Service = spatio-temporal datacube analytics language • "From MODIS scenes M1, M2, M3: difference red & nir, as TIFF" • “…but only those where nir exceeds 127 somewhere” for $c in ( M1, M2, M3 ) where some( $c.nir > 127 ) return encode( $c.red - $c.nir, “image/tiff“ ) • s/t regular & irregular grids Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
21. Elevation for $s in (SatImage), $d in (DEM) where $s/metadata/@region = “Glasgow" return encode( struct { red: (char) $s.b7[x0:x1,x0:x1], green: (char) $s.b5[x0:x1,x0:x1], blue: (char) $s.b0[x0:x1,x0:x1], alpha: (char) scale( $d, 20 ) }, “image/png" ) [JacobsU, Fraunhofer; data courtesy BGS, ESA] Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
22.WMS via WCPS for $p in (OrthoPhoto), $wl in (WaterLines), $wa in (WaterAreas), $d in (DEM) return encode( (unsigned char) ( $p * { 1, 1, 1 } overlay $wl * { 0, 128, 255 } overlay $wa * { 191, 255, 255 } overlay switch $d case $d > 260 return { red:255, green:0, blue:0 } case $d > 262 return { red:0, green:255, blue:0 } case $d > 264 return { red:0, green:0, blue:255 } default return { red:0, green:0, blue:0 } end ), "image/png" ) Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
23.Outlook: WCPS 2.0 merge WCPS with XQuery FLWOR let $r := doc(“WCPS”) //coverage/metadata/region/[ @name = "Austria" ] for $c in doc(“WCPS”) //coverage/[ some( $c.nir > 127 and $r ) ] ) return encode( abs( $c.red - $c.nir ), "image/tiff" ) for $c in doc(“WCPS”)//coverage/[ some( $c.nir > $c.red )] return <id> { $c/@id } </id> <area> { $c/boundedBy } </area> Athena Research Center - federation of eXist + rasdaman Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
24.Outlook: n-D Polygon/Raster Clipping Problem: given a polygon in space/time, find all cells inside - Special cases: curtains, corridors Use case: air traffic weather information Original idea by UK MetOffice [ M. Dixson & UK MetOffice] Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
25....But That‘s Not What You Want to See Users should stay in their comfort zone - navigation: Web client; ...; analytics:python, R, ... >>> con = Connection(hostname="127.0.0.1" , port=7001) >>> mr = RasCollection(con, "mr") >>> mr = mr[100,150] # Array Subsetting >>> mr += 1 >>> mr = mr ** 2 # Square of all elements >>> mr = mr. filter (oid=2) >>> mr.query <RasQueryObject> >>> str (mr.query) "Select exp(mr[100,150]+1,2) from mr where oid(mr) = 2" >>> arr = mr.eval () <RasArrayObject> >>> arr.to_array () # Default conversion : Numpy Array [[...],[...],[...]] Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
26.Overview Introduction Coverage data & service model Implementing Coverages Through Array Databases Coverages in Practice: EarthServer Summary Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann 27
27.rasdaman: Agile Array Analytics „raster data manager“: n-D arrays in SQL - Array Algebra [NGITS 1998] + declarative, optimizable QL - [VLDB 1994, VLDB 1997, SIGMOD 1998, VLDB 2003, … VLDB 2016] - Scalable, parallel architecture: „tile streaming“ Implemented & in use - 1+ PB databases Blueprint for OGC WCPS & ISO SQL/MDA Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
28. ISO Array SQL [SSDBM 2014] create table LandsatScenes( id: integer not null, acquired: date, scene: row( band1: integer, ..., band7: integer ) mdarray [ 0:4999,0:4999] ) select id, encode((scene.band1-scene.band2)/(scene.band1+scene.band2)), „image/tiff“ ) from LandsatScenes where acquired between „1990-06-01“ and „1990-06-30“ and avg( scene.band3-scene.band4)/(scene.band3+scene.band4)) > 0 modulo keyword choice Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
29.Some Simple Linear Algebra Ops Matrix multiplication select marray i in [0:m], j in [0:p] values condense + over k in [0:n] using a [ i, k ] * b [ k, j ] from matrix as a, matrix as b Histogram select marray bucket in [0:255] values count_cells( img = bucket ) from img Big Earth Data :: XLDB-Europe :: ©2017 P. Baumann
