LayoutTileSource.requireGridAligned is failing with GDALRasterSource #3292

Activity

1. pomadchin

   added

   bug

   on Sep 9, 2020
2. zacdezgeo commented on May 13, 2021

   

   zacdezgeo

   on May 13, 2021

   Contributor

   More actions

   Hi @pomadchin,

   I have been encountering this error inconsistently depending on the input image as well. I was able to find the definition of the requireGridAligned method. There is a TODO comment that confused me a bit. It says that the following code is ignored at the moment, but it is not. Should the comment read it is not ignored at the moment? There is also a mention of making the require condition softer.

   LayoutTileSourceScala

   /**
         * TODO: This is ignored at the moment to make it soft and to make GDAL work,
         * we need to reconsider these things to be softer (?)
         */
   
       require(a.extent.xmin === resultX +- epsX,
         s"x-aligned: offset by ${a.cellSize} ${offset(a.extent.xmin, resultX, a.cellwidth)}")
   
       require(a.extent.ymin === resultY +- epsY,
         s"y-aligned: offset by ${a.cellSize} ${offset(a.extent.ymin, resultY, a.cellheight)}")

   Are there any related issues around the topic and decision of the condition given? On my end, I am capable of reading the images with GDAL without using geotrellis even when the require condition does not pass within Geotrellis. I would expect to be able to ingest the geotiff with Geotrellis if GDAL is capable.

   A little background on the use case. I am ingesting geotiff and creating Tiles for different zoom levels. I am trying to define potential workarounds. Some options I have considered but am not completely sure that they would actually change anything:

   • Redefine the layout scheme used to indirectly meet the requireGridAligned condition
   • Pass an overview strategy different from the generic default one to the RasterSourceRDD
   • Modify the requireGridAligned method.

   I will be working on sorting out the data of images that meet the requireGridAligned from those that do not.

   Here is a snippet of the code for slicing an image into Tiles:

   def run(layerName: String)(implicit sc: SparkContext): (Int, Int) = {
   
       // TODO In a future version, base Zoom level should be computed based on pixel size.
       // This is not required to go in production
       val baseZoomLevel: Int = 21
       val path = s"s3://${Configurations.S3.bucket}/$geoimageryFileName"
       val source: GDALRasterSource = GDALRasterSource(path)
   
       // validation happens regardless
       validateCrs(source, path)
   
       val floatingLayoutScheme: FloatingLayoutScheme = FloatingLayoutScheme(
         Configurations.Geotrellis.tileWidth
       )
       val numberOfBands: Int = source.bandCount
       val layout: LayoutDefinition =
         floatingLayoutScheme.levelFor(source.extent, source.cellSize).layout
   
       val rasterSourceRDD = RasterSourceRDD.spatial(source, layout)
   
       val s3AttributeStore: S3AttributeStore = utils.ConfiguredS3AttributeStore.originalTileset()
       val s3LayerWriter = new S3LayerWriter(
         s3AttributeStore,
         Configurations.S3.bucket,
         Configurations.Geotrellis.originalTilesetCatalog
       )
       val layerId = LayerId(layerName, baseZoomLevel)
       s3LayerWriter.write(layerId, rasterSourceRDD, ZCurveKeyIndexMethod) // TODO: call to requireGridAligned happens here
       (numberOfBands, baseZoomLevel)
     }
   

   Would love to get your insights on a potential resolution!
3. pomadchin commented on May 13, 2021

   

   pomadchin

   on May 13, 2021

   Member

   More actions

   Hey @zacharyDez, that is a comment that reminds us (me in particular) that there are some differences in math between GeoTiffRasterSource and GDALRasterSource operations (reprojection and resample).

   The issue here is a bit wider than it is written in a comment. Just relaxing this required condition may make the code pass but it still may fail somewhere else.

   Just to clarify: to write a layer you'd need to tileToLayout (which is resampleToGrid) an RDD of RasterSources. That's where this issue happens.

   I don't have any insights on a potential solution, this can be not a trivial issue.
4. jdenisgiguere commented on May 13, 2021

   

   jdenisgiguere

   on May 13, 2021

   Last edited by jdenisgiguere

   Author

   More actions

   @pomadchin, we are creating a specific Layout for a given GDALRasterSource.

   I'm a bit surprised that the resampleToGrid returns too big offset as they should be perfectly aligned. Shouldn't be a GeoTiffRasterSource GDALRasterSource be perfectly aligned with a Layout that is based on it?
5. pomadchin commented on May 13, 2021

   

   pomadchin

   on May 13, 2021

   Last edited by pomadchin

   Member

   More actions

   @jdenisgiguere GeoTiffRasterSource should work fine, or what is your question?
6. pomadchin commented on May 13, 2021

   

   pomadchin

   on May 13, 2021

   Member

   More actions

   @jdenisgiguere I'm sorry, I don't have any guessses right now, it should be but there are definitely some differences with how the GeoTiffRasterSource behaves. That can be related to some heuristics that are used by GDAL, to math differences or to some a bit incorrect parameters passed into the underlying GDALWarp.