foreach - Parallel processing MODIS images in R -

-

i have downloaded vegetation data modis africa , trying yo create mosaics images , save them geo tif. have no problem doing until try run in parallels on macpro 6cores, 12 threads. code runs uses 1% of cores availability , takes ever complete processes.

i need help, have more 70gb of modis granules need convert mosaic , if don't use computer power got in lab take ever.

here code part of parallel processing:

there 2 parts run in parallel, 1. select ndvi; 2. create mosaics, project them , save them .tif.

# register cluster 10 cores  registerdoparallel(cores=11) micluster<-makecluster(11) registerdoparallel(micluster) library(domc) registerdomc(11)  # 1. # select ndvi each africa granule , put in sdslist sdslist <- foreach(k = 1:length(dategranules), .packages=c("raster", "gdalutils","foreach"))%dopar%{   for(j in 1:1:length(dategranules)){     return(sapply(x=dategranules[[j]], fun=function(x){get_subdatasets(x)[1]}))   } }  2. # generate mosaic africa ndvi aoutput foreach(j = 1:length(sdslist), .packages=c("raster", "gdalutils", "foreach"))%dopar%{     gdalwarp(srcfile=sdslist[[j]], t_srs="+proj=longlat +datum=wgs84   +no_defs",       dstfile=file.path(dest, names[j])) }

why use r @ all, when can in 2 lines, shell, multi threading , faster using gdal utilities?

if understand question , code correctly want extract ndvi subdataset from, presumably mod13q1, hdf5 containers, reproject lat/lon, change filetype geotiff , mosaic granules covering africa 1 large mosaic.

  1. build virtual raster table containing ndvi subdatasets.

gdalbuildvrt -sd 1 ndvi.vrt mod13q1.*.hdf

note use of -sd 1 takes first subdataset every hdf container.

  1. reproject , change format.

gdalwarp gdalwarp -t_srs "+proj=longlat +datum=wgs84 +no_defs" -multi ndvi.vrt ndvi-mosaic.tif

step 1 takes virtually no time since creates metadata file containing information hierarchy of hdf files , no actual processing done. if granules adjacent, e.g. h15v05, h15v04, h14v05,..., mosaiced correctly geolocation.

step 2 actual reprojection , file format change. tested on core2duo (3ghz) , reprojection of 20 modis granules took 1.5 minutes. without multi-threading should lot faster in r.


Comments

Post a Comment

Popular posts from this blog

Fail to load namespace Spring Security http://www.springframework.org/security/tags -

-
i have problem namespace of spring security on xhtml page. don't understand why can't load uri, , have following issue: whitelabel error page application has no explicit mapping /error, seeing fallback. tue jul 21 16:58:12 utc 2015 there unexpected error (type=not found, status=404). please need help! possible uri has been changed? best regards you using incorrect namespace url. correct url http://www.springframework.org/schema/security the error page seeing spring source website since not recognize url. try visiting http://www.springsource.org/security/tags in browser update : spring security tags used in jsp page correct tag lib declaration is: <%@ taglib prefix="sec" uri="http://www.springframework.org/security/tags" %>
Read more

sql - MySQL query optimization using coalesce -

-
can explain me why second query faster first? (first 1 takes around 40 seconds, second 1 less 0.1) select releases.feature_code, releases.platform_id, version, release_date, general_available, uri, filesize, display_name, feature_name.full_name, if ((select exists(select 1 release_exceptions releases.feature_code = release_exceptions.feature_code , releases.platform_id = release_exceptions.platform_id , releases.version = release_exceptions.version) = 1), 1, 0) release_exceptions releases left outer join platform using(platform_id) left outer join feature_name using(feature_code) order version desc; ~~~ query 2 ------------------ ~~~ select feature_code, version, release_date, general_available, platform_id, uri, filesize, display_name, feature_name.full_name, coalesce(release_exceptions, 0) release_exceptions mus.releases left outer join ( select distinct feature_code, version, platform_id, 1 release_exceptions mus.release_exceptions ) release_exceptions using ...
Read more

unity3d - Unity local avoidance in user created world -

-
i'm using unity3d networked multiplayer online game have large complex 3d terrain scene forest, trees, cliff, hills, mountains, bounders, etc. players can build structures sort of minecraft, , put them anywhere in scene, or move them around anytime. aside human controlled players, there automated ai players , objects animals roaming around scene following path. the problem how make these automated ai players , animals, able navigate around static , dynamic player created structures, because path follow can blocked player created structures, or other players , other ai objects, cliffs etc. have find away around them or on track if tumble down off high cliff example. so made navmesh , used navagents, takes care of static, non moving objects, how make ai players navigate around each other , dynamic structures created players can number in hundreds? i thought of adding navmeshobstacle everything, result in being attached hundreds of objects since user created structures b...
Read more