optimization - Compute new column based on values in current and following rows with dplyr in R -

-

i have big dataset (10+ mil x 30 vars) , trying compute new variables based on complicated interactions of current ones. clarity including important variables in question. have following code in r interested in other views , opinions. using dplyr package compute new columns based on current/following row values of 3 other columns. (more explanation below code)

i wondering if there way make faster , more efficient, or maybe rewrite it...

# main function-data dataframe, windowsize , ratio ints computenewcolumn <- function(data,windowsize,ratio){       #helper function used in second mutate down...      # args ints, return boolean out       windowahead <- function(timeto,window,reduction){       # subset original dataframe-only observations values of      # timetogo between timeto-1 , window (basically following x rows       # current one)      subframe <- data[(timeto-1 >= data$timetogo & data$timetogo >= window), ]      isthere <- any(subframe$price < reduction)      return(isthere)      }    # group value of id first , order timetogo...     data %<>% group_by(id) %>%    arrange(desc(timetogo)) %>%    # ...create 2 new columns simple interactions of existing ones...   mutate(window = ifelse(timetogo > windowsize, timetogo - windowsize, 0),          reduction = floor(price - (ratio * price))) %>%    rowwise() %>%    #...now comes more complex stuff- want compute third column    # depending on next (timetogo - window) number of values of price   mutate(advice = ifelse(windowahead(timetogo,window,reduction),1,0) )   return(data) }

we have dataset following columns: id,price, timetogo.

we first group values of id , compute 2 new columns based on current row values (window timetogo , reduction price). next thing compute new third column based on

1.current value of reduction

2.the next (window - timetogo) amount of values of price in dataframe.

i wondering if there simple way reference upcoming values of column within mutate()? ideally looking sliding window function on 1 column, limits of sliding window set 2 other current column values. solution uses custom function subsets on original dataframe manually, comparison , returns value mutate() call. , ideas appreciated!

p.s. heres sample of data... please let me know if need more info. thanks!

>            id timetogo price 1 aqsafoto30a       96    19 2 aqsafoto20a       95    19 3 aqsafoto30a       94    17 4 aqsafoto20a       93    18 5 aqsafoto25a       92    19 6 aqsafoto30a       91    17


Comments

Post a Comment

Popular posts from this blog

Fail to load namespace Spring Security http://www.springframework.org/security/tags -

-
i have problem namespace of spring security on xhtml page. don't understand why can't load uri, , have following issue: whitelabel error page application has no explicit mapping /error, seeing fallback. tue jul 21 16:58:12 utc 2015 there unexpected error (type=not found, status=404). please need help! possible uri has been changed? best regards you using incorrect namespace url. correct url http://www.springframework.org/schema/security the error page seeing spring source website since not recognize url. try visiting http://www.springsource.org/security/tags in browser update : spring security tags used in jsp page correct tag lib declaration is: <%@ taglib prefix="sec" uri="http://www.springframework.org/security/tags" %>
Read more

sql - MySQL query optimization using coalesce -

-
can explain me why second query faster first? (first 1 takes around 40 seconds, second 1 less 0.1) select releases.feature_code, releases.platform_id, version, release_date, general_available, uri, filesize, display_name, feature_name.full_name, if ((select exists(select 1 release_exceptions releases.feature_code = release_exceptions.feature_code , releases.platform_id = release_exceptions.platform_id , releases.version = release_exceptions.version) = 1), 1, 0) release_exceptions releases left outer join platform using(platform_id) left outer join feature_name using(feature_code) order version desc; ~~~ query 2 ------------------ ~~~ select feature_code, version, release_date, general_available, platform_id, uri, filesize, display_name, feature_name.full_name, coalesce(release_exceptions, 0) release_exceptions mus.releases left outer join ( select distinct feature_code, version, platform_id, 1 release_exceptions mus.release_exceptions ) release_exceptions using ...
Read more

unity3d - Unity local avoidance in user created world -

-
i'm using unity3d networked multiplayer online game have large complex 3d terrain scene forest, trees, cliff, hills, mountains, bounders, etc. players can build structures sort of minecraft, , put them anywhere in scene, or move them around anytime. aside human controlled players, there automated ai players , objects animals roaming around scene following path. the problem how make these automated ai players , animals, able navigate around static , dynamic player created structures, because path follow can blocked player created structures, or other players , other ai objects, cliffs etc. have find away around them or on track if tumble down off high cliff example. so made navmesh , used navagents, takes care of static, non moving objects, how make ai players navigate around each other , dynamic structures created players can number in hundreds? i thought of adding navmeshobstacle everything, result in being attached hundreds of objects since user created structures b...
Read more