django - Twisted connections timeout under heavy load -

-

we have django web app serves moderate number of users, running on ubuntu machine 8 cores , @ least 32gb ram. have no problems users connecting via browser. however, on backend (on same server) running twisted server. django webapp tries connect our twisted server, after 1100-1200 such connections (including bunch of persistent connections other devices on backend), connections start timeout. our twisted server worked fine under low load server seems unable handle new connections django. connections time out. not see wrong our code (which have been working on couple of years should pretty stable). have set our soft , hard ulimits in /etc/security/limits.conf 50000/65000 , have upped somaxconn 65536. print of limits our twisted process listed below. total number of files across tope 25 processes on 5000. unfortunately still cannot more 1100-1200 simultaneous connections our twisted server. things should @ make our twisted connections start connecting again? there other sysctl or other ubuntu linux parameters need change? there twisted parameters need change?

limit                     soft limit           hard limit           units max cpu time              unlimited            unlimited            seconds max file size             unlimited            unlimited            bytes max data size             unlimited            unlimited            bytes max stack size            8388608              unlimited            bytes max core file size        0                    unlimited            bytes max resident set          unlimited            unlimited            bytes max processes             465901               465901               processes max open files            50000                65000                files max locked memory         65536                65536                bytes max address space         unlimited            unlimited            bytes max file locks            unlimited            unlimited            locks max pending signals       465901               465901               signals max msgqueue size         819200               819200               bytes max nice priority         0                    0 max realtime priority     0                    0 max realtime timeout      unlimited            unlimited

twisted thin shell around application. when there performance problem, always, problem somewhere inside application , not in twisted. there no general answer question.

that said, there couple of investigation techniques use. twisted process consuming 100% cpu? if so, going need split multiple processes somehow (using spawnprocess, sendfiledescriptor , adoptstreamport allow i/o done in subprocesses). if not, problem inadvertent blocking i/o preventing reactor servicing requests: might use twisted_hang diagnose hot-spots reactor getting "stuck".

there's possibility problem on django's side of connection. however, no information how django making connections, there's little more can guess.


Comments

Post a Comment

Popular posts from this blog

Fail to load namespace Spring Security http://www.springframework.org/security/tags -

-
i have problem namespace of spring security on xhtml page. don't understand why can't load uri, , have following issue: whitelabel error page application has no explicit mapping /error, seeing fallback. tue jul 21 16:58:12 utc 2015 there unexpected error (type=not found, status=404). please need help! possible uri has been changed? best regards you using incorrect namespace url. correct url http://www.springframework.org/schema/security the error page seeing spring source website since not recognize url. try visiting http://www.springsource.org/security/tags in browser update : spring security tags used in jsp page correct tag lib declaration is: <%@ taglib prefix="sec" uri="http://www.springframework.org/security/tags" %>
Read more

sql - MySQL query optimization using coalesce -

-
can explain me why second query faster first? (first 1 takes around 40 seconds, second 1 less 0.1) select releases.feature_code, releases.platform_id, version, release_date, general_available, uri, filesize, display_name, feature_name.full_name, if ((select exists(select 1 release_exceptions releases.feature_code = release_exceptions.feature_code , releases.platform_id = release_exceptions.platform_id , releases.version = release_exceptions.version) = 1), 1, 0) release_exceptions releases left outer join platform using(platform_id) left outer join feature_name using(feature_code) order version desc; ~~~ query 2 ------------------ ~~~ select feature_code, version, release_date, general_available, platform_id, uri, filesize, display_name, feature_name.full_name, coalesce(release_exceptions, 0) release_exceptions mus.releases left outer join ( select distinct feature_code, version, platform_id, 1 release_exceptions mus.release_exceptions ) release_exceptions using ...
Read more

unity3d - Unity local avoidance in user created world -

-
i'm using unity3d networked multiplayer online game have large complex 3d terrain scene forest, trees, cliff, hills, mountains, bounders, etc. players can build structures sort of minecraft, , put them anywhere in scene, or move them around anytime. aside human controlled players, there automated ai players , objects animals roaming around scene following path. the problem how make these automated ai players , animals, able navigate around static , dynamic player created structures, because path follow can blocked player created structures, or other players , other ai objects, cliffs etc. have find away around them or on track if tumble down off high cliff example. so made navmesh , used navagents, takes care of static, non moving objects, how make ai players navigate around each other , dynamic structures created players can number in hundreds? i thought of adding navmeshobstacle everything, result in being attached hundreds of objects since user created structures b...
Read more