http status code 404 - 404 after 43 seconds TTFB -

i have script uses simple_html_dom parse different site data. looks through table of users, grabs various sites needed, , parses data , stores them db.

the problem when iterate through more 3 users 404 error. after lot of debugging (much of i'm learning go) looks ttfb hits 40 seconds 404 not found error. under page returns fine.

i included following in php file extend time problem seems ignore these statements.

// may take whils crawl site ...

ini_set("memory_limit", "-1"); ini_set('max_execution_time', 300); //300 seconds = 5 minutes ini_set('max_input_time', -1); //300 seconds = 5 minutes set_time_limit(0); 

but i've never had problem before 404 page exists. i'm new simple_html_dom , crawling through different pages problem wait time long? if how can fix that? thanks

so did not have execution time or setting change php script. having same issue fixed changing way simple_html_dom loads script from:

$html = new simple_html_dom(); $html->load_file($url_link); 

to:

$html = @file_get_contents($url_link); $html = str_get_html($html); 

hope helps else!