AR
do you want to only rescrape failed ones?
Size: a a a
2020 March 13
AR
or are there new 200000 URLs each day?
AK
New urls each day
AK
or are there new 200000 URLs each day?
Reply please
AR
and do you want to run several job attempts each day?
AK
No, I don't want line that. If I run the scraper once it should scrape all the urls and it should check the count of urls, if count is less than it should scrape the missing urls.
AR
that's not different from just increasing RETRY_TIMES
🎱
можешь в пауке сохранить респонс в файл и открыть в браузере
этот способ помог, спасибо)
открыл в шелле так
открыл в шелле так
view(response)AK
that's not different from just increasing RETRY_TIMES
I don't know whether it is getting error or not
AR
then the spider doesn't either
AK
I don't know whether it is skipping due to error or some other reason
AK
I need the reason for missing urls
AR
check the logs
AK
Ok
AK
check the logs
Thank you
i
Anil Kumar
I have given retry times as 70 and download delay. Concurrent requests = 50
btw are you aware of https://docs.scrapy.org/en/latest/topics/autothrottle.html ?
AR
with "I don't know whether it is getting error or not", definitely