AR
и у амазона всегда несколько вёрсток, отдающихся рандомно, может из-за этого
Size: a a a
2020 March 13
AR
да
AR
можешь в пауке сохранить респонс в файл и открыть в браузере
AK
extract URLs from your results (if that's impossible then from logs) and compare two lists
How can I do that in the spider and scrape the missing urls?
AR
you shouldn't do that in the spider, you should process the results outside it
AR
а нельзя разве в параметрах url передать
каких параметрах
AK
Is there a way to do it in the spider?
AR
yes, just write the code that loads the old results or logs, processes them and compares that with the old URL list
AR
which is suboptimal, of course
AR
to do that correctly you should have initially written your spider with retrying in mind, but it's too late now
AK
I have given retry times as 70 and download delay. Concurrent requests = 50
AR
whatever
AK
Anil Kumar
I have given retry times as 70 and download delay. Concurrent requests = 50
Download delay =0
AR
it doesn't matter now how you ran your previous job, if you don't have a list of URLs to retry you need to make it by comparing all URLs with successful URLs, or by finding failed URLs in the logs, there is no other way
AR
there would be a lot of options if you asked before running the initial job
AK
there would be a lot of options if you asked before running the initial job
Daily I have to scrape 200000 urls. Now tell me the options which you got
AK
I'll modify the scraper