Телеграмм чат группы scrapy

Size: a a a

Scrapy

2020 March 10

Anil Kumar in Scrapy

I have 1000 pages to follow and each page contains 40 links to the pages which I have to scrape data from. Which spider you guys suggest? Single domain only.

источник

17:39пожаловаться #1

Andrey Rahmatullin in Scrapy

Spider.

источник

17:39пожаловаться #2

Andrey Rahmatullin in Scrapy

the page counts don't matter in any case

источник

17:40пожаловаться #3

ildar in Scrapy

Anil Kumar

How to set retry times for a particular request in crawl spider?

just for curiosity, why do you need this?

источник

17:43пожаловаться #4

Anil Kumar in Scrapy

ildar

just for curiosity, why do you need this?

I'm doing freelancing job. I need it

источник

17:45пожаловаться #5

Anil Kumar in Scrapy

Andrey Rahmatullin

Spider.

I didn't get

источник

17:46пожаловаться #6

Andrey Rahmatullin in Scrapy

Anil Kumar

I didn't get

what answer did you want then?

источник

17:46пожаловаться #7

ildar in Scrapy

ehm, I'm asking why you have to set it for every request "manually"?

источник

17:46пожаловаться #8

Anil Kumar in Scrapy

Scrapy normal spider or crawl spider?

источник

17:46пожаловаться #9

Andrey Rahmatullin in Scrapy

Spider is the "normal spider"

источник

17:47пожаловаться #10

ildar in Scrapy

selecting between depends only on this thing - do you need to use Rule objects? If yes, use CrawlSpider.

источник

17:50пожаловаться #11

Anil Kumar in Scrapy

ildar

ehm, I'm asking why you have to set it for every request "manually"?

For pagination If I get error. I'll not get the remaining data. Thats why

источник

17:50пожаловаться #12

Andrey Rahmatullin in Scrapy

set it globally.

источник

17:51пожаловаться #13

ildar in Scrapy

you know, in most cases, scrapy will do this for you, automatically!

источник

17:51пожаловаться #14

ildar in Scrapy

you don't need to bother about this until encounter it, I suppose even default settings will do the job.

источник

17:52пожаловаться #15

Anil Kumar in Scrapy

ildar

you don't need to bother about this until encounter it, I suppose even default settings will do the job.

I got the error. That's why I'm asking

источник

17:52пожаловаться #16

ildar in Scrapy

in particular request and every other go without problems?

источник

17:53пожаловаться #17

Anil Kumar in Scrapy

I'm taking about only paginated links

источник

17:54пожаловаться #18

ildar in Scrapy

Then set max_retry_times on this particular request, see here: https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#retry-times

источник

17:56пожаловаться #19

МС

Михаил Синегубов in Scrapy

народ, может у него ajax какой с кастомным заголовком?
пускай сайт покажет... если это не военная тайна

источник

17:56пожаловаться #20