Телеграмм чат группы scrapy

oh, i'll not find such vpn

The website doesn't allow bots to scrape. Still can we scrape the website?

11:46пожаловаться #1

Anil Kumar

The website doesn't allow bots to scrape. Still can we scrape the website?

recaptcha or something?

11:46пожаловаться #2

Anil Kumar in Scrapy

recaptcha or something?

No captchas. But in robots.txt they mentioned "Disallow: /".

11:47пожаловаться #3

Anil Kumar

No captchas. But in robots.txt they mentioned "Disallow: /".

set ROBOTSTXT_OBEY = False in your settings

11:48пожаловаться #4

Anil Kumar in Scrapy

set ROBOTSTXT_OBEY = False in your settings

But page is rendering through Javascript. When I open the website in my chrome most of the times it is showing connection timed out.

11:49пожаловаться #5

https://docs.scrapy.org/en/latest/topics/dynamic-content.html

Anil Kumar

But page is rendering through Javascript. When I open the website in my chrome most of the times it is showing connection timed out.

11:50пожаловаться #6

Anil Kumar in Scrapy

https://docs.scrapy.org/en/latest/topics/dynamic-content.html

It is showing connection timed out in my browser. What might be the reason for this error.

11:50пожаловаться #7

same as for me, looks like it doesn't trust you

11:51пожаловаться #8

Anil Kumar in Scrapy

same as for me, looks like it doesn't trust you

Did it block my ip?

11:52пожаловаться #9

If you can open it from a browser, then no. If you can't open it with other tool, maybe it doesn't believe, that you use a browser

11:55пожаловаться #10

Anil Kumar in Scrapy

same as for me, looks like it doesn't trust you

Now open the link once.

11:55пожаловаться #11

D3301 in Scrapy

Andrey Rahmatullin

Find where does the website get that price

Gonna start looking now

11:59пожаловаться #12

Anil Kumar in Scrapy

Can we run splash without docker?

13:06пожаловаться #13

Elisei Badanin in Scrapy

sure

13:26пожаловаться #14

Anil Kumar in Scrapy

Elisei Badanin

sure

How?

13:30пожаловаться #15

Anil Kumar

How?

Look in the Dokerfile

13:34пожаловаться #16

Anil Kumar in Scrapy

Look in the Dokerfile

I don't want to install docker at all. But I nedd splash to run. Is it possible?

13:35пожаловаться #17

Read about Doker... Dokerfile describes installation flow in virtual OS

13:39пожаловаться #18

Also, "I don't want to install docker" sounds like, "I really want to make problems for myself"

13:42пожаловаться #19

Anil Kumar in Scrapy

Also, "I don't want to install docker" sounds like, "I really want to make problems for myself"

I mean is there any alternative.