>Very good knowledge of Python / Scrapy, MySQL,
>Good written English
>Working with medium SQL/NoSQL databases;
>Experience working with crawlers; …
>Knowledge of HTML/CSS
Parsing sites and social networks, online stores.
(Olx.ua *, prom.ua, avito.ru, facebook.com, vk.com, www.work.ua)
Filling with these data stores and sites, downloading pictures, creating unique images. Full collection of texts and data, all content, both from static sites and from dynamic ones. Prices from 0.001 $ per 1 page. Parsing takes place in asynchronous mode providing the ability to download more than a million records per day.
Also the creation of highly-loaded servers, for the delivery of large amounts of data in asynchronous mode.
Sample server and client script, real server, compare with your server!
The cycle of recording 10,000 records and reading these records takes 15 seconds
(1333 requests in 1 second)
- data on the server is stored in radish …
Start URL for the record - http://22.214.171.124:8080/set?0000000000=0000000000
The final URL for the record is http://126.96.36.199:8080/set?0000010000=0000010000
Start URL for reading - http://188.8.131.52:8080/get?0000000000
The final URL for reading is http://184.108.40.206:8080/get?0000010000
- Query test for data - Requests / sec: 1768.67 - http://pix.toile-libre.org/upload/original/1499242652.png
- Query test to the server without data access - Requests / sec: 33823.25 -http: //pix.toile-libre.org/upload/original/1499242827.png [email protected]