Python:Request

Request

Install Requests

pip install requests
pip install beautifulsoup4
pip install selenium

Carrega uma pagina

from urllib.request import urlopen 
html = urlopen("http://www.pagina_exemplo.com")

BeaultifulSoap

Métodos de coleta

[[1]]

= Recupera todos os href da pagina

############################## Carrega arquivo para memória
html = open('dataset/result/site-01.html')
bs = BeautifulSoup(html, 'html.parser')
############################## Busca todos os A HREF
curriculos = []
for link in bs.find_all('a'):
   js = link.get('href')
   curriculos.append(js)

Para ler

https://acervolima.com/raspar-sites-habilitados-para-javascript-usando-scrapy-selenium/