AtBS 11 Web Scraping

Question 1

How to open up Google from the shell?

Answer 1

import webbrowser

webbrowser.open (‘www.google.com’)

Question 2

What module lets you easily download files from the Web?

Answer 2

request module

need to pip install request the first time to get it.

Question 3

Syntax to download a webpage?

Answer 3

res = requests.get (‘URL’)

pg 237

Question 4

How to check if a request download worked?

Answer 4

res.status_code == requests .codes.ok

should equal True

Question 5

How to find the length of a requests download?

Answer 5

len (res.text)

Question 6

When saving a web page what is important about saving it and why?

Answer 6

Need to save it in a binary format.

Important to do this so that the file can maintain Unicode characters.

pg 239

Question 7

What does res stand for?

Answer 7

Response

It is what you get from a requests.get (‘URL’) pull

Question 8

What does res .iter_content (100000) do?

Answer 8

It helps to download files in chuncks instead of having to pull everything at once.

Question 9

Steps to download and save webpage to harddrive?

Answer 9

import requests

res = requests .get(‘URL’)

FileName= open (‘SaveFileName’, ‘wb’)

for chunk in res.iter_content(100000):
FileName .write(chunk)

FileName .close()

Question 10

What does the res.raise_for_status() do>

Answer 10

It checks for an error when downloading a webpage

pg 238

Question 11

how to import the Beautiful Soup module?

Answer 11

import bs4

Question 12

Steps to create a Beautiful Soup Object from a webpage?

Answer 12

import requests, bs4

res = requests.get(‘URL’)

res.raise _for_status()

Soup Var Name = bs4.BeautifulSoup (res.text)