Search on blog:

Python: How to scrape with requests

It is example code to scrape it:

#!/usr/bin/env python3

# date: 2020.01.07

import requests
from bs4 import BeautifulSoup as BS

url = ''

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:71.0) Gecko/20100101 Firefox/71.0', # need full UA

for x in (0, 20, 40):
    r = requests.get(url + '&limitstart={}'.format(x), headers=headers)
    print('\n---', x, '---\n')

    soup = BS(r.text, 'html.parser')

    all_items = soup.find_all('span', {'class': 'h3'})
    for item in all_items:
If you like it
Buy a Coffee