Search on blog:

Python: How to scrape associatedrealtorsaruba.com with selenium

It is example code to scrape it:

#!/usr/bin/env python3

# date: 2020.01.07
# https://stackoverflow.com/questions/59632031/how-to-extract-href-when-href-element-is-a-hyperlink?noredirect=1#comment105434826_59632031

import selenium.webdriver

url = 'https://associatedrealtorsaruba.com/index.php?option=com_ezrealty&Itemid=11&task=results&cnid=0&custom7=&custom8=&parking=&type=0&cid=0&stid=0 …

Python: How to scrape ausrealtimefueltype.global-roam.com with requests

It is example code to scrape it:

#!/usr/bin/env python3

# date: 2020.01.17
# https://stackoverflow.com/questions/59779978/python-requests-output-is-different-to-expected-output/

import requests

headers = {'User-Agent': 'Mozilla/5.0'}

url = 'https://ausrealtimefueltype.global-roam.com/api/SeriesSnapshot?time='

r = requests.get(url,  headers=headers)
data = r.json()

for item in data['seriesCollection …

Python: How to scrape automationpractice.com with selenium

It is example code to scrape it:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException, TimeoutException
import time

try …

Python: How to scrape avanza.se with bank with requests

It is example code to scrape it:

import requests
from bs4 import BeautifulSoup

def display(content, filename='output.html'):
    with open(filename, 'w') as f:
        f.write(content)
    webbrowser.open(filename)

session = requests.Session()
session.headers.update({'USER-AGENT': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:57.0) Gecko/20100101 …

Python: How to scrape bankier.pl with requests

It is example code to scrape it:

import requests
import datetime
import time

# https://www.bankier.pl/inwestowanie/profile/quote.html?symbol=CDPROJEKT

def one_day(symbol):

    print('Symbol:', symbol)

    # jeden dzien
    url = f'https://www.bankier.pl/new-charts/get-data\
?symbol={symbol}\
&intraday=true\
&today=true\
&type=area\
&init=true'

    r …

Python: How to scrape basketball-reference.com with requests, BS

It is example code to scrape it:

# date: 2019.04.28
# author: Bartłomiej 'furas' Burek
# https://stackoverflow.com/a/55885909/1832058

import requests
from bs4 import BeautifulSoup
from bs4 import Comment

url = 'https://www.basketball-reference.com/players/b/bogutan01.html#advanced::none'

r = requests.get(url)

soup = BeautifulSoup(r.content …

Python: How to scrape bcdental.org with requests with ASP.net

It is example code to scrape it:

#
# https://stackoverflow.com/a/48075115/1832058
# 

import requests
from bs4 import BeautifulSoup

url = 'https://www.bcdental.org/yourdentalhealth/findadentist.aspx'

# --- session ---

s = requests.Session() # to automatically copy cookies
#s.headers.update({'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:57.0) Gecko …

Python: How to scrape bing.com with requests, BS

It is example code to scrape it:

#!/usr/bin/env python3

# date: 2020.01.07
# ???

from bs4 import BeautifulSoup
import requests
#import webbrowser

#s = requests.Session()

#headers = {
#    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:73.0) Gecko/20100101 Firefox/73.0'
#}

#response = s.get("https://www.bing.com", headers …

Python: How to scrape blockchain.info with requests

It is example code to scrape it:

#!/usr/bin/env python3

# date: 2020.05.18
# https://stackoverflow.com/questions/61858764/is-there-an-easy-way-to-access-all-transactions-recorded-in-a-bitcoin-block-with/
# 
# https://www.blockchain.com/api/blockchain_api

import requests

r = requests.get('https://blockchain.info/block-height/100?format=json')
data = r.json()

#print(r.text)
#print(data)
print(data['blocks …

Python: How to scrape blog.prepscholar.com with urlib, BS, pandas

It is example code to scrape it:

#!/usr/bin/env python3

# date: 2020.02.26
# https://stackoverflow.com/questions/60407196/creating-csv-spreadsheets-from-web-tables-acquired-through-beautifulsoup

# with pandas 

import pandas as pd

all_tables = pd.read_html('https://blog.prepscholar.com/act-to-sat-conversion')
all_tables[0].to_csv("output1.csv")
all_tables[1].to_csv("output2.csv") 

# with BeautifulSoup it would need …

Python: How to scrape bluebet.com.au with scrapy

It is example code to scrape it:

#
# https://stackoverflow.com/a/47679861/1832058
#

class BlueBet(scrapy.Spider):
    name = "BlueBet"
    start_urls = ['https://www.bluebet.com.au/api/sports/SportsMasterCategory?withLevelledMarkets=true&id=100']

    custom_settings = {
        'FEED_FORMAT': 'csv',
        'FEED_URI': 'odds.csv',
        'FEED_EXPORT_ENCODING': 'utf-8',
    }

    def parse(self, response):
        data = json.loads(response.body)

        for …

Python: How to scrape booksy.com with requests

It is example code to scrape it:

#!/usr/bin/env python3 

# date: 2019.11.21
# https://stackoverflow.com/questions/58964487/beautifulsoup-scraping-other-pages-if-there-is-no-change-in-link-or-href-avail

import requests

headers = {
    'X-Api-Key': 'web-e3d812bf-d7a2-445d-ab38-55589ae6a121'
}

url = 'https://booksy.com/api/pl/2/customer_api/businesses/17101/reviews?reviews_page={}&reviews_per_page=5'

for x in range(1, 6):
    print('--- page:', x, '---')

    r …

Python: How to scrape cafe.daum.net with selenium

It is example code to scrape it:

#!/usr/bin/env python3

# date: 2020.02.23
# https://stackoverflow.com/questions/60362610/python-selenium-click-a-button/

import selenium.webdriver

url = 'http://cafe.daum.net/WekiMeki'

driver = selenium.webdriver.Chrome()
#driver = selenium.webdriver.Firefox()
driver.get(url)

frame = driver.find_element_by_id('down')
driver.switch_to.frame(frame)

driver …

Python: How to scrape cargurus.com with requests, BS

It is example code to scrape it:

#
# https://stackoverflow.com/a/47933667/1832058
#

from bs4 import BeautifulSoup
import requests

params = {
    'zip': '03062',
    'address': 'Nashua,+NH',
    'latitude': "42.73040008544922",
    'longitude': '-71.49479675292969',
    'distance': 50000,
    'selectedEntity': 'c24578',
    'entitySelectingHelper.selectedEntity2': 'c25202',
    'minPrice': '',
    'maxPrice': '', 
    'minMileage': '',   
    'maxMileage': '',   
    'transmission': 'ANY',
    'bodyTypeGroup': '',    
    'serviceProvider': '',  
    'page': 1,
    'filterBySourcesString': '',
    'filterFeaturedBySourcesString …

Python: How to scrape ceneo.pl with scrapy

It is example code to scrape it:

#!/usr/bin/env python3

#
# https://stackoverflow.com/a/47888293/1832058
# 

import scrapy

data = '''https://www.ceneo.pl/48523541, 1362
https://www.ceneo.pl/46374217, 2457'''


class MySpider(scrapy.Spider):

    name = 'myspider'

    start_urls = ['https://www.ceneo.pl/33022301']

    def start_requests(self):
        # get data from …

Python: How to scrape cgtrader.com with scrapy

It is example code to scrape it:

#!/usr/bin/env python3

import scrapy
#from scrapy.commands.view import open_in_browser
#import json

class FileDownloaderItem(scrapy.Item):
    file_urls = scrapy.Field()
    files = scrapy.Field()
    full_urls = scrapy.Field()

class MySpider(scrapy.Spider):

    name = 'myspider'

    allowed_domains = ['www.cgtrader.com']
    start_urls = ['https://www.cgtrader.com/free-3d-print-models …

Python: How to scrape claytoncountyga.gov with selenium with iframe

It is example code to scrape it:

import selenium.webdriver

url = "https://www.claytoncountyga.gov/government/sheriff/inmate-search"
driver = selenium.webdriver.Firefox()
driver.get(url)

iframes = driver.find_elements_by_tag_name('iframe')
print('iframes:', iframes)

driver.switch_to.frame(iframes[0])

item = driver.find_element_by_id('name')
print('name:', item)
item.send_keys("John")

item = driver.find_element_by_name …

Python: How to scrape cnbc.com with requests

It is example code to scrape it:

#
# https://stackoverflow.com/a/47744797/1832058
#

from bs4 import BeautifulSoup
import requests

html = requests.get("https://www.cnbc.com/2017/12/07/pinterest-hires-former-facebook-exec-gary-johnson-to-run-corporate-dev.html").text
soup = BeautifulSoup(html, 'html5lib')

all_paragraphs = soup.find_all('p')

for p in all_paragraphs:
    #print(p) # all HTML
    print(p …

« Page: 2 / 12 »