Search on blog:

Python: How to scrape aastocks.com with requests

It is example code to scrape it:

# date: 2019.09.16
# https://stackoverflow.com/questions/57861715/scrapy-infinite-scrolling-no-pagination-indication
# http://www.aastocks.com
import requests

newstime = '934735827'
newsid = 'HKEX-EPS-20190815-003587368'

url = 'http://www.aastocks.com/tc/resources/datafeed/getmorenews.ashx?cat=all&newstime={}&newsid={}&period=0&key=&symbol=00001'
url_artickle = "http://www.aastocks.com/tc/stocks/analysis/stock-aafn-con/00001/{}/all"

for x in range(3):

    print('---', x, '----')
    print('data:', url.format(newstime, newsid))

    r = requests.get(url.format(newstime, newsid))
    data = r.json()

    #for item in data[:3]: # test only few links
    for item in data[:-1]: # skip last link which gets next page
        r = requests.get(url_artickle.format(item['id']))
        print('news:', r.status_code, url_artickle.format(item['id']))

    # get data for next page
    newstime = data[-1]['dtd']
    newsid = data[-1]['id']
    print('next page:', newstime, newsid)

Python: How to scrape allegro.pl with scrapy

It is example code to scrape it:

# date: 2017.12.10
# https://stackoverflow.com/a/47744135/1832058

import scrapy

#from allegro.items import AllegroItem

#class AllegroItem(scrapy.Item):
#    product_name = scrapy.Field()
#    product_sale_price = scrapy.Field()
#    product_seller = scrapy.Field()

class AllegroPrices(scrapy.Spider):

    name = "AllegroPrices"
    allowed_domains = ["allegro.pl"]

    start_urls = [
        "http://allegro.pl …

Python: How to scrape alloschool.com with scrapy

It is example code to scrape it:

#!/usr/bin/env python3

# date: 2019.07.29
# https://stackoverflow.com/questions/57245315/using-scrapy-how-to-download-pdf-files-from-some-extracted-links

import scrapy

class MySpider(scrapy.Spider):

    name = 'myspider'

    start_urls = [
          'https://www.alloschool.com/course/alriadhiat-alaol-ibtdaii',
    ]

    def parse(self, response):

        for link in response.css('.default .er').xpath('@href').extract …

Python: How to scrape amazon.com (2) with selenium

It is example code to scrape it:

#!/usr/bin/env python3

# date: 2020.03.30

import selenium.webdriver

url = 'https://www.amazon.com/international-sales-offers/b/?ie=UTF8&node=15529609011&ref_=nav_cs_gb_intl'

driver = selenium.webdriver.Firefox()
driver.get(url)

for x in range(10):
    deal = driver.find_element_by_id('100_dealView_' + str(x))

    image …

Python: How to scrape aopa.org with selenium

It is example code to scrape it:

# https://stackoverflow.com/questions/60601053/python-selenium-for-loop-iterates-through-entire-website/60601428

from selenium import webdriver
import time

driver = webdriver.Chrome()

#wait = WebDriverWait(driver, 10)

driver.get("https://www.aopa.org/destinations/airports/state/AL")
time.sleep(3)

airport_list = []
paved_runway = []

airport_row = driver.find_elements_by_xpath('//div[@class = "state-airports__airport"]')
print(len …

Python: How to scrape api.weatherflow.com with requests

It is example code to scrape it:

#!/usr/bin/env python3

# date: 2020.02.10
# 

import requests

url = 'https://api.weatherflow.com/wxengine/rest/model/getModelDataBySpot?model_id=-1&spot_id=110&units_wind=mph&units_temp=F&format=json&wf_apikey=84e778ae-fe8e-4b8f-8d33-6bc88967a2b1&wf_token=f147702351af100d7c220b633d085318&v=1.1'
r = requests.get(url)
data = r.json …

Python: How to scrape apps.upenn.edu with selenium

It is example code to scrape it:

#!/usr/bin/env python3

# date: 2020.02.26
# 

import selenium.webdriver

def scrape(last_name, first_name):        
    url = 'https://directory.apps.upenn.edu/directory/jsp/fast.do'

    driver = selenium.webdriver.Firefox()
    driver.get(url)

    inputs = driver.find_elements_by_tag_name('input')

    #for item in inputs:
    #    print(item.get_attribute …

Python: How to scrape associatedrealtorsaruba.com with requests

It is example code to scrape it:

#!/usr/bin/env python3

# date: 2020.01.07
# https://stackoverflow.com/questions/59632031/how-to-extract-href-when-href-element-is-a-hyperlink?noredirect=1#comment105434826_59632031

import requests
from bs4 import BeautifulSoup as BS

url = 'https://associatedrealtorsaruba.com/index.php?option=com_ezrealty&Itemid=11&task=results&cnid=0&custom7=&custom8=&parking=&type …

Python: How to scrape associatedrealtorsaruba.com with selenium

It is example code to scrape it:

#!/usr/bin/env python3

# date: 2020.01.07
# https://stackoverflow.com/questions/59632031/how-to-extract-href-when-href-element-is-a-hyperlink?noredirect=1#comment105434826_59632031

import selenium.webdriver

url = 'https://associatedrealtorsaruba.com/index.php?option=com_ezrealty&Itemid=11&task=results&cnid=0&custom7=&custom8=&parking=&type=0&cid=0&stid=0 …

« Page: 1 / 22 »