furas.pl
# prywatne notatki - Python, Linux, Machine Learning, etc.

Last update: 2022.03.28

Get HTML from server

urllib.request

• standard module, preinstalled with Python
• some operations need more code than requests
• it has urlretrive() to download file

Requests

• popular module which makes life easier
• extensions and modifications
  • requests-html
  • requests-async
  • requests-threads
  • requests-futures
  • grequests
  • requests_toolbelt

Search data in HTML

BeautifulSoup

• it uses css or own functions which can use regex
• it doesn't use xpath

lxml

• it uses xpath
• it doesn't use css

Parsel

• it uses css or xpath with regex
• it is used by Scrapy

cssselector

• it converts css to xpath and use lxml to search

pyquery

• it uses selectors like jquery.
• it uses pseudo classes which doesn't exist in css ie. :first :last :even :odd :eq :lt :gt :checked :selected :file

Scraping framework(s)

Scrapy

• it uses xpath and css selectors
• it can run many processes at the same time
• it can use proxies
• it can be used on servers zyte.com
• extension to work with selenium: scrapy-selenium

Source: https://doc.scrapy.org/en/latest/topics/architecture.html

MechanicalSoup

• it is Requests + BeautifulSoup

RobotFramework

RoboBrowser

mechanize

Work with JavaScript

If page uses JavaScript then you may need one of this modules which can control real web browser

Selenium

• it controls real web browser which can run JavaScript
• use xpath, css selectors
• drivers
  • standard driver for Chrome: chromedriver
  • standard driver for Firefox: geckodriver
  • standard driver for Edge: edge driver
  • special driver for Chrome: undetected chromedriver
  • module to automatically download driver: webdriver-manager

See also selenium.dev

pyppeteer

playwright

Other tools for scraping

Portia

• docker with tool that allows for visually scraping.
• created by authors of Scrapy

Other tools for help

httpbin.org

• it can be used to test HTTP requests
• it sends back all data which it gets so you check if your requests creates correct data.

ToScrape.com

Web Scraping Sandbox with two fictional pages/portals which you can use it to learn scraping.

There are examples with:

• normal pagination
• infinite scrolling pagination
• JavaScript generated content
• a table based messed-up layout
• login with CSRF token (any user/passwd works)
• ViewState (C# DotNet)

curlconverter.com

• can convert curl command to code in Python (requests) or other languages
• some (API) documentation show examples as command curl
• some conversion may have mistake

Older link curl.trillworks.com

Similar:

• reqbin.com

• curl2scrapy (convert for module scrapy)

• tools Postman and Insomni also can generate code for Python

"DevTools" in Chrome and Firefox

• tab: Inspecion - to search items in HTML and get CSS or XPath selector. (but it gives selector which doesn't use classes and ids so it can be long and unreadable for human)
• tab: Network - to see all requests from browser to server and get requests used by JavaScript to get
• tab: Console - to test JavaScript code or use $("...") to test css selector or $x("...") to test xpath
• extensions:
  • button to turn off JavaScript:
    • JavaScript Toggle On and Off
    • disable-javascript

Extra doc for Firefox

Tools which can be used to test requests and API

• Postman
• Insomnia

They can also generate code in python using urllib or requests

• charlesproxy - local proxy server
• mitmproxy - local proxy server (created in Python)

By The Way

pandas.read_html() can read HTML from file or url and scrape data from all standard <table> and create list with DataFrames

Example codes for different pages and different tools on my GitHub:

• furas / python-examples / scraping

Load image and display 3 seconds

from manim import *


class Example(Scene):

    def construct(self):
        self.wait(.1) # to skip animation of first image

        image = ImageMobject('lenna.png')

        self.add(image)
        self.wait(5)


if __name__ == '__main__':
    import subprocess
    subprocess.run(['manim', '-p', '-qh', __file__, 'Example'])

Duplicate image and display …

There are two versions of manim.

• version created by 3Blue1Brown
• version maintained by community

There is also deprecated version but it is replaced by community version.

pip install manimgl

In documentation you can see that you can run manim code in console

manim -p -ql script.py Scene1 Scene2 Scene3

where

-p        :preview after rendering
-ql       :quality low
script.py :file with manim code
Scene     :class name(s) in manim code

To make life easier you can run it …

Python has complex numbers which use char j instead of i to define imaginary part.

You can be split complex number to real and imaginary part using .real and .imag

You can use complex() or 1j to convert parts back to complex number.

You can get conjugated number using .conjugate …

Python ma liczby zespolone - complex - które używają znaku j zamiast i do definiowania części wyimaginowanej.

Można ją rozdzielić na część rzeczywistą i wyimaginowaną używając .real i .imag.

Części można ponownie zamienić na liczbę zespoloną używając complex() lub 1j.

Liczbę sprzeżoną można pobrać za pomocą .conjugate()

Moduł (wartość bezwględną) można pobrać …

To jest notacja używana w Numpy i Pandas.

Pozwala on na dostęp do elementów w tablicy 2D bez użycia pętli for.

[ : , 0 ] oznacza (mniej więcej) [ pierwszy_wiersz:ostatni_wiersz , kolumna_0 ].

Dla dwuwymiarowej tablicy/matrycy/dataframe to daje wszystkie wartości w kolumnie 0 (we wszystkich wierszach).

col_0 = data[:,0]
col_1 = data[:,1]

Dla …

It is a notation used in Numpy and Pandas.

It lets you access elements in 2D array without for-loop.

[ : , 0 ] means (more or less) [ first_row:last_row , column_0 ].

For 2D array/matrix/dataframe this gives all values in column 0 (from all rows).

col_0 = data[:,0]
col_1 = data[:,1]

For …

Original sys.stdout has method write() and function print() uses this sys.stdout.write() to send text to console.

To redirect print() to element on page or in GUI you have to create class with method write() which gets text and puts on page or in GUI and assign this …

Oryginalnie sys.stdout posiada methodę write() i funkcja print() używa tego sys.stdout.write() do wysłania tekstu na konsolę.

Aby przekierować print() do elementu na stronie lub w GUI trzeba stworzyć klasę z metodą write(), która pobiera tekst i umiesza go na stronie lub w GUI, i należy przypisać tą …

• toscrape.com

  Web Scraping Sandbox created by Scrapinghub

  A fictional bookstore that desperately wants to be scraped. It's a safe place for beginners learning web scraping and for developers validating their scraping technologies as well.

  Available at: books.toscrape.com

  A website that lists quotes from famous people. It has …

• toscrape.com

  Web Scraping Sandbox stworzony przez Scrapinghub

  Fikcyjna księgarnia, która desperacko chce być zeskrapowana. To bezpieczne miejsce dla początkujących uczących się we scrapingu, jak również dla deweloperów sprawdzających swoje umiejętności scrapingu

  Dostępne na: books.toscrape.com

  Strona, która wyświetla cytaty znanych ludzi. Ma wiele odmian strony, które pokazują cytaty …

Widget Text has function insert() to insert text and it has option tags to assing tag (or many tags) to inserted text. This tag can be used to change color (background, foreground) and font and other properties.

import tkinter as tk

root = tk.Tk()

txt = tk.Text(root)
txt.pack …

To have only one Toplevel window you can disable button which open this Toplevel window and enable it when you close this Toplevel window

You can use button['state'] to set it 'disable' or 'normal'

You needs window.protocol("WM_DELETE_WINDOW", function) to execute function when you use button [X] to …

Aby mieć tylko jedno okno Toplevel można zablokować przycisk, który otwiera to okno Toplevel i odblokować przycisk gdy to okno jest zamykane.

Można do tego użyć button['state'] i ustawiać 'disable' lub 'normal'

Potrzebne też jest window.protocol("WM_DELETE_WINDOW", function) do wywołania funkcji gdy zostani wciśnięty przycisk [X] zamykający okno …

Widget Text ma funkcję insert() do wstawiania tekstów i ona ma opcję tags do przypisywania tagu (lub wiele tagów) do wstawianego tekstu. Ten tag może być użyty do zmiany koloru (background, foreground) oraz czcionki oraz innych właściwości.

import tkinter as tk

root = tk.Tk()

txt = tk.Text(root)
txt.pack …

Google-Text-To-Speech (shortly gtts) converts text to speech in file MP3 and popular method to play it without saving on disk was to use PyGame

from gtts import gTTS
from io import BytesIO

text = "Hello World!"

# get audio from server
tts = gTTS(text=text, lang=lang)

# convert to file-like object
fp …

• strftime.org
• pyformat.info
• Learn X in Y minutes Where X=Python
• Learntris - Create Tetris with TDD