Python: How to get many tweets with function search() in Twython

In module Twython you can use function search() to search tweets

import os
from twython import Twython

CONSUMER_KEY    = os.getenv('TWITTER_CONSUMER_KEY')
CONSUMER_SECRET = os.getenv('TWITTER_CONSUMER_SECRET')

# --- main ---

twitter = Twython(CONSUMER_KEY, CONSUMER_SECRET)

result = twitter.search(q='python', count=100)
tweets = result['statuses']

for number, item in enumerate(tweets, 1):
    print(number, '|', item['id'], '|', item['created_at'], '|', item['user']['name'], '|', item['text'][:20])

but it can find maximum 100 tweets (and sometimes even less) because option count can't use bigger values then 100. It is restricted by official Twitter API. You would have run this function many times with different option max_id= to get older and older tweets.

But there is simpler method. You can use function cursor() to create generator which can be used with for-loop to get as many tweets as you wish. You event have to use break to exit this loop because it will run (almost) forever.

import os
from twython import Twython

CONSUMER_KEY    = os.getenv('TWITTER_CONSUMER_KEY')
CONSUMER_SECRET = os.getenv('TWITTER_CONSUMER_SECRET')

# --- main ---

client = Twython(CONSUMER_KEY, CONSUMER_SECRET)

tweets = client.cursor(client.search, q='#python')

for number, item in enumerate(tweets, 1):
    print(number, '|', item['id'], '|', item['created_at'], '|', item['user']['name'], '|', item['text'][:20])

    if number >= 100:
        break

To save data in file you have to collect tweets in some list

import os
from twython import Twython
import json

CONSUMER_KEY    = os.getenv('TWITTER_CONSUMER_KEY')
CONSUMER_SECRET = os.getenv('TWITTER_CONSUMER_SECRET')

# --- main ---

client = Twython(CONSUMER_KEY, CONSUMER_SECRET)

tweets = client.cursor(client.search, q='#python')

data = []

for number, item in enumerate(tweets, 1):
    data.append(item)

    if number >= 100:
        break

print(data)

with open('output.json', 'w') as fh:
    json.dump(data, fh)

You can use data to create DataFrame

df = pd.DataFrame(data)

or you can use df.append(item, ignore_index=True) to append to existing DataFrame

import os
from twython import Twython
import pandas as pd

CONSUMER_KEY    = os.getenv('TWITTER_CONSUMER_KEY')
CONSUMER_SECRET = os.getenv('TWITTER_CONSUMER_SECRET')

# --- main ---

client = Twython(CONSUMER_KEY, CONSUMER_SECRET)

tweets = client.cursor(client.search, q='#python')

df = pd.DataFrame()

for number, item in enumerate(tweets, 1):
    #df = df.append(item, ignore_index=True)

    df = df.append({
                    'id': result['id'],
                    'user': result['user']['name'],
                    'text': result['text'],
                   }, ignore_index=True)

    if number >= 100:
        break

print(df)

df.to_csv('output.csv')

Notes:

Twython documentation: Search Generator

If you like it

Buy a Coffee