Python Twitter Tools
In the first episode of playing with twitter, I demo'ed
Tweepy. In this episode, I will use the
twitter library from Mike
Verdone.
- install the library:
pip install twitter
In the README and other sources of documentation, the config for the
user/app
credentials are accessed via os.path utilizing a
read_file_token method.
This method of credential retrieval has perplexed me in the past
because I was
unable to find a sort of "canonical" way of repeating this in other
projects. So
before started evaluating the code base, I set out to see if I could
find a more
'pythonic' way of storing and retrieving credentials. Lo and behold, we
have a
ConfigParser module built right into the standard library. Further,
Doug Hellmann
has graciously covered the ConfigParser module in great detail
here.
To get started with this config option:
- create a twitter.ini file in the same directory as this file (its a
simple text file) - type [twitter] to delineate a section (read about sections in the
ConfigParser tut) parser.getin this case looks in the twitter section and
retrieves the value of the argument as a string.
sample twitter.ini file
[twitter]
CONSUMER_KEY = iowef03434ijsasdfkdmjlfasdasdf2039sdkon
CONSUMER_SECRET = akldsjfsal3fadijfaosdfssasdk34
oauth_token = kasdljnbfglkajdsnfq3asdf4908
oauth_secret = dalkfewoieasdfjaosdkmcadfaslkjdfsdfalasdf
note: these values are bogus
from twitter import *
from ConfigParser import SafeConfigParser
parser = SafeConfigParser()
parser.read('twitter.ini')
consumer_key = parser.get('twitter', 'CONSUMER_KEY')
consumer_secret = parser.get('twitter', 'CONSUMER_SECRET')
oauth_access_token = parser.get('twitter', 'oauth_token')
oauth_access_secret = parser.get('twitter', 'oauth_secret')
twitter = Twitter(auth=OAuth(
oauth_access_token, oauth_access_secret, consumer_key, consumer_secret))
# Update your status
twitter.statuses.update(
status="Control the things that are in your control")
{u'contributors': None,
u'coordinates': None,
u'created_at': u'Tue Apr 23 05:15:56 +0000 2013',
u'entities': {u'hashtags': [], u'urls': [], u'user_mentions': []},
u'favorited': False,
u'geo': None,
u'id': 326565076066766848,
u'id_str': u'326565076066766848',
u'in_reply_to_screen_name': None,
u'in_reply_to_status_id': None,
u'in_reply_to_status_id_str': None,
u'in_reply_to_user_id': None,
u'in_reply_to_user_id_str': None,
u'place': None,
u'retweet_count': 0,
u'retweeted': False,
u'source': u'ipython-local',
u'text': u'Control the things that are in your control',
u'truncated': False,
u'user': {u'contributors_enabled': False,
u'created_at': u'Sun Oct 31 17:44:06 +0000 2010',
u'default_profile': False,
u'default_profile_image': False,
u'description': u'Father. Firefighter. Mobile Crafter. Web Crafter. Garage-Business-Dude. ',
u'entities': {u'description': {u'urls': []},
u'url': {u'urls': [{u'display_url': u'michaelmartinez.in',
u'expanded_url': u'http://michaelmartinez.in',
u'indices': [0, 22],
u'url': u'http://t.co/MTXlg5n87M'}]}},
u'favourites_count': 1,
u'follow_request_sent': False,
u'followers_count': 118,
u'following': False,
u'friends_count': 441,
u'geo_enabled': False,
u'id': 210539658,
u'id_str': u'210539658',
u'is_translator': False,
u'lang': u'en',
u'listed_count': 3,
u'location': u'Arizona',
u'name': u'Michael Martinez',
u'notifications': False,
u'profile_background_color': u'706B5C',
u'profile_background_image_url': u'http://a0.twimg.com/profile_background_images/543977615/tankdude1-Twit.jpg',
u'profile_background_image_url_https': u'https://si0.twimg.com/profile_background_images/543977615/tankdude1-Twit.jpg',
u'profile_background_tile': False,
u'profile_image_url': u'http://a0.twimg.com/profile_images/1496212678/myMascot_normal.png',
u'profile_image_url_https': u'https://si0.twimg.com/profile_images/1496212678/myMascot_normal.png',
u'profile_link_color': u'009999',
u'profile_sidebar_border_color': u'EEEEEE',
u'profile_sidebar_fill_color': u'EFEFEF',
u'profile_text_color': u'333333',
u'profile_use_background_image': True,
u'protected': False,
u'screen_name': u'MonkMartinez',
u'statuses_count': 1582,
u'time_zone': u'Arizona',
u'url': u'http://t.co/MTXlg5n87M',
u'utc_offset': -25200,
u'verified': False}}
The nice thing about PTT is that you can inspect the returned fields of
the
Twitter API requested as you work. This may save time in that you don't
need to
look at the API docs as much. It may also aid in the access of data, as
nested
dict's and tuples are not easy to iterate if you don't know how many
layers of
nesting to uncover. For example:
home_timeline = twitter.statuses.home_timeline() # returns sweet, parseable decoded json
home_timeline[0] # the most recent tweet in my timeline
{u'contributors': None,
u'coordinates': None,
u'created_at': u'Tue Apr 23 05:16:38 +0000 2013',
u'entities': {u'hashtags': [],
u'symbols': [],
u'urls': [{u'display_url': u'oreilly.com/openbook/',
u'expanded_url': u'http://oreilly.com/openbook/',
u'indices': [20, 42],
u'url': u'http://t.co/vtk5YWe44y'}],
u'user_mentions': []},
u'favorite_count': 0,
u'favorited': False,
u'geo': None,
u'id': 326565255465558017,
u'id_str': u'326565255465558017',
u'in_reply_to_screen_name': None,
u'in_reply_to_status_id': None,
u'in_reply_to_status_id_str': None,
u'in_reply_to_user_id': None,
u'in_reply_to_user_id_str': None,
u'lang': u'en',
u'place': None,
u'possibly_sensitive': False,
u'retweet_count': 0,
u'retweeted': False,
u'source': u'<a href="http://www.tweetdeck.com" rel="nofollow">TweetDeck</a>',
u'text': u"Free O'Reilly books http://t.co/vtk5YWe44y",
u'truncated': False,
u'user': {u'contributors_enabled': False,
u'created_at': u'Tue Mar 13 20:12:40 +0000 2007',
u'default_profile': False,
u'default_profile_image': False,
u'description': u'Now putting podcasts in the cloud at http://t.co/kuIpq9dj. Previously Google, Osmosoft. #HTML5 #Rails #UX #DevExp michael@mahemoff.com',
u'entities': {u'description': {u'urls': [{u'display_url': u'player.fm',
u'expanded_url': u'http://player.fm',
u'indices': [37, 57],
u'url': u'http://t.co/kuIpq9dj'}]},
u'url': {u'urls': [{u'expanded_url': None,
u'indices': [0, 19],
u'url': u'http://mahemoff.com'}]}},
u'favourites_count': 1747,
u'follow_request_sent': None,
u'followers_count': 5715,
u'following': True,
u'friends_count': 1138,
u'geo_enabled': False,
u'id': 1112431,
u'id_str': u'1112431',
u'is_translator': False,
u'lang': u'en',
u'listed_count': 486,
u'location': u'London + Melbourne',
u'name': u'Michael Mahemoff',
u'notifications': None,
u'profile_background_color': u'022330',
u'profile_background_image_url': u'http://a0.twimg.com/profile_background_images/97774687/twilk_background_4bddb8dda6006.jpg',
u'profile_background_image_url_https': u'https://si0.twimg.com/profile_background_images/97774687/twilk_background_4bddb8dda6006.jpg',
u'profile_background_tile': False,
u'profile_image_url': u'http://a0.twimg.com/profile_images/1150432577/speakhdr_normal.jpg',
u'profile_image_url_https': u'https://si0.twimg.com/profile_images/1150432577/speakhdr_normal.jpg',
u'profile_link_color': u'717273',
u'profile_sidebar_border_color': u'000000',
u'profile_sidebar_fill_color': u'99CC33',
u'profile_text_color': u'3E4415',
u'profile_use_background_image': True,
u'protected': False,
u'screen_name': u'mahemoff',
u'statuses_count': 12996,
u'time_zone': u'London',
u'url': u'http://mahemoff.com',
u'utc_offset': 0,
u'verified': False}}
home_timeline[0]['text']
u"Free O'Reilly books http://t.co/vtk5YWe44y"
urls is a nested dict inside a dict of from the tweet above. Lets see
how we
would access that data if we needed it...
def double_nest():
for i in home_timeline[0]['entities']['urls']:
print i
return i
double_nest()
{u'url': u'http://t.co/vtk5YWe44y'indices': [20, 42], u'expanded_url':
u'http://oreilly.com/openbook/', u'display_url': u'oreilly.com/openbook/'}
{u'display_url': u'oreilly.com/openbook/',
u'expanded_url': u'http://oreilly.com/openbook/',
u'indices': [20, 42],
u'url': u'http://t.co/vtk5YWe44y'}
So now that we have the basics down, lets poke around some of the more
interesting API endpoints. I'm going to investigate trends and this
requires a
WOEID
and because I am a bit lazy, this WOEID look up should suffice: WOEID
Look
up.
trends_available = twitter.trends.available() # returns a long list of places where trends currently "exist"
trends_available
[{u'country': u'',
u'countryCode': None,
u'name': u'Worldwide',
u'parentid': 0,
u'placeType': {u'code': 19, u'name': u'Supername'},
u'url': u'http://where.yahooapis.com/v1/place/1',
u'woeid': 1},
{u'country': u'Brazil',
u'countryCode': u'BR',
u'name': u'Manaus',
u'parentid': 23424768,
u'placeType': {u'code': 7, u'name': u'Town'},
u'url': u'http://where.yahooapis.com/v1/place/455833',
u'woeid': 455833},
u'placeType': {u'code': 7, u'name': u'Town'},
u'url': u'http://where.yahooapis.com/v1/place/455867',
u'woeid': 455867},
{u'country': u'Argentina',
u'countryCode': u'AR',
u'name': u'Cxf3rdoba',
u'parentid': 23424747,
u'placeType': {u'code': 7, u'name': u'Town'},
u'url': u'http://where.yahooapis.com/v1/place/466861',
u'woeid': 466861},
{u'country': u'Argentina',
u'countryCode': u'AR',
u'name': u'Rosario',
u'parentid': 23424747,
u'placeType': {u'code': 7, u'name': u'Town'},
u'url': u'http://where.yahooapis.com/v1/place/466862',
u'woeid': 466862},
... cut to save space
Tucson, Arizona USA = 2508428. Note for passing parameters into
requests, the
magic underscore is nessesary as seen in _id="2508428
trends_tucson = twitter.trends.place(_id="2508428")
for i in trends_tucson:
for b in i['trends']:
print b['name']
#5PainfulThings
#IntialsOfSomeoneYouCareAbout
#MediaQuestionsForDzhokhar
#encorepaso
#ThereGoesMyPants
Tony Allen
Eric Bledsoe
Matt Barnes
Brandon Belt
Lamar Odom
Nothing really happening in Tucson as these closely match the global
trends...
but San Diego is mentioned, what about San Diego? WOEID = 2487889
trends_SD = twitter.trends.place(_id="2487889")
for i in trends_SD:
for b in i['trends']:
print b['name']
#middleschoolmemories
#IntialsOfSomeoneYouCareAbout
#LHHATL
#ItsTooEarlyTo
#earthday
Tony Allen
Z Bo
Jason's Lyric
Matt Barnes
Rudy Gay
Cool, some of these look interesting but I'll leave the investigation to
you ...
Boston had some major incidents this past week. Are Bostonians still
talking
about it? Boston WOEID = 2367105
trends_Bos = twitter.trends.place(_id="2367105")
for i in trends_Bos:
for b in i['trends']:
print b['name']
#IntialsOfSomeoneYouCareAbout
#ItsTooEarlyTo
#LHHATL
#RAW
Happy Earth Day
Stevie J
#dragrace
Jason's Lyric
Tony Allen
Eric Bledsoe
Hmmm... TV shows and big topics... One would assume (you know about
assumptions) that perhaps we could find
something about the recent events in Boston trending.
Lets see what a geo search for Tucson yields. There are a million of
cool and novel ways to use this data,
this little demo simply shows you how to access the data.
geo_tweets = twitter.geo.search(query="tucson")
geo_tweets['result']
{u'places': [{u'attributes': {},
u'bounding_box': {u'coordinates': [[[-111.059406, 32.033883],
[-110.708206, 32.033883],
[-110.708206, 32.32095],
[-111.059406, 32.32095]]],
u'type': u'Polygon'},
u'contained_within': [{u'attributes': {},
u'bounding_box': {u'coordinates': [[[-114.816591, 31.332177],
[-109.045223, 31.332177],
[-109.045223, 37.00426],
[-114.816591, 37.00426]]],
u'type': u'Polygon'},
u'country': u'United States',
u'country_code': u'US',
u'full_name': u'Arizona, US',
u'id': u'a612c69b44b2e5da',
u'name': u'Arizona',
u'place_type': u'admin',
u'url': u'https://api.twitter.com/1.1/geo/id/a612c69b44b2e5da.json'}],
u'country': u'United States',
u'country_code': u'US',
u'full_name': u'Tucson, AZ',
u'id': u'e6a7d49161470b37',
u'name': u'Tucson',
u'place_type': u'city',
u'url': u'https://api.twitter.com/1.1/geo/id/e6a7d49161470b37.json'},
{u'attributes': {},
u'bounding_box': {u'coordinates': [[[-110.977543, 32.185824],
[-110.960795, 32.185824],
[-110.960795, 32.203223],
[-110.977543, 32.203223]]],
u'type': u'Polygon'},
u'contained_within': [{u'attributes': {},
u'bounding_box': {u'coordinates': [[[-114.816591, 31.332177],
[-109.045223, 31.332177],
[-109.045223, 37.00426],
[-114.816591, 37.00426]]],
u'type': u'Polygon'},
u'country': u'United States',
u'country_code': u'US',
u'full_name': u'Arizona, US',
u'id': u'a612c69b44b2e5da',
u'name': u'Arizona',
u'place_type': u'admin',
u'url': u'https://api.twitter.com/1.1/geo/id/a612c69b44b2e5da.json'}],
u'country': u'United States',
u'country_code': u'US',
u'full_name': u'South Tucson, AZ',
u'id': u'9ba0ab30258b7362',
u'name': u'South Tucson',
u'place_type': u'city',
u'url': u'https://api.twitter.com/1.1/geo/id/9ba0ab30258b7362.json'},
{u'attributes': {},
u'bounding_box': {u'coordinates': [[[-111.218168, 32.154521],
[-111.044793, 32.154521],
[-111.044793, 32.248227],
[-111.218168, 32.248227]]],
u'type': u'Polygon'},
u'contained_within': [{u'attributes': {},
u'bounding_box': {u'coordinates': [[[-114.816591, 31.332177],
[-109.045223, 31.332177],
[-109.045223, 37.00426],
[-114.816591, 37.00426]]],
u'type': u'Polygon'},
u'country': u'United States',
u'country_code': u'US',
u'full_name': u'Arizona, US',
u'id': u'a612c69b44b2e5da',
u'name': u'Arizona',
u'place_type': u'admin',
u'url': u'https://api.twitter.com/1.1/geo/id/a612c69b44b2e5da.json'}],
u'country': u'United States',
u'country_code': u'US',
u'full_name': u'Tucson Estates, AZ',
u'id': u'2df92895f5362a8a',
u'name': u'Tucson Estates',
u'place_type': u'city',
u'url': u'https://api.twitter.com/1.1/geo/id/2df92895f5362a8a.json'},
{u'attributes': {},
u'bounding_box': {u'coordinates': [[[-110.789464, 31.932823],
[-110.755804, 31.932823],
[-110.755804, 31.963138],
[-110.789464, 31.963138]]],
u'type': u'Polygon'},
u'contained_within': [{u'attributes': {},
u'bounding_box': {u'coordinates': [[[-114.816591, 31.332177],
[-109.045223, 31.332177],
[-109.045223, 37.00426],
[-114.816591, 37.00426]]],
u'type': u'Polygon'},
u'country': u'United States',
u'country_code': u'US',
u'full_name': u'Arizona, US',
u'id': u'a612c69b44b2e5da',
u'name': u'Arizona',
u'place_type': u'admin',
u'url': u'https://api.twitter.com/1.1/geo/id/a612c69b44b2e5da.json'}],
u'country': u'United States',
u'country_code': u'US',
u'full_name': u'Corona de Tucson, AZ',
u'id': u'4d35b22f6680264a',
u'name': u'Corona de Tucson',
u'place_type': u'city',
u'url': u'https://api.twitter.com/1.1/geo/id/4d35b22f6680264a.json'}]}
Cool, Now we'll perform a query for tweets containing python job and
iterate
the results:
fun_search = twitter.search.tweets(q="python job",result_type='recent', count=20)
#fun_search # uncomment to see the returned data for fields that look interesting...
for b in fun_search['statuses']:
for c in b['entities']['urls']:
print b['text'], b['created_at'], c['url']
Lead Software Engineer (Server side, Python) * http://t.co/qObCXSv46P
* Empresa: F-Secure * Lugar: Helsinki #empleo #trabajo #finlandia Tue Apr 23
05:22:07 +0000 2013 http://t.co/qObCXSv46P
#Python Install and configure reddit clone script on amazon web services by
gorrrra: Configure existing r... http://t.co/VgYuI3te4T #Job Tue Apr 23 05:14:09
+0000 2013 http://t.co/VgYuI3te4T
New Job Alert: Senior Software Engineers (Ruby on Rails, Python and/or Java)
at Rafte... http://t.co/Ef7ZsAGdAe #python #jobs Tue Apr 23 05:07:38 +0000 2013
http://t.co/Ef7ZsAGdAe
Site Reliability Engineer http://t.co/J784yxGA3w #java #python #jobs #hiring
#careers Tue Apr 23 05:00:47 +0000 2013 http://t.co/J784yxGA3w
#uitgeverij #Belgie Python Developer gevraagd voor interne functie te
Antwerpen: Comput... http://t.co/NstoYH6KIh http://t.co/6R25reAyu2 Tue Apr 23
04:47:51 +0000 2013 http://t.co/NstoYH6KIh
#uitgeverij #Belgie Python Developer gevraagd voor interne functie te
Antwerpen: Comput... http://t.co/NstoYH6KIh http://t.co/6R25reAyu2 Tue Apr 23
04:47:51 +0000 2013 http://t.co/6R25reAyu2
[Full-time] Seeking Passionate Python developers at Soshio
http://t.co/YVWCEEBBhA Tue Apr 23 04:36:33 +0000 2013 http://t.co/YVWCEEBBhA
RT @zs_frontline: ［Web求人情報］
Python，Ruby，Perl，PHP，Javaなどを用いたWebアプリの開発やフロントエンド、デザイン案件まで多数あります！...
http://t.co/t8WBRghjFF Tue Apr 23 04:03:26 +0000 2013 http://t.co/t8WBRghjFF
［Web求人情報］
Python，Ruby，Perl，PHP，Javaなどを用いたWebアプリの開発やフロントエンド、デザイン案件まで多数あります！...
http://t.co/t8WBRghjFF Tue Apr 23 04:00:06 +0000 2013 http://t.co/t8WBRghjFF
#job http://t.co/ew5ADWt8oE
- Senior Python / MongoDB Developer, Array Tue Apr 23 03:55:12 +0000 2013
http://t.co/ew5ADWt8oE
#job http://t.co/4WMtLjTtJ0
- Разработчик Python, Array Tue Apr 23 03:54:08 +0000 2013
http://t.co/4WMtLjTtJ0
Python Flask Data Collection http://t.co/cLcyyLaqri Tue Apr 23 03:53:40
+0000 2013 http://t.co/cLcyyLaqri
RT @findmjob: Site Reliability Engineer http://t.co/J784yxGA3w #php #git
#java #rest #unix #ruby #mysql #redis #python #hadoop... http://t.… Tue Apr 23
03:46:13 +0000 2013 http://t.co/J784yxGA3w
Site Reliability Engineer http://t.co/J784yxGA3w #php #git #java #rest #unix
#ruby #mysql #redis #python #hadoop... http://t.co/6xrPUo9zRr Tue Apr 23
03:42:08 +0000 2013 http://t.co/J784yxGA3w
Site Reliability Engineer http://t.co/J784yxGA3w #php #git #java #rest #unix
#ruby #mysql #redis #python #hadoop... http://t.co/6xrPUo9zRr Tue Apr 23
03:42:08 +0000 2013 http://t.co/6xrPUo9zRr
Programador Python: Porto Alegre - RS - Anúncio nº: 919784 Programador(a)
PythonAnúncio p... http://t.co/fYoyZ35xLT #vagas #programador Tue Apr 23
03:29:42 +0000 2013 http://t.co/fYoyZ35xLT
New Job Alert: Software Engineer - Python at Jawbone (San Francisco, CA):
technical design speci... http://t.co/02proruK5j #python #jobs Tue Apr 23
03:09:55 +0000 2013 http://t.co/02proruK5j
#job #python Write up 3 algorithm base on an interface and abstraction in
OCaml by kennguy13: Sorry... http://t.co/PGb3klkXbV #freelance Tue Apr 23
03:06:00 +0000 2013 http://t.co/PGb3klkXbV
#C Write up 3 algorithm base on an interface and abstraction in OCaml by
kennguy13: Sorry the project has... http://t.co/eV5enpwGcD #JOB Tue Apr 23
03:02:37 +0000 2013 http://t.co/eV5enpwGcD
#C Write up 3 algorithm base on an interface and abstraction in OCaml by
kennguy13: Sorry the pro... http://t.co/V5cPAOv7PX #Programming Tue Apr 23
03:02:36 +0000 2013 http://t.co/V5cPAOv7PX
#Python Write up 3 algorithm base on an interface and abstraction in OCaml
by kennguy13: Sorry the projec... http://t.co/57Kw3abxVV #Job Tue Apr 23
02:18:35 +0000 2013 http://t.co/57Kw3abxVV
If the traditional method of accessing Twitter bores you or you want
finer
control over your use of the Twitter services, I think Python Twitter
Tools is a
viable option. It reliably hits and maps to the Twitter API and comes
packed
with an IRCBOT and commandline utilities. PTT is actively maintained as
of this
writing and looks well constructed in my opinion.