US Politics Part I

Christopher Marks

2019-01-18

PART I: Initialize and Collect User Timelines & Friends

This is the first part of a two-part vignette designed to collect and analyze Twitter posts from members of the US Congress and the US President. This part of the vignette carries out the database initialization and initial status collections.

Very Important Note: This vignette is set up to collect the relationships and timelines of over 500 politicians. Because people in this profession tend to be prolific Tweeters, the initial calls to the collection functions can take a very long time to complete. In particular, collecting all of the 500+ users’ friends will take approximately 15-20 hours. Collection of all of the users’ timelines, and performing sentiment analysis on them, will likely take several days.

For an abbreviated version of this vignette, you can perform the same collection and analysis on a small sample of the politicians. Code to perform this sampling is included in this vignette, but is commented out.

The second part of this vignette performs updates the database and analyzes the results.

Set Java Options

The sentiment analysis we conduct during Twitter collection tends to require more Java heap space than the default. Therefore, we start by allocating more space.

Package Import

Let’s start by importing the twinfoR namespace. Don’t forget to install the package, if you haven’t already.

# install.packages("twinfoR.zip",repos=NULL,type="binary")
library(twinfoR)

Some important initializations

Initialize database

We need to set a working directory to store the database and analysis products. Once we set the directory, we’ll change into it and initialize the politician database.

Authenticate to Twitter API

We’ll use the INFORM-Tutorial Twitter Application. Alternatively, you can supply your own application credentials using the authorize_app function.

You will need a Twitter Account to continue! If you don’t have one, you can easily create one on the Twitter website. If you do not have a Twitter account and do not wish to make one, this package will be of very limited use to you.

I recommend logging onto Twitter in your Browser before running this line. Either way, be careful not to hit immediately after running it, for example by copying and pasting the whole line with the ‘newline’ character into an R console.

Run this line, log into Twitter, and get the authentication PIN. If you get an error message from Twitter saying the URL is expired, the probable cause is that you hit before entering the PIN, possibly when you copied and pasted the command.

It is important to name the result auth.vector as shown above, as the functions in the twinfoR package look for this global variable. If you give this result a different name, you must manually supply it to the Twitter API methods. Optionally, save this vector in order to skip the authentication process in the future.

Creation of User data.frame

In practice, construction of the list of Twitter Users for collection and analysis would normally occur without using the functions in this package. In this case, however, we are going to analyze the content produced by US Politicians. Fortunately, we do not have to spend time manually looking up the Twitter Accounts of the individual members of Congress; the US political parties keep these accounts in Twitter lists. These lists might not be up-to-date, but we are going to assume they are accurate enough for our purposes. These lists are

This is a good opportunity to show the workhorse function of the twinfoR package: twitter_request. While typical use of this package will not require direct calls to twitter_request, this function is called by all of the functions provided by this package that access the Twitter API.

We want to get the members of an existing Twitter list. This package does not provide a specific method for that purpose, but there is a Twitter API endpoint that provides this service. Therefore, we use the general twitter_request function.

Note: to best understand the code below, you should read the documentation for the twitter_request function and the get-lists-members Twitter API endpoint documentation. At then end we should have four lists of Twitter user objects: house.dems, senate.dems, house.reps, senate.reps, as well as the official user account of the US President (@POTUS).

# Set the URL and count
url <- 'https://api.twitter.com/1.1/lists/members.json'
count <- 5000


# House Democrats
slug <- 'house-democrats'
owner_screen_name <- 'HouseDemocrats'
response <- twitter_request(
  auth.vector,
  url,
  query.param.list = list(
    slug = slug,
    owner_screen_name = owner_screen_name,
    count=count
  )
)
# Check for 200 status
if(response$status==200){
  house.dems.obj <- httr::content(response)
  house.dems <- house.dems.obj$users
} else {
  cat(paste("API request failed; status ",response$status,"\n",sep=""))
}


# Senate Democrats
slug <- 'SenateDemocrats'
owner_screen_name <- 'SenateDems'
response <- twitter_request(
  auth.vector,
  url,
  query.param.list = list(
    slug = slug,
    owner_screen_name = owner_screen_name,
    count=count
  )
)
if(response$status==200){
  senate.dems.obj <- httr::content(response)
  senate.dems <- senate.dems.obj$users
} else {
  cat(paste("API request failed; status ",response$status,"\n",sep=""))
}


# House Republicans
slug <- 'house-republicans'
owner_screen_name <- 'houseGOP'
response <- twitter_request(
  auth.vector,
  url,
  query.param.list = list(
    slug = slug,
    owner_screen_name = owner_screen_name,
    count=count
  )
)
if(response$status==200){
  house.reps.obj <- httr::content(response)
  house.reps <- house.reps.obj$users
} else {
  cat(paste("API request failed; status ",response$status,"\n",sep=""))
}


# Senate Republicans
slug <- 'SenateRepublicans'
owner_screen_name <- 'SenateGOP'
response <- twitter_request(
  auth.vector,
  url,
  query.param.list = list(
    slug = slug,
    owner_screen_name = owner_screen_name,
    count=count
  )
)

if(response$status==200){
  senate.reps.obj <- httr::content(response)
  senate.reps <- senate.reps.obj$users
} else {
  cat(paste("API request failed; status ",response$status,"\n",sep=""))
}


# President
pres <- user_show('potus')

Sampling

If you want this script to run in a short amount of time, take a small sample from the results. Sampling three from each set collected, and including the President (13 politicians in all) will keep the total collection time to about an hour.

If you want to reduce collection time, sample using code similar to the following

Now that we have the finalized lists of politicians we are going to collect, we can put them into a query_user data.frame structured as described in the twitter_database function documentation. Then, we add this table to the Twitter database connection (con).

While we could upload the Twitter user objects we already collected, we’ll just use the update_users function to go through the query_users table in the database and collect each user profile from the Twitter API to insert into the user table of the database.

Get Relationships

Now that we have initialized a Twitter database with a list of US Politician Twitter accounts, we can collect their connections. Because politicians often have many followers, we are not going to bother collecting these. Rather, we are interested in whom the politicians are following, as each of these ‘friends’ represents a concious choice on the part of the politician.

Note: this code takes a long time to execute. Expect a 15-20 hour run time on the full U.S. Congress. If you only kept a small sample, this time will be considerably shorter (typically 1.5-2 minutes per user). The prolonged collection times result from query limits imposed by Twitter on the REST API. The functions in this package include delays between queries to ensure these rate limits are not exceeded. However, attempts to run multiple collection functions in parallel processes using the same authentication credentials runs a high risk of violating the API rate limits, which will result in HTTP errors.

# THIS TAKES A LONG TIME (15-20 hours) ON THE FULL SET.  You have been warned.
get_all_friends(con)

Get Statuses

Next, we collect all of the posts from each politician, going back to up to 3200 Tweets. As in the previous step, this process can take a long time. During collection, we will perform sentiment analysis on the text of each Tweet (note: this is not done by default, so the parameters must be set as shown below). As a result of the sentiment analysis processing time and the Twitter API rate limits, expect this collection to take several days on the full data set. Once all of the statuses have been collected, future updates will take much less time as they will only collect new Twitter posts that have not yet been collected.

Setting up a sink to the NULL device prevents package RSentiment from outputing each status text to the screen. For large collections, this is recommended.

# THIS TAKES A LONG TIME (2-3 days) ON THE FULL SET.  You have been warned.

# sink("/dev/null") # Recommended to suppress output, but might not work
## on all platforms
update_user_timelines(
  con,
  calc.Rsentiment = TRUE,
  calc.syu = TRUE
)
# sink() # Close the sink to /dev/null

Summarize the database

To wrap up part I, we can view a summary of what is in our Twitter database. Note that many Tweets and users have been collected due to embedded retweets.

summarize_database(con)
#> Query users: 558
#> User categories:
#> party
#> body
#> since_id
#> friends_collected
#> followers_collected
#> No query text table.
#> Users: 579774
#> Statuses: 1599290
#> Unique user mentions: 123710
#> Unique hashtags: 81459
#> Unique URLs: 521248
#> Total Images: 532138
#> Unique Images: 497016
#> Total relationships: 1095918
#> Database file size: 1390MB

Finally, to conclude this part of the vignette, we close the data connection.

DBI::dbDisconnect(con)