Tweet sentiments during Boxing Day matches

If you ask me, Boxing Day is the best of Christmas holidays. On Boxing Day my local club plays in a traditional charity match, giving the fan of Finnish football a small glimpse of things to come during the world's longest (painfully so) pre-season. But besides that, some less interesting action goes around in the one of the most revered institution of English football, the traditional Boxing Day match round when all English Premier League sides are in action. This is a prime time to analyze Twitter feed to see whether sentiment analysis techniques can catch signal of the score of the game.

For this, we broadly need three components. First is of course access to Twitter API and the ability to collect data from it. For this, we use excellent TwitteR package. Here is one tutorial on how to set it up.

Second, we want an algorithm for estimating expressed sentiment. Since the R's sentiment package which did all this for us is apparently now defunct, we can either access one of the many sentiment analysis APIs on the Internet or create our own classification algorithm. For this purpose, I will loan the ideas of Jeffrey Breen and an opinion lexicon from Hu and Liu (also used by Jeffrey). We don't concern ourselves so much with the quality of individual classifications but the overall sentiment expressed in the tweets. For this we will calculate a positivity index.

Thirdly, some datetime functions to limit the tweets to be analysed to a pre-defined time frame is required. For this example I select only the tweets tweeted by official accounts of EPL clubs during the match itself and two hours afterwards. (the reason to download only official tweets from clubs is the fact that keyword -based search produces far too noisy data as supporters of a club, those likely to feel based on the scoreline, are hard to filter from the great mass of tweets. Although, as we shall see, official tweets of EPL clubs contain about as much information as interviews of managers)

We will mash all this functionality together to define a single function that is easy to call for different clubs. Functions are in R what macros are in SAS, allowing to run same batch of code over and over again dynamically by changing few parameters. Of course, R functions are more than that and all in all more flexible than SAS macros but let's not get into that on any deeper level.

Ok, lets begin! Again we start with loading the required libraries.

library(twitteR)
library(stringr)
library(dplyr)
library(lubridate)

Twitter is to access the API of, well you know. Stringr is mainly needed for one function in order to dissect tweets into individual words to be matched to opinion lexicons. Dplyr we can't live without and lubridate provides the function to work with datetime columns.


The function we define is as follows. We will dissect that beast to smaller sections below:

tweetsentiment <- function(user, n, dateday, datemonth, hourstart, hourend, minutestart, minuteend) {

 
tweets <- userTimeline(user,n=n)
tweetdf <- twListToDF(tweets)
tweetdf <- filter(tweetdf, month(created) == datemonth)
tweetdf <- filter(tweetdf, day(created) == dateday)
tweetdf <- filter(tweetdf, hour(created) >= hourstart &hour(created) <= hourend)
tweetdf <- filter(tweetdf, !(hour(created) == hourstart & minute(created) < minutestart)  
                  & !(hour(created) == hourend & minute(created) > minuteend)  )



tweetdf$text = gsub("[[:punct:]]", "", tweetdf$text)
tweetdf$text = gsub("[[:digit:]]", "", tweetdf$text)

tweetdf$text <-  tolower(tweetdf$text)


pos <- scan('C:/data/positive-words.txt',
            what='character', comment.char=';')
neg <- scan('C:/data/negative-words.txt',
            what='character', comment.char=';')

tweetdf$score <- 0
for(i in 1:length(tweetdf$text)) {
 
  words = str_split(tweetdf$text[i], '\\s+')
  words = unlist(words)
  pos.matches = match(words, pos)
  neg.matches = match(words, neg)
  pos.matches = !is.na(pos.matches)
  neg.matches = !is.na(neg.matches)
  tweetdf$score[i] = sum(pos.matches) - sum(neg.matches)
}

tweetdf$class[tweetdf$score < 0] <- "negative"
tweetdf$class[tweetdf$score == 0] <- "neutral"
tweetdf$class[tweetdf$score > 0] <- "positive"


x <- table(tweetdf$class)
x <- as.matrix(x)
y <- (x[3,]-x[1,])/colSums(x)
returnlist <- list(y, tweetdf)
return(returnlist)
}

Ok, lets start diving into individual pieces:

tweetsentiment <- function(user, n, dateday, datemonth, hourstart, hourend, minutestart, minuteend) {

The code starts with a function definition ending with curly brackets. Here we define the name and the parameters of the function. At the end of the function we obviously need to close the curly brackets.

tweets <- userTimeline(user,n=n)
tweetdf <- twListToDF(tweets)
tweetdf <- filter(tweetdf, month(created) == datemonth)
tweetdf <- filter(tweetdf, day(created) == dateday)
tweetdf <- filter(tweetdf, hour(created) >= hourstart &hour(created) <= hourend)
tweetdf <- filter(tweetdf, !(hour(created) == hourstart & minute(created) < minutestart)  
                  & !(hour(created) == hourend & minute(created) > minuteend)  )

In this portion we access the Twitter API with userTimeline function, feeding it parameters user and n in the function call. Then we create a dataframe out of the resulting object.

Next, we use a bunch of filter-statements from dplyr to narrow down the resulting time frame, using the datetime parameters specified in the function call. We begin by selecting the correct month, then day and then hours. Finally we exclude the rows based on starting and ending minutes knowledge. Lubridate functions are conveniently named and it is easy to figure which datetime unit specific functions return.

tweetdf$text = gsub("[[:punct:]]", "", tweetdf$text)
tweetdf$text = gsub("[[:digit:]]", "", tweetdf$text)

tweetdf$text <-  tolower(tweetdf$text)


pos <- scan('C:/data/positive-words.txt',
            what='character', comment.char=';')
neg <- scan('C:/data/negative-words.txt',
            what='character', comment.char=';')

tweetdf$score <- 0
for(i in 1:length(tweetdf$text)) {
 
  words = str_split(tweetdf$text[i], '\\s+')
  words = unlist(words)
  pos.matches = match(words, pos)
  neg.matches = match(words, neg)
  pos.matches = !is.na(pos.matches)
  neg.matches = !is.na(neg.matches)
  tweetdf$score[i] = sum(pos.matches) - sum(neg.matches)
}

This portion is much the same as in Jeffrey's example. Text-field is slightly cleaned, opinion lexicons are loaded and sentiment score is calculated by matching words used in the tweets to two opinion lexicons. To bring atleast some of my own flair, I did the scoring via a for-loop.

tweetdf$class[tweetdf$score < 0] <- "negative"
tweetdf$class[tweetdf$score == 0] <- "neutral"
tweetdf$class[tweetdf$score > 0] <- "positive"


x <- table(tweetdf$class)
x <- as.matrix(x)
y <- (x[3,]-x[1,])/colSums(x)
returnlist <- list(y, tweetdf)
return(returnlist)
}

In the final portion we first classify any sentiment scores below 0 as negative, above 0 as positive and those exactly 0 as neutral. This is done via the somewhat unintuitive base R syntax.

Next we calculate an index which I call the "positivity index". It is one simple answer to the problem of presenting classified data in a simple, numeric format to gauge at the general feeling. It can get values from -1 (all tweets classified as "negative" to 1 (all tweets classified as "positive"), with 0 possibly meaning both "all tweets classified as neutral" or "amount of tweets classified as negative equals the amount of tweets classified as positive". The index has a tendency to hover near zero just because of the latter quality but it still serves a purpose of quantifying classification data.

Without explicitly returning the dataframe or other resulting temporary help objects the function works as a "black box", with all the resulting objects vanishing into the thin air. This is not always desirable as we might want to check out the raw data or processed data from different phases of the function, especially in testing phase. In this case we might want to see individual examples of classification, to gauge at the quality of our algorithm.

So, I want to return the dataframe with tweets classified into either positive, neutral or negative. But I also want the positivity index already calculated for me inside the function. This means that I want to return two objects. In R, if we want to return more than one object, we need to use a list. This is why we save the positivity index to vector y and save that along with tweetdf to returnlist, which we pass to return-function. This makes the function to return a list, with first slot occupied by the positivity index and second the data frame with all the tweets classified. When calling the function, we should want to save the results to an object as well.

Ok, now we have a new toy to play with. Let's try it out! We pass to the function call the parameters user, the amount of tweets we collect, and the date-parameters to limit the final dataframe to be analysed to tweets made during and soon after the match.

results <- tweetsentiment("@arsenal",500,26,12,19,23,45,45)

results2 <- tweetsentiment("@southamptonfc",500,26,12,19,23,45,45)

We begin with the most surprising result of the match day, Southampton's 4-0 trashing of title contenders Arsenal. As a team looking to leapfrog Leicester to pole position, having your hopes crushed by such a rubbing must result in some pretty negative tweets, right?

results[1] 

0.1428571 

results2[1]

0.2909091 

Nooot exactly. Arsenal's official twitter account receives a positivity index of 0.14, as opposed to 0.29 of Southampton's. Southampton's official tweeter is obviously more thrilled but still Arsenal manages to be surprisingly positive. This means that our very simple and naive sentiment classifier doesn't quite work up to 100 % accuracy and that the official Twitter accounts are not exactly the forum to vent the frustration.


Lets try it out the second time:

results <- tweetsentiment("@manutd",500,26,12,12,16,45,45)

results2 <- tweetsentiment("@stokecity",500,26,12,12,16,45,45) 

Manchester United slumped to another defeat at the hands of Stoke City, resulting in the most successive defeats in over 40 years. This put Louis van Gaal's position as the manager on a very insecure footing and sacking can be thought to be coming in the very near future. Manchester United should then omit little positive signals while Stoke should celebrate the defeat of the fallen giant of English football. 

results[1] 
0.03703704 

results2[1] 
0.1935484

 Again, we notice a clear difference in sentiments. Manchester United is overall nearly neutral while Stoke's positivity index is a solid 0.19. Not quite the ecstasy exhibited by Southampton but pretty high still.

Overall, based on this small sample, we can be reasonably satisfied with our effort. Our naive classifier seems to find a signal amid all the noise and meaningless football jargon. And our function possessess enough flexibility to allow quick analysis of match and post-match tweets by certain users. And it would be reasonably simple to add another parameter to the function to control an if-else statement, which could then be used to control whether we want the download the tweets of specific user or search for a specific keyword.

Deducing US presidential candidates' core demography with Facebook activists first names

In this blog post we will first gather data from Facebook and use that data to perform a simple comparison wordcloud. The end goal is to make simple inference on the demography of American presidential candidates support base. Names are not just words but they also carry information. Most obviously you can deduce the person's gender with reasonable confidence and his/her ethnicity with tolerable accuracy. It also carries a highly unreliable indication of it's carrier's age (some names are more common in certain era than others).

First, we begin by loading the required R libraries.

library(Rfacebook)
library(dplyr)
library(wordcloud)
library(tm)
library(RColorBrewer)

Rfacebook is used to access Facebook API and you can read a tutorial on setting it up for example here. Dplyr is the must-have package for data wrangling, making the horrendously unintuitive subsetting tasks of base R much more bearable and also introducing the neat pipe operator %>% which connects numerous operations called on data together, making your R code cleaner and more efficient.

Wordcloud is used for visualizing the, well, final wordcloud and RColorBrewer to give the plot a bit nicer visual look (as you can see from this blog I may not the be the most visually competent person). Tm is a package used for text mining and is used to create the term matrix to feed the comparison.cloud-function. Ok, let's begin!

load("fb_oauth") 

pages <- c("DonaldTrump","hillaryclinton") 

df <- data.frame()
for (i in 1:length(pages)) {
  temp <- getPage(pages[i], token=fb_oauth, n=30) 
  temp2 <- temp$id 
  helpframe <- data.frame()
 
  for (j in 1:length(temp2))  {
    post <- getPost(temp2[j],token=fb_oauth)
    id <- post$likes$from_id
    name <- post$likes$from_name
    likesdf <- data.frame(name,id,stringsAsFactors=FALSE)
    likes <- likesdf$likesname
    name <- post$comments$from_name
    id <- post$comments$from_id
    comsdf <- data.frame(name, id,stringsAsFactors=FALSE)
    alldf <- rbind(likesdf, comsdf)
    helpframe <- rbind(helpframe, alldf,stringsAsFactors=FALSE)
  }
  helpframe <- unique(helpframe)
  fullname <- helpframe$name
  candidate <- pages[i]
  candidate <- rep(candidate,length(fullname))
  helpdf <- data.frame(fullname, candidate,stringsAsFactors=FALSE)
  df <- rbind(df,helpdf,stringsAsFactors=FALSE)
}

Let's look a little closer what that code does. For preparations, first we load up the fb-authentication token I have saved earlier, as in the RFacebook-tutorial. Then we save the public Facebook pages I am interested in a vector called pages. I limited this blogpost to just the Hillary Clinton (who's deadset to win the democratic primary) and Donald Trump (who despite his strong showing in the polls remains a long-shot to actually win the Republican race) Finally we initialize the empty dataframe df for storing the data.

What follows is a nested loop. We use getPage-function from RFacebook to get page data for ith member of earlier initialized pages vector using the fb_oauth. N limits the amount of posts we collect from the page so in this example we get 30 latest posts from each page. We save this all in a dataframe called temp.

However, what we are really interested in is the details of the likes and comments found under the posts published on the page. For that we collect post id's from results of getPage and store them in temp2, also initializing helpframe for storing data in the phase. The second loop uses temp2 as input for getPost-function, much like pages was for getPage. What follows is getting the userID and username for likes and comments respectively. The rest is all about creating a succession of data frames to eventually collect all the userIDs and usernames from all the posts stored in temp2 to helpframe. After all likers and commenters under the page posts are collected, we leave the inner loop and go back to outer loop.

Here we use unique-function to select distinct username-userID pairs (the reason we collected id in the first place, calling unique on username alone might have dropped few unintentional doubles). After unique we no longer have any use for the id-variable so we select only the variable username to a vector called fullname. To accompany name information in our final dataframe, we create the variable candidate to give us a clue which presidential nominee's post said person had liked/commented on Facebook. We do this by using the page name from our pages vector and repeating it for the number of unique persons we collected in the inner loop. Finally, we store these two vectors in helpdf which we combine with our initially empty df to collect the data from all the iterations of the loop in a single data frame.

df$firstname <- word(df$fullname)


vectorofvectors <- list()
for (i in 1:length(pages)) {
 vector  <- df  %>% filter(grepl(pages[i],candidate))  %>% select(firstname)
 vector <- vector$firstname
  vectorofvectors[[i]] <- vector
  }


mycorpus <- Corpus(VectorSource(vectorofvectors))
mytdm <- TermDocumentMatrix(mycorpus)
mymatrix <- as.matrix(mytdm)
colnames(mymatrix) <- pages
comparison.cloud(mymatrix,colors=brewer.pal(6,"Dark2"),title.size=1,random.order = FALSE)

Now we have finally gathered the data from Facebook using a little nested for-loop. Next we use word-function to split the first name from the full name (we make the assumption that people usually put their first name, well, first in Facebook) and save it as third column of the dataframe df.

What we do next felt somewhat unintuitive and there must be a cleaner way to accomplish the task. Nevertheless, the idea behind the code was following:

First we initialize an empty list vectorofvectors to store name-data in vector format. This is because I want to use corpus-related functions from the tm package and want as many documents in my corpus as there are presidential nominees in pages. We use a little for-loop filter the rows where value of candidate equals the i:th value in pages. %>% - pipe operator is used to connect two operations, filter and select, from dplyr-package together in a nice and clean way, used on dataframe df. We turn the resulting one-column dataframe to more appropriate vector and store that in the list at i:th slot.

Finally we read the list into a corpus, turn the corpus to a term-document-matrix, and the term-document-matrix to a normal matrix. Before finally calling the comparison.cloud-function to visualize differences in the first names of the selected presidential nominees, we name the matrix columns to know which portion of the cloud belongs to which nominee. In the comparison.cloud-call, note the colors=brewer.pal-portion in which use colour palette from RColorBrewer to give some nicer visuals to the resulting plot.

And finally, we can enjoy the fruits of our labour:

Two things spring immediately to mind from the comparison cloud. First we notice that the Trump portion of the cloud is more crowded than the Clinton part. Trump did lead in the number of unique likers/commenters in our sample by almost 4000 people, 20 990 to Hillary's 17 151. But we could also deduce that the variability in the names of Trump supporters is smaller and thus they get a proportionally bigger share of the resulting pie, which holds only so many names (cloud collects names from the entire matrix, not for each column individually).

Secondly we see that Trump supporters seem to possess mostly male, Anglo-sounding names (John, James, Michael) whereas Hillary's supporters names are more feminine and ethnically diverse (Sarah, Jose, Mohammed).

I'd reckon most surveys would find the demographics of two nominee's supporters broadly in line with the above, simplistic deduction.

The purpose of this blog

Hello and welcome to Time is Data, yet another R/data-blog.

I believe that if I am to embark on a project, there should be both some sort of a goal and a guiding principle to keep me focused - there has been more than a few aimlessly joined MOOC courses abandoned after only couple videos viewed. It turned out that learning new skills without a specific purpose in mind is an act of futility. Therefore Time is data is born, to provide the right incentives.

Or more precisely, that was how Aika on dataa was originally born (that's Finnish, not Elvish by the way!). It was a way to motivate me to properly get into R programming, something that I thought to be highly useful in long-term but which had no immediate use in my position at Statistics Finland. And to a varying degree, it has served it's purpose. After a couple months, I still find myself toying with R for my general amusement after work.

Goal of this blog is similar to that of her Finnish sibling/precessedor; to motivate me in pursuit of improved R programming skills, and to a lesser degree to show off my skills for the world to see, also called networking by some HR specialists. However, the guiding principle will slightly differ. In this English version of the blog I will take a more technical and programmatic focus, showing and explaining R code, putting less time on the actual substance and the data itself. My Finnish blog will then cut back on technical details and focus more on presenting the insights derived from the analysis and the theoretical background that made embarking on the specific analysis interesting in the first place.

A few words on the author, some of which might have been inferrable from text above. I received a social scientific education in university, majoring in political sciences. However already during my studies I delved much more deeply to the dark (quantitative) side than most of my peers, starting and almost finishing a Bachelors in Statistics. Even before graduation my career was set to be starkly data-centred and this is the road I have decided to passionately follow, currently finding myself at Statistics Finland. However, my social scientific background has left a deep mark in me as I always want data to tell me a story, hopefully backing a theory. My usage of data is thus highly practical and my programming somewhat quixotic, due to being for a large part self-taught with only couple computer sciences courses behind me.

A lot has been said about what I desire this blog to accomplish for me, less about what I hope it provides to the occasional reader. Internet is filled with brilliant R tutorials and blogs and I likely won't be able to compete with the best of them with the quality of my programming. This won't be a blog by a programmer for the programmers. However, I hope to bring another hands-on resource with the code examples hopefully easily digestable. And naturally present few interesting and motivating themes to apply the skills shown here!

The name of the blog is obviously a play on the proverb found in many languages. It was more phonetically pleasing in Finnish were "dataa" and "rahaa" (money) are more similarly pronunced.