America First 40s style: using NLP methods to analyse Lindberg Des Moines Speech

I recently read Plot Against America, a terrific alternate history novel written by Philip Roth. It charts the fate of a Jewish family in USA in which Charles Lindbergh runs for presidency in 1940 and defeats FDR on America First platform. Inspired by the book, and the fantastic cleanNLP package I learned about in useR 2017 conference in Brussels, I decided to run a small text analysis on Lindberghs Des Moines speech.

What makes cleanNLP such a great package is mainly how it tidies raw text data to nice data frames. It has backends to both CoreNLP package in Java and spaCy package in Python so in doing the tidying of data it harnesses state-of-the-art dependency parsers, POS taggers and entity recognizers built in aforementioned libraries. Thus, with a couple simple lines of code, you are presented with a data frame containing not only tokens but various useful objects you can take advantage of.

You can read (and listen to) the Des Moines speech here, For my purposes, I simply copied and pasted the speech to a simple txt-file but you could as well use rvest or other web scraping tools to get the speech right of the website.

library(cleanNLP)
library(readr)
library(wordcloud)
library(dplyr)
library(RColorBrewer)
library(syuzhet)
library(ggplot2)

init_spaCy()

desMoines <- read_file("C:/desMoines.txt")

anno <- run_annotators(desMoines, as_strings = TRUE)
df <- get_combine(anno)

Code above shows the steps taken to get the tokenised data frame from one wall of text. I chose the SpaCy backend which is initialized in a simple step. Then run_annotators tokenises the text using the spaCy library. Finally the various data frame objects created by the function are put together with the get_combine. Text is tidied and we are ready to start analysis!

I begin by looking at what countries and nationalities get mentioned in the speech. Mind you, the speech is held in 1941 when WW2 is in full speed and Pearl Harbor is not too far away. With a tidy data frame containing a column for recognized entities, it is simple to apply so-called tidyverse grammar from my favourite dplyr package to check this out.

df %>% filter(entity_type %in% c("GPE","NORP")) %>% select(entity) %>% group_by(entity) %>% summarise(N=n()) %>% arrange(desc(N))

              entity     N
               <chr> <int>
1           American    13
2            America    10
3            British     9
4             Jewish     9
5            England     8
6  the United States     7
7           European     6
8            Germany     5
9             France     3
10     Great Britain     2
11        Washington     2
12            Allies     1
13         Americans     1
14       Anglophiles     1
15              Axis     1
16              Iowa     1
17              ours     1

Up there with the obvious America are the would-be allies British are the time's trendy scapegoats the Jews. Germany, which was wrecking havoc across Europe was mentioned five times and already surrendered France 3 times. The Pacific front and the Japanese get no mention at all. From the list it can be seen that this was a speech concerned keeping USA out of war and war out of USA.

Another way to analyse the speech is look at dependencies, that is which words accompany others. Good thing we have that spaCy dependency parser at hand! Here I look at direct object dependencies.

df %>% filter(relation=="dobj") %>% mutate(wordPair = paste(word, "-->", word_source)) %>% select(wordPair) %>%  group_by(wordPair) %>% summarise(N=n() ) %>% arrange(desc(N)) %>% print(n=142)

Among the direct object dependencies are found the following telling word pairs: intervention --> oppose and Jewish --> attacking and the most frequent dependency appearing total of 4 times, us --> involve.

Ok, lets do something traditional next; the classic text miners servant the wordcloud. Using part-of-speech tags produced by spaCy to filter the data frame allows us to use a kind of enlarged version of stopword lists, focusing only on the interesting words that carry most information, that is nouns, proper nouns, verbs and adjectives but discounting pronouns.

cloudWords <- df %>% filter(upos %in% c("NOUN","PROPN", "VERB","ADJ") & pos != "PRP$") %>% select(lemma) %>% unlist()

wordcloud(cloudWords, max.words = 100, colors=brewer.pal(6,"Dark2"))

Finally, I will track the progression of the speech' sentiment. That is accomplished by aggregating the data frame from levels of tokens (individual words) to sentences and then applying a sentiment analysis algorithm from suyzhet package. Finally to inspect slightly larger chunks than individual sentences, the sentiment scores are aggregated to the levels of chunks of 5 sentences.

sents <- df %>% group_by(sid) %>% summarise(lause=paste(lemma, collapse=" "))

sentiments <- get_sentiment(sents$lause, method="nrc")
rk <- rep(1:26, each=5)

sentiDF <- data.frame(rk, sentiments) %>% group_by(rk) %>% summarise(MeanSent=mean(sentiments))

ggplot(data=sentiDF, aes(y=MeanSent, x=rk)) + geom_line()
 

As you can see as far as sentiment score in positive/negative scale goes, the speech does not follow a clear pattern but rather positive peaks are followed by negative slumps. The longest consistently positively classified part of the speech happens around the tenth chunk. This is the part of the speech where first talks about British attempts to draw America into war, followed by talk about the Jews and the most humane sentence of speech "No person with a sense of the dignity of mankind can condone the persecution of the Jewish race in Germany".

CleanNLP, enhanced by the powerful spaCy library, is a great and easy-to-use addition to the R text analysis universe. The great part about the package is that rather than work in abstact corpus objects like the traditional tm package, it crunches the text data to the data frame form we all R users are so familiar with. This allows one to conduct quite elaborate text analysis tasks without knowing much about text analytics per se.