Analyzing the relationship of Twitter users towards brands (e. g. Air Berlin)

Social media platforms such as Twitter and Facebook enable everyone to voice their opinions about topics, companies, and products online.

These comments are a great source for companies to analyze their customers’ opinion about their brand or product. However, with billions of Tweets and posts daily, this is can take a lot of time.

Unless of course, you use R J With just a few lines of R-code and the help of machine learning, we’re able to build mood monitoring tools quickly, so that the public opinion about your or anyone’s company can be monitored and evaluated.

Opinions on a brand over time

The aim of our script is to analyze the opinions about individual companies over a longer period of time. In order to get an “opinion” we decided to analyze the mood on Twitter. The airline AirBerlin, which had to file for bankruptcy only a few weeks ago, is a good example.
So we have our channel: Twitter and our brand: AirBerlin. Now it’s time to collect their users’ opinions

The problem with the Twitter API

Twitter offers a free API that return on a specific topic (and other filtering options) returned to them. In addition, there is a very good package called “TwitteR” for this API, which makes querying the data extremely easy. It is only necessary to create an API project.

Unfortunately, the Twitter API has a disadvantage, which is that we only get data from the last seven days. The API is therefore not suitable for long-term monitoring. The Twitter interface, on the other hand, gives us all Tweets. They can be reached through the “Advanced Search” on Twitter, which can be found in the right sidebar of the Twitter-SERPs. The options for finding tweets are almost unlimited.

In addition, the URL of the search results filtered by Advanced Search contains the individual filter settings. That makes crawling the results very easy:

https://twitter.com/search?l={{LANGUAGE}}&q={{SEARCH_TERM}}20since%{{START_DATE}}until%3A{{END_DATE}}&src=typd

Building the Selenium Crawler

To get to the Tweets, we had to build a crawler that calls the corresponding URL and saves the Tweet. In order to obtain a broad range of opinions, we randomly query 50 selected Tweets every week since 2013.

driver<- rsDriver(browser = c("firefox"))
remDr <- driver[["client"]]
remDr$navigate("https://twitter.com")
#Log IN
#Get Tweets
out<-capture.output(for (i in 1:nrow(Start_End)){
page<-paste0("https://twitter.com/search?l=de&q=@airberlin%20since%3A",Start_End$Start_date[i],"%20until%3A",Start_End$End_Date[i],"&src=typd")
remDr$navigate(page)
Sys.sleep(5)
try(webElem <- remDr$findElements(using = "css", '.tweet-text'))
for (a in 1:20){
w<-try(webElem[[a]])
messages[[a]]<-try(w$getElementText())}
m<-as.data.frame(do.call(rbind, messages))
print(m)
Sys.sleep(5)
})

score<-NULL;
score<-capture.output(for (i in 1:5000){
q<-Tweets$X1[i]
request_body <- data.frame(
language = c("de","de"),
id = c("1","2"),
text = out[[q]])request_body_json <- toJSON(list(documents = request_body), auto_unbox = TRUE)
result <- POST("https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/sentiment",
body = request_body_json,
add_headers(.headers = c("Content-Type"="application/json","Ocp-Apim-Subscription-Key"="YOUR_KEY")))
Output <- content(result)
print(Output$documents[[1]]$score)
Sys.sleep(0.5)
})