\documentclass[11pt]{article}
\textheight 9.5in
\textwidth 6.6in
\parskip=\medskipamount
\parindent=0.2in
\oddsidemargin -.1in
\evensidemargin -.1in
\topmargin -.4in
\headsep 0in
\topsep 0in
\itemsep 3ex

\usepackage{tikz}
\usepackage{hyperref}

\newcommand{\blank}{\mbox{\_\_\_\_\_\_\_\_\_\_\_\_}}
\newcommand{\rma}{{\rm a}}
\newcommand{\rmb}{{\rm b}}
\newcommand{\rmma}{{\rm\bf a}}
\newcommand{\rmmb}{{\rm\bf b}}
\newcommand{\set}[1]{\{\,#1\,\}}
\newcommand{\mdot}{\!\cdot\!}
\newcommand{\AND}{\; \wedge \;}
\newcommand{\OR}{\; \vee \;}
\newcommand{\intersect}{\cap}
\newcommand{\implies}{\Longrightarrow}
\newcommand{\symmdiff}{\bigtriangleup}

%% KWR: Added macros for boxes.
\newsavebox{\fmbox}

\newenvironment{fmquote}[1] % Centers within the box, best used for
{\vspace{1ex}\begin{center}\begin{lrbox}{\fmbox} % one-line quotes
 \begin{minipage}{#1}\centering\vspace{0.5ex}}
{\vspace{0.2ex}\end{minipage}\end{lrbox}\fbox{\usebox{\fmbox}}
 \end{center}\vspace{1ex}}

\newenvironment{flquote}[1] % Multi-line quotes with ragged right
{\vspace{1ex}\begin{center}\begin{lrbox}{\fmbox}
 \begin{minipage}{#1}\vspace{0.5ex}}
{\vspace{0.2ex}\end{minipage}\end{lrbox}\fbox{\usebox{\fmbox}}
 \end{center}\vspace{1ex}}

\begin{document}
\sloppy
\pagestyle{empty}
%\noindent
%{\bf Name and St.ID\#:}\blank\blank\blank\blank\blank\blank\blank\blank 
%
%\bigskip
\noindent
{\Large\bf
\begin{tabular*}{\textwidth}
{@{}l@{\extracolsep{\fill}}c@{\extracolsep{\fill}}r@{}}
CSE199 & Activity: Textual Sentiment Analysis & Fall 2024
\end{tabular*}
}


\noindent
This week's activity is a `get-acquainted' experience with: using Python, data-mining the web, sentiment analysis, and numerical interpretation of results.  No past programming experience in Python is needed---the program is given and runs within (the free level of) a web app called \href{https://trinket.io/features/python3}{Python 3 Trinkets}.  Besides having fun, the goal is to raise discussion of data policy issues pointing toward the third week's mini-project.  The discussion includes what it means to build and train a \emph{predictive model}.  This activity needs {\bf Chrome} or {\bf Firefox} or a downloaded Python system; the two browsers seem to work on any platform. 

%%%%%%%%%%%%%%%%%
\section{Introduction}

Sentiment analysis means evaluating the emotions and possible intents of the authors of textual material.  Here are two mainstream applications:

\begin{itemize}
\item
Filtering communications and webpages for terrorist indications and gauges of threat levels.
\item
Evaluating customer (dis-)satisfaction with your company's products---and possibly diagnosing faults.
\end{itemize}

The presence of emotion distinguishes this from other textual modeling, such as the \href{http://www.philocomp.net/humanities/signature.htm}{Signature} \emph{stylometry} system by which Peter Millican and co-workers identified the pseudonym Robert Galbraith as belonging to J.K.\ Rowling.  The particular emotions are five personified in the 2015 Pixar movie \href{https://en.wikipedia.org/wiki/Inside_Out_(2015_film)}{\em Inside Out\/}: \emph{joy}, \emph{anger}, \emph{fear}, \emph{disgust}, and \emph{sadness}.\footnote{%
%
This year's sequel \href{https://en.wikipedia.org/wiki/Inside_Out_2}{\em Inside Out 2\/} features five new emptions:  \emph{anxiety}, \emph{envy}, \emph{embarrassment}, \emph{ennui}, and \emph{nostalgia}.  Meanwhile, the creator of our tool and its training lexicon went in a different direction with \emph{anticipation}, \emph{trust}, and \emph{surprise}, and points toward \emph{optimism} and \emph{hope} at the top \href{http://saifmohammad.com/WebPages/ResearchInterests.html#EmotionAnalysis}{here}.  We will stay with the basic five for simplicity.
} %% end of footnote

We will use this tool to make simple measures of the ``emotional temperature'' of various webpages.  One component of our measure is scientific.  It is a human-scored (crowdsourced) \emph{rating} of a lexicon of words for \emph{intensity} in one or more of the five categories.  For instance, the word ``surprise'' scores {\bf 0.172} for \emph{fear} and {\bf 0.562} for \emph{joy}.  It scores zero for the other three categories.  The plural ``surprises'' scores only for \emph{joy}: {\bf 0.606}.  The intensity is always less than $1$; the most intense word for \emph{anger} is ``outraged'': {\bf 0.964}.  There are numerous NSFW terms in the lexicon---well, this activity is structured so that you won't see them unless you load a webpage that has them.  

The lexicon is copyrighted by Dr.\ Saif Mohammed and the Canadian National Research Council (NRC).  We are using an older version of his lexicon with permission from him and his co-worker Pierre Charron.  Although the current version is freely downloadable from the \href{http://saifmohammad.com/WebPages/AffectIntensity.htm}{project page}, it carries terms of use (bottom of page) that forbid \emph{re-distributing} it and were in effect for our version as well.  


%%%%%%%%%%%%%%%%%
\section{To Do Before Recitation}

For background, please \emph{skim-read} Dr.\ Mohammed's 10-page paper ``\href{http://saifmohammad.com/WebDocs/lrec2018-paper-word-emotion.pdf}{Word Affect Intensities}'' (2018). You need not understand all the technical bits but should appreciate the following points:

\begin{itemize}
\item
Even at professional level this kind of work is (and in 2024, still is) in early stages.  
\item
This was almost the first lexicon with numerical rather than binary (i.e. 0-1 or yes/no) data on words for general purposes.
\item
The numbers were computed (to three decimal places) not by whim but by some kind of regular scientific procedure based on examining large amounts of data and using ``cloud and crowd'' methods.
\item
The crowd-sourced numbers were checked against human annotators.  They were found to correlate significantly highly with an average of human opinions.  The correlation measures mentioned on page 2 are not the same as the ``$R^2$'' from the linear-model examples in lecture but are related.
\end{itemize}

\noindent
Then finally please skim-read the Python code files {\tt heatindex.py} and {\tt heatlib.py}, which can be found in my {\tt\verb+https://www.cse.buffalo.edu/~regan/cse199/+} \href{https://www.cse.buffalo.edu/~regan/cse199/}{folder}.  Note especially the lines with {\tt location =} in {\tt heatindex.py} and with {\tt pageStr} and uses of {\tt re.sub} in the {\tt heatScore} function in {\tt heatlib.py}, where there are alternatives to comment in or out.  They include an upgrade to version 3 of the Python URL library {\tt urllib3} and a switch in the order of lines in the lexicon from word-score-category to word-category-score.  These work in the 2024 current Python 3 Trinkets server under the directions given below.  You are optionally welcome to run your code files in your own Python 3 environment.  The Trinkets server has behaved well the past two years, but in case of glitches, the commented-out lines might possibly be of use.

In brief, what the code does is tally all the lexicon words that appear in a loaded webpage.  The score total of words in the \emph{joy} category is multiplied by $-1$.  The other categories each have multipliers of $+1$, which means their positive scores are just summed together.  Adding the scores gives a \emph{negative} result if there are more ``joy'' words than others; else a positive score 




%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Setup Directions}

The runs are relatively simple and will be demo'ed in the week 2 Wednesday lecture.  If using \href{https://trinket.io/python3}{https://trinket.io/python3} (or the \href{https://trinket.io/python3/e949bcacc5}{alternate Trinket page}):

\begin{enumerate}
\item
{\it Create a new window\/} in your browser to go there.
\item
Copy-and-paste {\tt heatindex.py} into ``main.py'' there.  There is no need to change the name {\tt main.py} and no file upload. 
\item
Then click the `{\tt +}' to add a new file, name it {\tt heatlib.py}, and copy-and-paste the text from your own download or browser view of {\tt heatlib.py}.
\item
Click the right-pointing triangle (standard ``play'' icon) to run.  You will be prompted to enter a URL.  With Trinket it is best to \emph{copy-and-paste} the URL from a webpage and hit Enter.  
\item
Output may take some seconds to appear---even 15 seconds in some 2024 runs.  The output dumps all the found words in their categories and gives the ``heat score'' at the end.  You can slide the bar between the code and output windows to see more of the final output lines.  
\item
The program prompts for another URL, but on Trinket this glitches over half the time, so please instead restart everything by clicking the ``play'' icon again.  
\end{enumerate}

The Trinket app also often freezes if it is left untouched for some length of time.  These glitches are not observed in a native Python environment (?), so it is hard to tell if improving {\tt heatindex.py} and {\tt heatlib.py} themselves would matter on Trinket.

Some webpages cause errors of two kinds:

\begin{itemize}
\item
The webpage refuses the ``GET'' request in a way that causes Trinket to report an error (the output pane turns pink and shows code trackbacks).  Restart and try another webpage.
\item
The webpage gives no output, or nondescript output.  Twitter/X pages now (2024) seem to return only the string {\tt x.com}.  The \emph{Wall Street Journal} currently gives a boiler-plate site index dump---which gives a score of {\tt -2.031} favoring {\tt joy} in the activity below but doesn't count.  In previous years, some pages returned the text of an error saying ``Something went wrong.  Don't fret---let's give it another shot'' and elsewhere, ``Terms of service'' and ``Access control disabled.''  This matched the NRC lexicon on words including ``wrong,'' ``fret,'' ``shot,'' ``service,'' and ``disabled,'' which at first seemed thematic for a news story covering a war.  If you suspect this, comment-in the line {\tt print(\"I read \",pageStr)} (around line 170 of {\tt heatlib.py}) to get a dump of the entire text received.  Or just restart and try another webpage.
\end{itemize}

\noindent
Basically the catchall for troubleshooting is ``just restart and try another webpage.''


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Activity Directions}

Get into your groups, but the first part is to be done {\bf individually} by everyone---for practice---but discussed briefly in groups.  It also furthers the discussion about caveats to look for when loading webpages.



\begin{enumerate}
\item
First, go to the new personal homepage \href{https://www.kimberlymarshall.com/}{https://www.kimberlymarshall.com/}of my friend Kimberley Marshall.  The score should be in the vicinity of {\bf -10}.  
Try the platform/issues pages of the two presidential campaigns: 

Once the setup is accomplished, the 



On a home Python3 system or the CSE machines the command options are:

\begin{verbatim}
python3 heatindex.py                or give a URL argument at the command line, say
python3 heatindex.py www.cnn.com
\end{verbatim}

Or from within the Python environment, enter {\tt from heatindex import *} on the first run, and if you quit the menu (but not Python) and want to restart, enter {\tt processUserInput(url,heatDict,mulDict)}. 

\begin{enumerate}
\item
Play around with a few webpages.  You can copy and paste URLs from the browser window.  Twitter pages will give you the person's last yea-many Tweets.
\item
Can you find a webpage with a negative score?  Negative means more `joy'---
\item
Stop---!  We need a ``coach's huddle'' before we get carried away\dots
\end{enumerate}

The most important thing to know about the numerical results at the very end is that they are {\bf not designed by Dr.\ Mohammad} but rather by me, KWR.  This project design performs a ``reduction'' on his work.  The final number comes from adding up the `anger,' `fear,' and `sadness' scores and \emph{subtracting} the total `joy' score.   Then my code divides by the total number of words read and---totally arbitrarily for better ``eye candy''---multiplies that by 1,000.  In spot-tests this puts the final numbers roughly on a familiar 0-to-10 scale, concentrated more near zero, and possibly negative---when `joy' outweighs the three other categories.  So it has some street sense.  But it's still doing ``push a button and get a number''---which is what everybody wants in order to make their jobs simpler but which comes with blinkers and dangers.

Put another way, I have slapped a layer of ``multipliers'' onto his model.  The default multipliers pass the ``Occam's Razor'' test of being simple: $+1$ each for \emph{anger}, \emph{disgust}, \emph{fear}, and \emph{sadness}, and $-1$ for \emph{joy}.  The code allows you to change them at will, but they are still \emph{arbitrary}---that is, not justified by theoretical considerations and/or empirical testing. They also represent some presumptions about \emph{what} we want to model:

\begin{itemize}
\item
They presume we are trying to model `light'-versus-`dark' in some way.  
\item
If we want to model \emph{intensity} of any kind, we should use $+1$ for \emph{joy} too.  
\item
At least the $+1$ multipliers are leaving Dr.\ Mohammad's carefully-crafted numbers alone\dots
\item
But adding them up and dividing by the total number of words is still ``reducing'' his work.  What is that ratio supposed to represent?  
\item
If we append to the bottom of a page an equal amount of completely neutral text, we will \emph{halve} the score.  Does this make sense---shouldn't we give more weight to text at the top?  
%Think of what ``TL-DR'' means.  (On the other hand, a Twitter sample might be best treated as uniformly weighted over time.)
\item
Perhaps `anger' should be regarded as generating more ``heat'' than `fear' (which is more passive) and certainly `sadness.'  (Indeed, those who have seen \emph{Inside Out} may recall its upshot about sadness.)  
\item
Try multipliers of $+10$ for anger, $+6$ for fear, $+2$ for sadness and (really ``winging it'' now) $-5$ for joy.  Repeat some earlier URLs.  Do any scores shift notably?  (Note: Just doubling the multipliers will not double the score, because the code has a further ``normalizing'' division by the sum of the absolute values of the multipliers.)
\item
Try other multipliers too.  Each combo constitutes a different \emph{model} of ``webpage intensity.'' Then reflect on the most important question:
\end{itemize}

\begin{flquote}{5in}
From the myriad parameter combinations, how can we determine {\bf one} that is \emph{correct}, \emph{best}, or at least \emph{neutrally justifiable}?  In particular, how could we \emph{train} the model?
\end{flquote}

Answering this question first requires determining what we want to model and what the purpose is.  Suppose we adopt the ``light-versus-dark'' purpose.  The first thing we might try is to identify a corpus of pages that represent the ``neutral'' middle.  Then we want to define our scale and train our multipliers so that those pages get a score of $0$.  This still does only a quarter of the work we need---in technical jargon, it leaves three ``degrees of freedom'' in the four parameters we are fitting.

Further progress needs asking, what are we trying to \emph{predict}?  Here we might come full circle to the ``threat levels'' application mentioned at the outset.  Suppose we have a corpus of pages that were reliably associated with the same threat level in past history.  Some pages might have more `anger,' others more `fear' and so on.  We can train our multipliers to equalize those pages.

A general methodology emerges from the idea that how morose or bubbly a webpage written today is can predict how morose or bubbly a page written by the same person(s) tomorrow will be.  Or we can apply this prediction idea from sentence to sentence or phrase to phrase.  Now we are in the world of those $N$-grams.  Predicting a probability range on the nature of the $(N+1)$st item from the previous $N$ in a continuing series involves building a so-called \emph{Markov model} (or \emph{Markov chain}, named for the Russian mathematician Andrey Andreyevich Markov).  The Markov model $M$ can use the principle that words continuing the score trend of the previous $N$ are most likely.  Its results thus depend on the scores and hence on the multipliers we choose.  The \href{https://en.wikipedia.org/wiki/Likelihood_principle}{likelihood principle} in this context yields the training policy:

\begin{flquote}{5in}
Find the combination of parameters that maximizes the probability that $M$ projects for the record of what actually happened.
\end{flquote}

This sounds like ``making yourself look good by predicting the past'' but the power of this principle carries into the future.  It still, however, takes quite a bit of work to build $M$ and then some computer muscle to carry out the \emph{maximum likelihood estimation} (MLE).




%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Discussion Points (including the NFL data too)}

If you think of Data Science as `sexy' then it has been my purpose to spend a lot of time metaphorically on safety and STDs---and even literally on matters of personal consent and privacy.  Instead of brilliant results with clear correlations, we've seen:

\begin{itemize}
\item
Inconclusive results from small data (when that is all the data that is);
\item
How to ``cheat'' with Cincinnati---or rather, how sensitive results can be to one data point;
\item
How quantities for which there is obvious and immediate demand are currently \emph{inchoate};
\item
How the desire for simplicity induces arbitrary choices that can be fraught with biases;
\item
How fixing these issues requires lots of attentive hard work.
\end{itemize}

Scoring is 3 for participation, 3 for completing the runs, and 3 for discussion.  This may be re-scaled from being out of 9 to being out of 3.

%\noindent
%Grading will be based on understanding shown, not just running a few Python commands.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%\section{More Fiddling Around (optional)}
%
%Note that the Python code is pretty `modular' not only in the library module {\tt heatlib.py} but even in the client file---it avoids referencing the four categories by name as much as possible.  Thus it is ready to handle updates of the lexicon to include other categories.  Try this yourself by adding the category \emph{disgust} and a dozen words like {\tt gross} and {\tt yuck} and {\tt nausea} right to the lexicon file, giving them scores at-your-whim.  Make the few edits needed to {\tt heatindex.py} and see the results.


% \medskip
% \centerline{\sc End of Exam.}
\end{document}


