\documentclass[11pt]{article}

\usepackage{classDM}
\usepackage{hyperref}
\usepackage[normalem]{ulem}

\title{Asmt 6: Regression}
\author{Turn in through Canvas by 2:45pm: \\
Wednesday, March 27 \\
100 points}
\date{}



\begin{document}
\maketitle

%\end{titlepage}




%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{Overview}

In this assignment you will explore regression techniques on high-dimensional data.  

You will use a few data sets for this assignment:
\begin{itemize} \denselist
\item \href{http://www.cs.utah.edu/~jeffp/teaching/cs5140/A6/X.dat}{\texttt{http://www.cs.utah.edu/\~{}jeffp/teaching/cs5140/A6/X.dat}}
\item \href{http://www.cs.utah.edu/~jeffp/teaching/cs5140/A6/Y.dat}{\texttt{http://www.cs.utah.edu/\~{}jeffp/teaching/cs5140/A6/Y.dat}}
\item \href{http://www.cs.utah.edu/~jeffp/teaching/cs5140/A6/M.dat}{\texttt{http://www.cs.utah.edu/\~{}jeffp/teaching/cs5140/A6/M.dat}}
\item \href{http://www.cs.utah.edu/~jeffp/teaching/cs5140/A6/W.dat}{\texttt{http://www.cs.utah.edu/\~{}jeffp/teaching/cs5140/A6/W.dat}}
\end{itemize}

These data sets are in matrix format and can be loaded into MATLAB or OCTAVE.  By calling 
\\
\texttt{load filename} (for instance \texttt{load X.dat})
\\
it will put in memory the data in the file, for instance in the above example the matrix \texttt{X}.  You can then display this matrix by typing 

\texttt{X}


For python, you can for instance use the following approach:

\includegraphics[width=.6\linewidth]{python-read.png}

\vspace{.1in}

\emph{As usual, it is highly recommended that you use LaTeX for this assignment.  If you do not, you may lose points if your assignment is difficult to read or hard to follow.  Find a sample form in this directory:
\url{http://www.cs.utah.edu/~jeffp/teaching/latex/}}



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Linear Regression (75 points)}

We will find coefficients \texttt{A} to estimate \texttt{X*A $\approx$ Y}, using the provided datasets \texttt{X} and \texttt{Y}.  We will compare two approaches \emph{least squares} and \emph{ridge regression}.  

\begin{itemize} \denselist
\item[\textsf{Least Squares:} ]  Set \texttt{A = inverse(X' * X)*X'*Y}
\item[\textsf{Ridge Regression:} ] Set \texttt{As = inverse(X'*X + s\^{}2*eye(15))*X'*Y}
\end{itemize}

\paragraph{A (30 points): }
Solve for the coefficients \texttt{A} (or \texttt{As}) using Least Squares and Ridge Regression with $s = \{1, 5, 10, 15, 20, 25, 30\}$ (i.e. $s$ will take on one of those $7$ values each time you try, say obtaining \texttt{A05} for $s=5$).  
For each set of coefficients, report the error in the estimate $\hat{Y}$ of $Y$ as 
\texttt{norm(Y - X*A,2)}.

\paragraph{B (30 points): }
Create three row-subsets of \texttt{X} and \texttt{Y}
\begin{itemize} \denselist
\item \texttt{X1 = X(1:66,:)} and \texttt{Y1 = Y(1:66)}
\item \texttt{X2 = X(34:100,:)} and \texttt{Y2 = Y(34:100)}
\item \texttt{X3 = [X(1:33,:); X(67:100,:)]} and \texttt{Y3 = [Y(1:33); Y(67:100)]}
\end{itemize}

Repeat the above procedure on these subsets and \emph{cross-validate} the solution on the remainder of \texttt{X} and \texttt{Y}.  Specifically, learn the coefficients \texttt{A} using, say, \texttt{X1 and Y1} and then measure \texttt{norm(Y(67:100) - X(67:100,:)*A,2)}.  

\paragraph{C (15 points): }
Which approach works best (averaging the results from the three subsets): Least Squares, or for which value of $s$ using Ridge Regression?  

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Orthogonal Matching Pursuit (25 points)}

Consider a linear equation \texttt{W = M*S} where \texttt{M} is a measurement matrix filled with random values $\{-1, 0, +1\}$ (although now that they are there, they are no longer random), and \texttt{W} is the output of the sparse signal \texttt{S} when measured by \texttt{M}.  

Use Orthogonal Matching Pursuit (as described in the notes as \textbf{Algorithm 18.2.1}) to recover the non-zero entries from \texttt{S}.  Record the order in which you find each entry and the residual vector after each step.  



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section*{BONUS: $L_1$ ball geometry (2 points)}
In class we examined pictures of $L_1$-balls in $\mathbb{R}^2$, but in higher dimensions their geometry is more wild.  Its boundary has "facets" of $k$-dimensions for each $k \in \{0, \ldots, d\}$, these are linear subspaces of of rank $k$.  

As a function of $d$ and $k$, derive how many different facets there are of $k$-dimensions when on the boundary of $L_1$-balls in $\mathbb{R}^d$.  Explain how you derive your answer.  




\end{document}
