Final Exam  
Instructor: Amrit Singh Bedi  
Instructions  
This exam is worth a total of 100 points. Please answer all questions clearly  
and concisely. Show all your work and justify your answers.  
• For Question 1 and 2, please submit the PDF version of your solution  
via webcourses. You can either write it in latex or do it on paper and  
submit the scanned version. But if you do it on paper and scan it,  
you are responsible for ensuring it is readable and properly scanned.  
There will be zero marks if it is not clearly written or scanned.  
• The total time to complete the exam is 24 hours and it is due at 4:00  
pm EST, Friday (April 25th, 2025). This is a take-home exam. Please  
do not use AI like ChatGPT to complete the exam. There are zero  
marks if found (believe me, we would know if you use it).  
Question 1 50 marks  
Context: In supervised learning, understanding the bias-variance tradeoff  
is crucial for developing models that generalize well to unseen data.  
Problem 1 10 marks  
Define the terms bias, variance, and irreducible error in the context of su pervised learning. Explain how each contributes to the total expected error  
of a model.  
1  
Problem 2 20 marks  
Derive the bias-variance decomposition of the expected squared error for a  
regression problem. That is, show that:  
ED,ε[(y − f  
ˆ(x))2  
] = Bias[f  
ˆ(x)]  
2  
+ Var[f  
ˆ(x)] + σ  
2  
where f  
ˆ(x) is the prediction of the model trained on dataset D, y = f(x)+ε,  
and σ  
2  
is the variance of the noise ε.  
Hint: You can start by taking y = f(x) + ε, where E[ε] = 0, and  
Var[ε] = σ  
2  
. Let f  
ˆ(x) be a learned function from the training set D. Then  
proceed towards the derivation.  
Problem 3 10 marks  
Consider two models trained on the same dataset:  
• Model A: A simple linear regression model.  
• Model B: A 10th-degree polynomial regression model.  
Discuss, in terms of bias and variance, the expected performance of each  
model on training data and 代写CAP 4611 Final Exam unseen test data. Which model is more likely  
to overfit, and why?  
Problem 4 10 marks  
Explain how increasing the size of the training dataset affects the bias and  
variance of a model. Provide reasoning for your explanation. (10 marks)  
Question 2: Using Transformer Attention 50  
marks  
Context. Consider a simplified Transformer with a vocabulary of six to kens:  
• I (ID 0): embedding  1.0, 0.0  
  
• like (ID 1): embedding  0.0, 1.0  
  
• to (ID 2): embedding  1.0, 1.0  
  
2  
• eat (ID 3): embedding  0.5, 0.5  
  
• apples (ID 4): embedding  0.6, 0.4  
  
• bananas (ID 5): embedding  0.4, 0.6  
  
All three projection matrices are the 2 × 2 identity:  
WQ = WK = WV = I2.  
When predicting the next token, the model uses masked self-attention: the  
query comes from the last position, while keys and values come from all  
previous tokens. (Note: show step by step calculation for all questions  
below)  
(a) (10 marks) For the input sequence [I, like, to] (IDs [0, 1, 2]),  
compute the query, key and value vectors for each token.  
(b) (15 marks) Let Q be the query of the last token and K, V the keys  
and values of all three tokens.  
• Compute the row vector of raw attention scores qK⊤, where q is  
the query of the last token and K is the 3×2 matrix of keys. .  
• Scale by √  
dk (with dk = 2) and apply softmax to obtain attention  
weights.  
• Compute the context vector as the weighted sum of the values.  
(c) (15 marks) Given the context vector c ∈ R  
2  
from part (b), com pute the unnormalized score for each vocabulary embedding via c ·  
embed(w), i.e. dot-product.  
• Apply softmax over these six scores to get a probability distribu tion.  
• Which token has the highest probability? [Note: Because the six  
embeddings are synthetic and not trained on real text, the token  
that receives the highest probability may look ungrammatical in  
normal English; this is an artifact of the toy setup.]  
(d) (10 marks) Explain why the model selects the token you found in  
(c). In your answer, discuss:  
• How the attention weights led to that choice.  
• Explain why keys/values may include the current token but never