Question
Can low-rank fine-tuning save activation memory without freezing a factor that the model still needs to learn?
Takeaway
The useful frame is to separate memory pressure from expressivity loss. A method that saves memory but blocks adaptation may look efficient while quietly capping quality.
Follow-up
Implement a small experiment that compares full LoRA, LoRA-FA, and the reconstruction-style variant on a toy task.