This had me scratching my head for a while.. The retrospective debugger was working great in a client org then just stopped working one day - all the different methods of getting into the debugger were failing with various errors.
What I eventually figured out was that the difference between orgs where the debugger worked and orgs where it didn't was the first line in the log. In my case, for an org where it doesn't work, it looks like this:
37.0 APEX_CODE,FINEST;APEX_PROFILING,INFO;CALLOUT,INFO;DB,INFO;SYSTEM,FINE;VALIDATION,INFO;VISUALFORCE,FINE;WAVE,INFO;WORKFLOW,INFO
If I remove ';WAVE,INFO' from this line then save the log, I can debug it. I suspect it's having a hard time with any component it doesn't expect, not just Wave.