If your Mac refuses to go into Firewire mode, check your keyboard! Chances are, it is plugged into your monitor. Plug it directly into the computer (you'll probably have to dig out your old USB extension cable to do this). Also, unplug all other firewire devices.

For more on Firewire Target Disk mode, see this page [Link]