You are gong to have to redo and recheck everything starting from scratch
here and
here until you get an error-free dump (you can try a different Nand-dumping tool like iHc NandTool or XNand Healer GUI). Problems may or may not occur if you continue on...
Make sure you use a 32-bit operating system like Windows XP to lessen the amount of problems. Also, refollow the
XeLL-JTAG-XBR-Hack guide again. Make sure the connections on the USB-SPI and directly on the Xbox 360 are OK, and the Bootloader Mode jumper is in the right place before dumping. Also, check to see if the Xbox 360 is turned off, but power adapter still plugged in, etc. Also do not Solder the SMC-JTAG connections yet, because they shouldn't be connected while dumping the Nand. Remove them as shown
here.
Some more tips -
http://www.tiaowiki.com/forums/xbox-360-hardware-talk/nandpro-2-0e-jtag-starting-checklist-(all-windows)/