To use this tool, you must first set the filename of the script you intend to test/debug (including the .js extension) in the 'filename' field. Then hit 'Reload' to load it. To make your algorithm object accessible to this tool, you must create a global variable within your algorithm file called testAlgo, which then actually contains your algorithm object i.e. the object that provides rgbMap() and rgbMapStepCount() functions. Now you're ready to load this file with your browser and see your first results.
If you see what you expect to see in the Algorithm Properties section below, you'll know your script has been evaluated successfully. Next, put some values to the Width and Height boxes in the Grid Size section and see if your script calculates the rgbMapStepCount(width, height) result correctly. If everything is OK, you should already see your RGB script running through frames. To inspect individual frames, stop the script and hit the previous and next buttons.
For troubleshooting purposes, I would suggest using some good JavaScript development tools for your browser. Mozilla FireFox has a built-in Web Console behind Tools -> Web Developer which is very useful in hunting errors within the JavaScript code. Google Chrome has one as well, that you can access with F12 or with More Tools -> Developer Tools (Ctrl+Shift+I)