If your wad is not launching through the game (but is working in the editor for test runs) then try this. Note that this also would work for those struggling to launch some else’s wad.
Click on pictures to enlarge.
1: Open the folder where your Doom 64 map and the Doom64.EXE is located.
2: Hold Shift+Right Click inside this folder (Be sure you don’t highlight/select anything inside the folder, do this action in a blank spot in your folder).
3: Step 2 will bring up some options find and choose “open command window here”
4: When greeted with the command window simply type your wad name in and hit enter.
That should do it! You should be able to launch wads using Doom64.EXE now.