You don't compile this mod... You drag and drop it into the sm64coopdx window
If you're struggling to compile sm64coopdx, well, you don't have to compile it! You can download coop here, get the OpenGL ver (if that doesn't work, use directx), boot up the game, and drag and drop your rom!