Open finder, and in the menu bar, hit "Go," then hit "Go to Folder". Once you do that, copy and paste this in:
~/Library/Application Support/sm64ex-coop/
That should work, if it doesn't, you can try checking the app file, going to Contents/MacOS, then opening up the mods folder, and handling...