You can also try out the
Nexus Launcher (beta). The launcher offers options to change the graphics settings such as resolution or fullscreen and windowed mode. After you made your changes, you can either run Nexus through this new launcher or through the old one. This should handle all necessary registry changes for you.
This launcher requires
Microsoft .Net Framework 4 (Client Profile), if you don't have it already installed.
Manually changing the registry:You sure, you looked at the correct path in the registry? For Win7 64bit it is per default:
HKEY_CURRENT_USER\Software\Classes\VirtualStore\MACHINE\SOFTWARE\Wow6432Node\Mithis\Nexus - The Jupiter IncidentImportant: Make sure that you access the registry with the same Windows user account that you play Nexus with. If you use "run as administrator" when playing Nexus, also run regedit as administrator (in case your normal user account and your administrator account are not the same).
If you switched off Windows' UAC (it's on by default), the correct path is:
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Mithis\Nexus - The Jupiter IncidentAlso don't try to change the ingame options once you've changed the registry, or else it'll overwrite your resolution settings.