Servoy developer until version 8.2 is build on Eclipse 4.5.0 (mars) or lower, that version of eclipse doesn't work with Java9.
Servoy 8.3 will upgrade to Eclipse 4.7 (Oxygen) that eclipse does support Java9 but it could be that the servoy.ini file does need to contain this vm-arg:
--add-modules=ALL-SYSTEM (with the latest 8.3.x this is not needed anymore. eclipse will auto add this setting)
Java 11 is for the developer only supported from Servoy 8.4 on. Because Java 11 doesn't have a runtime anymore that integrates into the OS. (registry settings and so on). You need to adjust the servoy.ini to point to that install, you need to add:
see for more info also the page of eclipse itself: https://wiki.eclipse.org/Configure_Eclipse_for_Java_9
See http://openjdk.java.net/jeps/261 for more information about those new Java 9 Module system properties
For the smart client we have a page here Java 9 and 10 for smart client