Until java 9 (which should have full support for high dpi screens) a tweak must be done on the client system that runs the SmartClient, what works for now is forcing first windows to use an external manifest:
Press Windows Logo key + R to open a Run. Type in regedit and press Enter.
In the left pane of the Registry Editor, navigate to the following directory:
Click on SideBySide in the left pane to expand its contents in the right pane. Right-click on an empty space in the right pane, hover over New and click on DWORD (32-bit) value.
Name the file PreferExternalManifest and press Enter. Right-click the new DWORD value and click on Modify. Type 1 into the Value Data. Change the value’s Base to Decimal. Click on OK. Close the Registry Editor.
Then create a xxxx.exe.manifest file with the following content:
That first xxxxx part of the manifest needs to be of the exe file you want to have an external manifest
For the smartclient this means java, and then in particular the java webstart launcher file
so go to the java installation folder, mostly located in a location like this:
And then copy the manifest above to files like: jp2launcher.exe.manifest, javaws.exe.manifest, java.exe.manifest and javaw.exe.manifest
So all those exe files have a manifest file right besides it.
There can be 1 problem with this is and that is that the next release of Windows 10 doesn't support this external manifest setting anymore (it ignores the above registry setting). For me the current slow ring build of Windows 10 (beta release build of 14986 seems to broke this).
If that is the case then this can be only fixed by adjusting the manifest file inside the exe file, you need tools for this like ResourceTuner: http://www.restuner.com/
With that you can open the exe file an then adjust the manifest that the exe file has and change the ms_windowsSettings:dpiAware section if it has it to false (there should be a true then). If it doesn't have such a section copy the above manifest contents and replace it fully.