Recently performed an upgrade from ArcGIS Server 10.3.1 to 10.4.1. Everything seemed fine but two geoprocessing services refused to start and were throwing an error of;
GPServer: java.rmi.UnmarshalException: Error unmarshaling return header; nested exception is: java.net.SocketException: Connection reset
Turns out (according to this link) that the registry key under HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.7 might be empty. Checked the registry and indeed it was empty.
Turns out uninstalling the background geoprocessing for ArcGIS Desktop shouldnt be done after upgrading ArcGIS Server (forgot that it was installed!).
Luckily I could recover the registry keys from another environment and all was well.