Watch out, updating Linux can break the WebLogic AdminServer

 I ran into a situation recently that had me puzzled for a while. 

Situation: Fusion Middleware 12.2.1.4 with WebCenter 12.2.1.4 running on top of Oracle Linux 8. On the MiddleWare and WCC side all the most recent patches are applied.

Everything is working fine. And then the system administrator decides to update the Linux installation. The server is reboot, and at first view the Middleware stack also starts correctly. Or does it?

Yes all the parts of the installation are up and running, but there is an issue.

When I tried to access the enterprise manager interface http://<server>:7001/em I get a big fat stacktrace instead of the login screen.

Error 500--Internal Server Error
javax.servlet.UnavailableException: Service is not initialized correctly. Verify that the repository-less configuration is correct.

Digging through the different log files I found a number of other messages indicating that something isn't coping well with the OS patches.

####<Apr 19, 2022 10:32:45,622 AM CEST> <Warning> <oracle.dms.instrument> <server2> <UCM_server2> <pool-21-thread-1> <<anonymous>> <> <0f7ad2be-0aef-477a-9b38-f2b97a57ddc6-0000000b> <1650357165622> <[severity-value: 16] [rid: 0:670] [partition-id: 0] [partition-name: DOMAIN] > <DMS-50763> <Attempt to create pre-existing noun /wccprd12c/UCM_server2/Oracle Universal Content Management - Content Server/WEBs, of type wls_module, with a conflicting type wls_context.>
####<Apr 19, 2022 10:32:58,797 AM CEST> <Error> <oracle.wsm.resources.policymanager> <server2> <UCM_server2> <pool-21-thread-1> <<anonymous>> <> <0f7ad2be-0aef-477a-9b38-f2b97a57ddc6-0000000b> <1650357178797> <[severity-value: 8] [rid: 0:670] [partition-id: 0] [partition-name: DOMAIN] > <WSM-02311> <One or more documents cannot be retrieved from the repository due to underlying error "This RJVM has already been shutdown 4949743297242877227S:10.81.82.204:[11700,11700,-1,-1,-1,-1,-1]:inetAddress=server1/10.81.82.204:wccprd12c:AdminServer"
java.rmi.ConnectException: This RJVM has already been shutdown 4949743297242877227S:10.81.82.204:[11700,11700,-1,-1,-1,-1,-1]:inetAddress=crkucmappoci01p.statoilfuelretail.com/10.81.82.204:wccprd12c:AdminServer
at weblogic.rjvm.RJVMImpl.getOutputStreamByURL(RJVMImpl.java:485)
at weblogic.rjvm.RJVMImpl.getMsgAbbrevOutputStream(RJVMImpl.java:802)
at weblogic.rjvm.RJVMImpl.getRequestStream(RJVMImpl.java:793)
at weblogic.rjvm.RJVMImpl.getOutboundRequest(RJVMImpl.java:1064)
at weblogic.rmi.internal.BasicRemoteRef.getRequest(BasicRemoteRef.java:235)
at weblogic.rmi.internal.BasicRemoteRef.getOutboundRequest(BasicRemoteRef.java:199)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:542)
at weblogic.rmi.cluster.ClusterableRemoteRef.invoke(ClusterableRemoteRef.java:313)

2022-04-19 09:40:56,741 [[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'] WARN help.BuildMasterHelpSet logp.251 - Not a help jar file.Ignoring the current file :emOtnMap_core.xml
2022-04-19 09:41:25,512 [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] ERROR app.ContextInitFilter handleNoSecureTargetRepolessMode.628 - No secure targets in the EM with repoless mode

2022-04-19 09:41:25,521 [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] ERROR exception.EMRuntimeExceptionFilter logp.251 - javax.servlet.UnavailableException: Service is not initialized correctly. Verify that the repository-less configuration is correct.
javax.servlet.UnavailableException: Service is not initialized correctly. Verify that the repository-less configuration is correct.
at oracle.sysman.eml.app.ContextInitFilter.handleNoSecureTargetRepolessMode(ContextInitFilter.java:633)
at oracle.sysman.eml.app.ContextInitFilter.doFilter(ContextInitFilter.java:274)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)
at oracle.adfinternal.view.faces.caching.filter.AdfFacesCachingFilterImpl.doFilter(AdfFacesCachingFilterImpl.java:134)
at oracle.adf.view.rich.webapp.AdfFacesCachingFilter.doFilter(AdfFacesCachingFilter.java:66)
at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:78)

How on earth can updating the OS suddenly break the enterprise manager? That doesn't make sense. It took quite some investigative effort, and it turned out to be quite surprising.

For some no doubt security reason one of the OS patches decided to update the contents of the /etc/hosts file. Before installing the OS patches there was a line in /etc/hosts that read:

10.81.82.204 shorthostname hostname.domainname

After the update the names of the server had been deleted and only the IP address was left. And yes, this does break the em interface of WebLogic.

The fix is easy. Just vi the /etc/hosts file to add the server names once again, restart the AdminServer, and everything is restored to working order.