The Openedge adminserver is setup to auto start and has been running smoothly for about 4 months on my Virtual Box Amazon Linux 2 server.
This morning the machine would boot and Apache would run, but no admin server.
On investigation of the OE Admin Server log (admserv.log) the last line in the log where it seems to have stalled has :
The optional ActiveMQ broker configuration file /opt/oe11/properties/activemq.xml does not exist. Default configuration will be used. (17212)
A quick google of that message along with “adminserver stuck” revealed the following link:
DescriptionAdminServer takes a long time to start on Linuxproadsv -q shows “starting” but looks like it does not startAdminServer takes at least 30 minutes to start on Ubuntu.On Red Hat Linux the AdminServer can take 4 mins or longer to start.On the AdminServer side the delay seems to occur after the ActiveMQ broker configuration and before the log4j warnings:
[ActiveMQS] The optional ActiveMQ broker configuration file /dlc/properties/activemq.xml does not exist. Default configuration will be used. (17212)[STDERR] log4j:WARN No appenders could be found for logger (org.apache.shiro.io.ResourceUtils)
This seemed close enough to my issue so I’ve copied the Cause and Resolution notes below and also noted what I actually did to get around the issue quickly below.
Since OpenEdge shipped with Java8 and requires JDK11 since 12.2, this issue affects java processes that require using encryption seed data for “NativePRNG”, “SHA1PRNG” and “DRBG” crypto mechanisms from file:/dev/random.
Linux needs entropy when working with cryptography. Entropy is randomness collected from the pseudo-device /dev/random from data generated by OS device drivers and services. The data sent to /dev/random is known as the entropy pool. Crypography uses a blocking function that waits until entropy reaches a certain level before providing its result. From a practical standpoint, this means that programs using /dev/random can take longer to complete and this can additionally affect timeouts or stop the process from working if the pool is empty.
This is not specific to OpenEdge, for additional vendor specific information regarding random being slow, is provided in the Notes section below.
When linking to an article, use a Smart LinkTo determine if the Operating System exhibits this behaviour at the time:
Try to display a portion of these files from a shell prompt and note the time difference:
$ head -n 10 /dev/random
$ head -c 10 file:/dev/urandomor,
due to a defect in certain distributions:
$ head -c 10 ///dev/urandom [ or ] head -c 10 /dev/./urandom
An OpenEdge 12.2 catalina.log demonstrates the delay in the: o.a.c.v.StuckThreadDetectionValve.notifyStuckThreadDetected & eventually o.a.c.v.StuckThreadDetectionValve.notifyStuckThreadCompleted messages.
To use /dev/urandom generator instead. This does not make cryptography less secure.
The following Blog summarises why: Myths about /dev/urandomhttps://www.2uo.de/myths-about-urandom/
Configure the JVM to use /dev/urandom:
The java.security file, used for security management is found in the following locations:
$ sudo find / -iname java.security
Java 8, and earlier versions: $JAVA_HOME/jre/lib/security/java.security.
For OpenEdge 11 which ships with java, this would be: $DLC/jre/lib/security/java.security
Open the java.security file in a text editor, as root user:
Java 11: $JAVA_HOME/conf/security/java.security
Change the following line:securerandom.source=file:/dev/randomto read:securerandom.source=file:/dev/urandomAfter Saving the update, restart the AdminServer.
Another method is to use the RNGD Service
The rngd service is part of the rng-tools package on both Debian and Red Hat / CentOS based systems.
As a quick verification, run the following command as root, then start the AdminServer which will no longer experience the delay on startup:
This command can be added to /etc/rc.local or crontab to be run at system startup.
$ rngd -r /dev/urandom $ proadsv -start
The following information is provided as a guideline, refer to OS documentation for details specific to your Linux distribution.
Install “rng-tools” if not exists. Login with root privileges and run:
UBUNTU: apt-get install rng-toolsUpdate “/etc/default/rng-tools” file to add “HRNGDEVICE=/dev/urandom”Start rng-tools: /etc/init.d/rng-tools start
CentOS: yum install rng-tools
Edit the service unit
ExecStart=/sbin/rngd -f -r /dev/urandom
Reload the unit file, start the service and check it’s status:
$ systemctl enable rngd$ systemctl daemon-reload
$ systemctl start rngd$ systemctl status rngd
What I actually did
I tried the head command and it hung. I then moved to another Amazon Linux 2 VM I have running that had not yet been updated with “sudo yum update”, and the head command worked.
I followed the instructions above about changing the Java.Security file references to be /dev/urandom, but they were already set to /dev/urandom.
I then followed the steps for the RNGD solution.
$ rngd -r /dev/urandom
As soon as this command was entered, the wait on the other terminal that was trying to start the admin server kicked in. I queried and stopped the admin server, then restarted.
$ proadsv -query all
This said everything was running – but obviously it wasn’t.
$ proadsv -stop
$ proadsv -start
Everything started up again afterwards, however I need to add the “rngd” command into the boot process of the VM, as a reboot just gives me the same problem.
As mentioned, my clone of the VM which was not updated with “sudo yum update” still works – so it may be that an Amazon Linux 2 update has changed the interaction of /dev/random and affects OpenEdge.
I’ll back these machines up, and test further, but at least I was able to start up my db server!
Further testing reveals
It looks like the RNGD service is not running.
After following these steps in the following post everything was back to normal.
So, does that mean that the AMZ yum update broke the rngd service? Let’s try on the machine that was working (after backing it up first!)
Before update : systemctl status rngd
Then I do a sudo yum update and noticed….
There are rng-tools getting updated!
All looks fine, the I reboot the machine.