After the iOS extender sends the push notifications to refresh all devices, the following fatal error shows up multiple times in the iosmdm.log:
2012-05-01 13:56:08
FATAL
iOSMDM
ActiveRecord::ConnectionTimeoutError (could not obtain a database connection within 5 seconds. The max pool size is currently 5; consider increasing it.):
Is this something I should consider increasing, and if so, how would I increase that? Is it something I should be concerned about? Thus far, we haven’t had any problems.
We currently have a little over 100 devices on TEM. The iOS extender is running on 2 cores @ ~3GHz and 4GB RAM.
TEM just sent the refresh all command and I monitored the task manager:
CPU usage hit a peak of 98% and averaged ~70-75% with the following processes being the main hitters - ProxyAgent.exe, BESiOSServer.exe, and iOSProxyPlugin.exe
RAM usage hovered at about 2GB
When working on individual devices through actions in the console, CPU usage averages below 50% and RAM hovers at 2GB
RAM usage doesn’t really change too much. During idle operation, the Resource Monitor shows BESiOSServer.exe and ProxyAgent.exe using the most RAM at ~390MB and ~265MB, respectively.
Our refresh interval is set to 3 hours. Most devices are active, but I’ve noticed there are times when some devices may miss some refresh cycles. For example, recently I noticed my iPhone’s last reported time was sometime on the night before (it was never turned off and had good signal) meaning that it missed at least 3 refresh intervals. I manually did a “Send Refresh” on it, and it came through fine. It never occurred to me that this would be related to the database error message.
I believe that the Apple built-in behavior is that sometimes when the device is idle, it will not check-in if nothing has changed (even if we send it requests for updates). Since the MDM API behavior is controlled by Apple, we can’t change it…
So I am not sure if this is related to the error messages you saw, but if you start to see unresponsive iOS agents when you are trying to push actions, let us know and we can investigate further.
One more thing… You might consider a longer interval for the refresh cycle for the agents. There is certainly nothing is wrong with a 3-hour cycle, but since the heartbeats do take a bit of bandwidth and a bit of battery life (although not much), it sometimes make sense to move the iOS refresh interval to something longer (maybe 6 hours or 12 hours?) This boils down to your deployment preference, but to be cautious, you might start with longer intervals and then increase them if you need to…
ActiveRecord::ConnectionTimeoutError (could not obtain a database connection within 5 seconds. The max pool size is currently 5; consider increasing it.):
2012-05-23 12:09:10
FATAL
iOSMDM
ActiveRecord::ConnectionTimeoutError (could not obtain a database connection within 5 seconds. The max pool size is currently 5; consider increasing it.):
2012-05-23 12:09:10
FATAL
iOSMDM
ActiveRecord::ConnectionTimeoutError (could not obtain a database connection within 5 seconds. The max pool size is currently 5; consider increasing it.):
2012-05-23 12:09:10
INFO
iOSMDM
Started PUT “/mdm” for ###.###.###.### at 2012-05-23 12:09:10 -0400
2012-05-23 12:09:10
FATAL
iOSMDM
ActiveRecord::ConnectionTimeoutError (could not obtain a database connection within 5 seconds. The max pool size is currently 5; consider increasing it.):
2012-05-23 12:09:10
FATAL
iOSMDM
ActiveRecord::ConnectionTimeoutError (could not obtain a database connection within 5 seconds. The max pool size is currently 5; consider increasing it.):
2012-05-23 12:09:11
FATAL
iOSMDM
ActiveRecord::ConnectionTimeoutError (could not obtain a database connection within 5 seconds. The max pool size is currently 5; consider increasing it.):
2012-05-23 12:09:11
FATAL
iOSMDM
ActiveRecord::ConnectionTimeoutError (could not obtain a database connection within 5 seconds. The max pool size is currently 5; consider increasing it.):
2012-05-23 12:09:11
INFO
iOSMDM
Started PUT “/mdm” for ###.###.###.### at 2012-05-23 12:09:11 -0400
2012-05-23 12:09:11
INFO
iOSMDM
Started PUT “/mdm” for ###.###.###.### at 2012-05-23 12:09:11 -0400
2012-05-23 12:09:11
FATAL
iOSMDM
ActiveRecord::ConnectionTimeoutError (could not obtain a database connection within 5 seconds. The max pool size is currently 5; consider increasing it.):
2012-05-23 12:09:11
FATAL
iOSMDM
ActiveRecord::ConnectionTimeoutError (could not obtain a database connection within 5 seconds. The max pool size is currently 5; consider increasing it.):
We have made some changes in our latest release (just released) that will hopefully ease some of these issues, It appeared the issue was that our sqlite database was not holding up under the amount of concurrency that the iOS server requires.
Two changes will help out here:
We have added in the next release the ability to customize the database connection pool size, as well as the timeout length. By tweaking these values, you can avoid these kinds of errors.
We have added in a default limit on the number of APNS messages sent per second (currently customizable already, but with no limit by default), which would prevent too many devices checking at once that caused this issue.