About Agent Caching

SUMMARY

Jun 19, 2017

For the Snare Enterprise Agent for Windows the primary cache is the Windows event log.  This is the source of all cache management when TCP or TLS are used, so when a system is rebooted and/or the destination is down, then events are bookmarked to the relevant points in the event log so the agent knows where its got to with sending logs. Once the destination is back online or contactable then the agent will resume sending logs from the known position in the event log.

Memory caching regards the internal workings of the agent. As the agent reads events, it caches them before being sent. When there are multiple destinations being used then the cache helps ensure that each destination is handled correctly with sending the logs, another words it’s the internal buffer. When the system restarts, anything that’s in memory will be written to disk as part of a clean shutdown. It will be hard to induce this write to disk for the windows agent as it depends on if the destination was not available before the shutdown and there were events to send.  It can happen in some circumstances but it's more of a safety net to minimise the chance of data loss.  So for the windows agent this cache does not need to be very large as the primary disk cache is the windows event log itself.  If the agent did not send the event then the bookmarks for the events from the windows event log don’t move, so the event should not be lost from any outage. So the memory cache in this instance has not lost anything.

The memory cache is used a lot more for the other agents Snare Enterprise Epilog for Windows, Snare Enterprise Agent for MSSQL, Snare Enterprise Agent for Linux etc as the source of the events is a stream of data and not a fixed source like the windows event logs is. So the cache for these agents is more important and will write to disk more often during a shutdown as they will more likely have data in the memory cache.

In conclusion, the basic architecture is the same for all agents and they have this memory cache but in the case of the windows agent it's less of an issue.

How to test caching

If you are looking to test the caching then it’s the event log cache usage is what you are after. This works well when using TCP or TLS as the destination protocol as UDP is 'fire and forget' with no acknowledgement the other end has received any data.  The bookmarks we use in this case are stored in the Status registry location for the agent and are updated as the agent sends the logs, so it knows where it is at any time if the system is rebooted or the destination goes down.