↓Skip to main content
  1. Posts/

vRealize Log Insight : pre-classifying events for SIEM

·769 words·4 mins·
vRealize Log Insight blogposts vRealize Log Insight blogposts

For a project i’m currently working on we are using vRealize Log Insight as a logging and monitoring solution. One of the important requirements this customer has is the forwarding of syslog events to HP Arcsight, a SIEM solution. However, as the amount of syslog events is increased significantly with the addition of NSX and enabling the logging of the NSX DFW, we cannot filter and index all events on their existing HP Arcsight solution, as it would cause excessive strain on the systems. Because Loginsight can be scaled up with relative ease, we’ve instead decided to perform filtering and indexing of a variety of security events in vRealize Log Insight, before the events even reach HP Arcsight.

Our production vRealize Log Insight environment is a relatively straightforward one. Each datacenter runs a cluster of 3 nodes with multiple ILB VIPs, which each perform mutual forwarding to eachother.

From thereon we could forward all events directly to HP Arcsight through syslog. However, as mentioned before, due to the amount of events generated by vSphere and NSX (due to a large amount of DFW rules having logging enabled), we want to classify events which are interesting to the Security Operations team.

To achieve this, first we need to define our specific security events. An example of one of these events would be “Changing the syslog configuration for a physical router” or “Deleting the syslog configuration for a physical router”.

As the filtering options for forwarding in vRealize Log Insight are relatively simple, we’ll use extracted fields for this purpose.

First, create a search to find your event. In our case, we purposely executed the use case, in our case changing the syslog server on a physical router and checked which syslog events were generated.

Next, we’ll create an extracted field that matches our exact use case:

Highlight any text in the syslog message, and the filter menu should pop up. Click “Extract field” and you should see a custom field definition on the right.

Now, we need to create our custom field. One thing to keep in mind here is that some use cases may be matched by multiple commands. In the case of a router, due to the way command completion works, one could set the syslog configuration with both set system syslog as well as the slightly more obscure s sy sy command, or even s sy sy, to obfuscate any matches that have a fixed space count.

For example, our matching regular expression, and the extracted value would be as follows:

se?t?\s+(sy)s?t?e?m?\s+(sy)s?l?o?g?(.*)$

which matches with everything from the moment a user ran the “set system syslog” (or any shorthand) command, and which is also what our security operations team is mostly interested in. For the people interested, the rules of the regex above are:

  • Match exactly 1 instance of the character “s”
  • Match 0 or 1 instances of the characters e, t
  • Match 1 or more whitespace characters
  • Match the string “sy”
  • Match 0 or 1 instances of each of the characters s, t, e, m
  • Match 1 or more whitespace characters
  • Match the string “sy”
  • Match 0 or 1 instances of each of the characters s, l, o, g
  • Match any characters
  • Match the end of string.

note that it seems that Log Insight always matches case insensitive, so we did not need to worry about that in our above regex

In addition, in some cases we do not wish to match this use case on every event that would match this specific regular expression, as it could pollute the SIEM event stream and potentially generate false positives. To do so, we set an additional context on “hostname” and have it match a regular expression on the format of the router hostnames. In our case, the hostnames for the physical routers follow the following pattern:

  • Country Code
  • City Code
  • Site Number
  • the string “FRW”
  • A 4 length Device Number

So in our case, we can match the hostname field with the following regular expression:

(CC1|CC2|CC2)(CITY1|CITY2|CITY3)[1-9](FRW)[0-9]{4}

When we store this extracted field for all users, this will now add an additional field to each event that matches exactly this use case. While vRealize Log Insight cannot filter forwarding based on dynamic fields, we can still provide this field to the SIEM team to reduce the amount of indexing and filtering that needs to be done. They only need to look at the existence of a specific field to link it to a specific use case, and can even directly ingest the contents of this extracted field as the relevant data without having to filter the message themselves.

↑