diff --git a/README.md b/README.md index 098ebb4..27fcba5 100644 --- a/README.md +++ b/README.md @@ -1,20 +1,31 @@ TimberWinR ========== A Native Windows to Redis Logstash Agent which runs as a service. - ## Why have TimberWinR? TimberWinR is a native .NET implementation utilizing Microsoft's [LogParser](http://technet.microsoft.com/en-us/scriptcenter/dd919274.aspx). This means no JVM/JRuby is required, and LogParser does all the heavy lifting. TimberWinR collects the data from LogParser and ships it to Logstash via Redis. -## Supported Input Formats - - - [WindowsEvents](https://github.com/efontana/TimberWinR/blob/master/WindowsEvents.md) - - IIS Logs (W3C) - - LogFiles (Tailing of files) - - TCP Port +## Basics +TimberWinR uses a configuration file to control how the logs are collected, filtered and shipped off. +These are broken down into: + 1. Inputs (Collect data from different sources) + 2. Filters (Are applied to all Inputs) + 3. Outputs (Currently ships only to Redis) +## Input Formats +The current supported Input format sources are: + 1. Logs (Files, a.k.a Tailing a file) + 2. Tcp (listens on a port for JSON messages) + 3. IISW3C (Internet Information Services W3C Format) + 4. [WindowsEvents](https://github.com/efontana/TimberWinR/blob/master/WindowsEvents.md) +## Filters +The current list of supported filters are: + 1. [Grok](https://github.com/efontana/TimberWinR/blob/master/mdocs/GrokFilter.md) + 2. [Mutate](https://github.com/efontana/TimberWinR/blob/master/mdocs/MutateFilter.md) +## JSON +Since TimberWinR only ships to Redis, the format generated by TimberWinR is JSON. All fields referenced by TimberWinR can be +represented as a JSON Property or Array. ## Supported Output Formats - Redis - ## Sample Configuration TimberWinR reads a JSON configuration file, an example file is shown here: ```json diff --git a/TimberWinR.sln b/TimberWinR.sln index f076ae4..cc26501 100644 --- a/TimberWinR.sln +++ b/TimberWinR.sln @@ -16,9 +16,11 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "TimberWinR.UnitTests", "Tim EndProject Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution Items", "{1B2F600B-2400-45B9-A28E-CFC391D9EFA9}" ProjectSection(SolutionItems) = preProject + mdocs\DateFilter.md = mdocs\DateFilter.md Filters.md = Filters.md - GrokFilter.md = GrokFilter.md + mdocs\GrokFilter.md = mdocs\GrokFilter.md LICENSE.txt = LICENSE.txt + mdocs\MutateFilter.md = mdocs\MutateFilter.md Package.nuspec = Package.nuspec README.md = README.md WindowsEvents.md = WindowsEvents.md diff --git a/foo.mkd.txt b/foo.mkd.txt new file mode 100644 index 0000000..e69de29 diff --git a/mdocs/GrokFilter.md b/mdocs/GrokFilter.md index 579fb0f..c1a47f6 100644 --- a/mdocs/GrokFilter.md +++ b/mdocs/GrokFilter.md @@ -1 +1,143 @@ # Grok Filter +The grok filter allows you to parse unstructured data into something structured and +queryable. The very same patterns used by logstash are supported in TimberWinR. +See [Grok Patterns](https://github.com/elasticsearch/logstash/blob/v1.4.2/patterns/grok-patterns) + +## Grok Basics + +Grok works by combining text patterns into something that matches your logs. +The syntax for a grok pattern is %{SYNTAX:SEMANTIC} The SYNTAX is the name of the pattern +that will match your text. For example, “3.44” will be matched by the NUMBER pattern and “55.3.244.1” +will be matched by the IP pattern. The syntax is how you match. The SEMANTIC is the identifier you +give to the piece of text being matched. For example, “3.44” could be the duration of an event, so you could +call it simply ‘duration’. Further, a string “55.3.244.1” might identify the ‘client’ +making a request. For the above example, your grok filter would look something like this: + +%{NUMBER:duration} %{IP:client} + + +## Mutate Operations +The following operations are allowed when mutating a field. + +| Operation | Type | Description +| :---------------|:----------------|:-----------------------------------------------------------------------| +| *condition* | property:string |C# expression +| *match* | property:string |Required field must match before any subsequent grok operations are executed. +| *add_field* | property:array |If the filter is successful, add an arbitrary field to this event. Field names can be dynamic and include parts of the event using the %{field} syntax. This property must be specified in pairs. +| *remove_field* | property:array |If the filter is successful, remove arbitrary fields from this event. Field names can be dynamic and include parts of the event using the %{field} syntax. +| *add_tag* | property:array |If the filter is successful, add an arbitrary tag to this event. Tag names can be dynamic and include parts of the event using the %{field} syntax. +| *remove_tag* | property:array |If the filter is successful, remove arbitrary tags from this event. Field names can be dynamic and include parts of the event using the %{field} syntax. + +## Operation Details +### match +The match field is required, the first argument is the field to inspect, and compare to the expression specified by the second +argument. In the below example, the message is spected to be something like this from a fictional sample log: + +``` +55.3.244.1 GET /index.html 15824 0.043 +``` + +The pattern for this could be: +``` +%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration} +``` +And if the message matches, then 5 fields would be added to the event: client, method, request, bytes and duration. + +```json + "Filters": [ + { + "grok": { + "match": [ + "message", + "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" + ], + "add_tag": [ + "rn_%{Index}", + "bar" + ], + "add_field": [ + "foo_%{logsource}", + "Hello dude from %{ComputerName}" + ] + } + } + ] +``` + +### condition "C# expression" +If present, the condition must evaluate to true in order for the remaining operations to be performed. If there is no condition specified +then the operation(s) will be executed in order. +```json + "Filters": [ + { + "grok": { + "condition": "[type] == \"Win32-EventLog\"" + "add_field": [ + "ComputerName", "%{Host}" + ] + } + } + ] +``` +The above example will add a field ComputerName set to the value of Host only for Win32-EventLog types. + +### add_field ["fieldName", "fieldValue", ...] +The fields must be in pairs with fieldName first and value second. +```json + "Filters": [ + { + "grok": { + "add_field": [ + "ComputerName", "Host", + "Username", "%{SID}" + ] + } + } + ] +``` + +### remove_field ["tag1", "tag2", ...] +Remove the fields. More than one field can be specified at a time. +```json + "Filters": [ + { + "grok": { + "remove_tag": [ + "static_tag1", + "Computer_%{Host}" + ] + } + } + ] +``` + + +### add_tag ["tag1", "tag2", ...] +Adds the tag(s) to the tag array. +```json + "Filters": [ + { + "grok": { + "add_tag": [ + "foo_%{Host}", + "static_tag1" + ] + } + } + ] +``` + +### remove_tag ["tag1", "tag2", ...] +Remove the tag(s) to the tag array. More than one tag can be specified at a time. +```json + "Filters": [ + { + "grok": { + "remove_tag": [ + "static_tag1", + "Username" + ] + } + } + ] +``` diff --git a/mdocs/MutateFilter.md b/mdocs/MutateFilter.md index 7313969..788276a 100644 --- a/mdocs/MutateFilter.md +++ b/mdocs/MutateFilter.md @@ -7,14 +7,15 @@ The following operations are allowed when mutating a field. | Operation | Type | Description | :-----------|:----------------|:-----------------------------------------------------------------------| -| *condition* | property:string |Windows event logs +| *condition* | property:string |C# Expression | *rename* | property:array |Rename one or more fields | *replace* | property:array |Replace a field with a new value. The new value can include %{foo} strings to help you build a new value from other parts of the event. | *split* | property:array |Separator between values of the "Strings" field. ## Details ### condition "C# expression" -If present, the condition must evaluate to true in order for the remaining operations to be performed. +If present, the condition must evaluate to true in order for the remaining operations to be performed. If there is no condition specified +then the operation(s) will be executed in order. ```json "Filters": [ {