TerribleDev/TimberWinR
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

More docs

Browse Source
This commit is contained in:
Eric Fontana
2014-07-30 07:36:16 -04:00
parent 0758893d90
commit fea6750f13
5 changed files with 167 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
README.md
View File

@@ -1,20 +1,31 @@
TimberWinR
==========
A Native Windows to Redis Logstash Agent which runs as a service.
## Why have TimberWinR?
TimberWinR is a native .NET implementation utilizing Microsoft's [LogParser](http://technet.microsoft.com/en-us/scriptcenter/dd919274.aspx). This means
no JVM/JRuby is required, and LogParser does all the heavy lifting. TimberWinR collects
the data from LogParser and ships it to Logstash via Redis.
## Supported Input Formats
- [WindowsEvents](https://github.com/efontana/TimberWinR/blob/master/WindowsEvents.md)
- IIS Logs (W3C)
- LogFiles (Tailing of files)
- TCP Port
## Basics
TimberWinR uses a configuration file to control how the logs are collected, filtered and shipped off.
These are broken down into:
1. Inputs (Collect data from different sources)
2. Filters (Are applied to all Inputs)
3. Outputs (Currently ships only to Redis)
## Input Formats
The current supported Input format sources are:
1. Logs (Files, a.k.a Tailing a file)
2. Tcp (listens on a port for JSON messages)
3. IISW3C (Internet Information Services W3C Format)
4. [WindowsEvents](https://github.com/efontana/TimberWinR/blob/master/WindowsEvents.md)
## Filters
The current list of supported filters are:
1. [Grok](https://github.com/efontana/TimberWinR/blob/master/mdocs/GrokFilter.md)
2. [Mutate](https://github.com/efontana/TimberWinR/blob/master/mdocs/MutateFilter.md)
## JSON
Since TimberWinR only ships to Redis, the format generated by TimberWinR is JSON. All fields referenced by TimberWinR can be
represented as a JSON Property or Array.
## Supported Output Formats
- Redis
## Sample Configuration
TimberWinR reads a JSON configuration file, an example file is shown here:
```json

4
TimberWinR.sln
View File

@@ -16,9 +16,11 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "TimberWinR.UnitTests", "Tim
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution Items", "{1B2F600B-2400-45B9-A28E-CFC391D9EFA9}"
ProjectSection(SolutionItems) = preProject
mdocs\DateFilter.md = mdocs\DateFilter.md
Filters.md = Filters.md
GrokFilter.md = GrokFilter.md
mdocs\GrokFilter.md = mdocs\GrokFilter.md
LICENSE.txt = LICENSE.txt
mdocs\MutateFilter.md = mdocs\MutateFilter.md
Package.nuspec = Package.nuspec
README.md = README.md
WindowsEvents.md = WindowsEvents.md

0
foo.mkd.txt Normal file
View File

142
mdocs/GrokFilter.md
View File

@@ -1 +1,143 @@
# Grok Filter
The grok filter allows you to parse unstructured data into something structured and
queryable. The very same patterns used by logstash are supported in TimberWinR.
See [Grok Patterns](https://github.com/elasticsearch/logstash/blob/v1.4.2/patterns/grok-patterns)
## Grok Basics
Grok works by combining text patterns into something that matches your logs.
The syntax for a grok pattern is %{SYNTAX:SEMANTIC} The SYNTAX is the name of the pattern
that will match your text. For example, “3.44” will be matched by the NUMBER pattern and “55.3.244.1”
will be matched by the IP pattern. The syntax is how you match. The SEMANTIC is the identifier you
give to the piece of text being matched. For example, “3.44” could be the duration of an event, so you could
call it simply ‘duration’. Further, a string “55.3.244.1” might identify the ‘client’
making a request. For the above example, your grok filter would look something like this:
%{NUMBER:duration} %{IP:client}
## Mutate Operations
The following operations are allowed when mutating a field.
| Operation | Type | Description
| :---------------|:----------------|:-----------------------------------------------------------------------|
| *condition* | property:string |C# expression
| *match* | property:string |Required field must match before any subsequent grok operations are executed.
| *add_field* | property:array |If the filter is successful, add an arbitrary field to this event. Field names can be dynamic and include parts of the event using the %{field} syntax. This property must be specified in pairs.
| *remove_field* | property:array |If the filter is successful, remove arbitrary fields from this event. Field names can be dynamic and include parts of the event using the %{field} syntax.
| *add_tag* | property:array |If the filter is successful, add an arbitrary tag to this event. Tag names can be dynamic and include parts of the event using the %{field} syntax.
| *remove_tag* | property:array |If the filter is successful, remove arbitrary tags from this event. Field names can be dynamic and include parts of the event using the %{field} syntax.
## Operation Details
### match
The match field is required, the first argument is the field to inspect, and compare to the expression specified by the second
argument. In the below example, the message is spected to be something like this from a fictional sample log:
```
55.3.244.1 GET /index.html 15824 0.043
```
The pattern for this could be:
```
%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}
```
And if the message matches, then 5 fields would be added to the event: client, method, request, bytes and duration.
```json
"Filters": [
{
"grok": {
"match": [
"message",
"%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}"
],
"add_tag": [
"rn_%{Index}",
"bar"
],
"add_field": [
"foo_%{logsource}",
"Hello dude from %{ComputerName}"
]
}
}
]
```
### condition "C# expression"
If present, the condition must evaluate to true in order for the remaining operations to be performed. If there is no condition specified
then the operation(s) will be executed in order.
```json
"Filters": [
{
"grok": {
"condition": "[type] == \"Win32-EventLog\""
"add_field": [
"ComputerName", "%{Host}"
]
}
}
]
```
The above example will add a field ComputerName set to the value of Host only for Win32-EventLog types.
### add_field ["fieldName", "fieldValue", ...]
The fields must be in pairs with fieldName first and value second.
```json
"Filters": [
{
"grok": {
"add_field": [
"ComputerName", "Host",
"Username", "%{SID}"
]
}
}
]
```
### remove_field ["tag1", "tag2", ...]
Remove the fields. More than one field can be specified at a time.
```json
"Filters": [
{
"grok": {
"remove_tag": [
"static_tag1",
"Computer_%{Host}"
]
}
}
]
```
### add_tag ["tag1", "tag2", ...]
Adds the tag(s) to the tag array.
```json
"Filters": [
{
"grok": {
"add_tag": [
"foo_%{Host}",
"static_tag1"
]
}
}
]
```
### remove_tag ["tag1", "tag2", ...]
Remove the tag(s) to the tag array. More than one tag can be specified at a time.
```json
"Filters": [
{
"grok": {
"remove_tag": [
"static_tag1",
"Username"
]
}
}
]
```

5
mdocs/MutateFilter.md
View File

@@ -7,14 +7,15 @@ The following operations are allowed when mutating a field.
| Operation | Type | Description
| :-----------|:----------------|:-----------------------------------------------------------------------|
| *condition* | property:string |Windows event logs
| *condition* | property:string |C# Expression
| *rename* | property:array |Rename one or more fields
| *replace* | property:array |Replace a field with a new value. The new value can include %{foo} strings to help you build a new value from other parts of the event.
| *split* | property:array |Separator between values of the "Strings" field.
## Details
### condition "C# expression"
If present, the condition must evaluate to true in order for the remaining operations to be performed.
If present, the condition must evaluate to true in order for the remaining operations to be performed. If there is no condition specified
then the operation(s) will be executed in order.
```json
"Filters": [
{