TerribleDev/TimberWinR
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

restructure docs

Browse Source
This commit is contained in:
Eric Fontana
2014-07-30 07:55:07 -04:00
parent 7265514b4c
commit e819fbae55
14 changed files with 347 additions and 83 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
README.md
View File

@@ -18,12 +18,13 @@ The current supported Input format sources are:
1. Logs (Files, a.k.a Tailing a file) 1. Logs (Files, a.k.a Tailing a file)
2. Tcp (listens on a port for JSON messages) 2. Tcp (listens on a port for JSON messages)
3. IISW3C (Internet Information Services W3C Format) 3. IISW3C (Internet Information Services W3C Format)
4. [WindowsEvents](https://github.com/efontana/TimberWinR/blob/master/WindowsEvents.md) 4. [WindowsEvents](https://github.com/efontana/TimberWinR/blob/master/TimberWinR/mdocs/WindowsEvents.md)
## Filters ## Filters
The current list of supported filters are: The current list of supported filters are:
1. [Grok](https://github.com/efontana/TimberWinR/blob/master/mdocs/GrokFilter.md) 1. [Grok](https://github.com/efontana/TimberWinR/blob/master/TimberWinR/mdocs/GrokFilter.md)
2. [Mutate](https://github.com/efontana/TimberWinR/blob/master/mdocs/MutateFilter.md) 2. [Mutate](https://github.com/efontana/TimberWinR/blob/master/TimberWinR/mdocs/MutateFilter.md)
3. [Date](https://github.com/efontana/TimberWinR/blob/master/TimberWinR/mdocs/DateFilter.md)
## JSON ## JSON
Since TimberWinR only ships to Redis, the format generated by TimberWinR is JSON. All fields referenced by TimberWinR can be Since TimberWinR only ships to Redis, the format generated by TimberWinR is JSON. All fields referenced by TimberWinR can be

6
TimberWinR.ServiceHost/Program.cs
View File

@@ -34,8 +34,7 @@ namespace TimberWinR.ServiceHost
serviceConfigurator.WhenStopped(myService => myService.Stop()); serviceConfigurator.WhenStopped(myService => myService.Stop());
}); });
hostConfigurator.AddCommandLineDefinition("configFile", c => arguments.ConfigFile = c); hostConfigurator.AddCommandLineDefinition("configFile", c => arguments.ConfigFile = c);
hostConfigurator.AddCommandLineDefinition("jsonFile", c => arguments.JsonFile = c);
hostConfigurator.ApplyCommandLine(); hostConfigurator.ApplyCommandLine();
hostConfigurator.RunAsLocalSystem(); hostConfigurator.RunAsLocalSystem();
@@ -51,7 +50,6 @@ namespace TimberWinR.ServiceHost
internal class Arguments internal class Arguments
{ {
public string ConfigFile { get; set; } public string ConfigFile { get; set; }
public string JsonFile { get; set; }
public Arguments() public Arguments()
{ {
@@ -95,7 +93,7 @@ namespace TimberWinR.ServiceHost
/// </summary> /// </summary>
private void RunService() private void RunService()
{ {
_manager = new TimberWinR.Manager(_args.ConfigFile, _args.JsonFile, _cancellationToken); _manager = new TimberWinR.Manager(_args.ConfigFile, _cancellationToken);
} }
} }
} }

6
TimberWinR.ServiceHost/TimberWinR.ServiceHost.csproj
View File

@@ -63,12 +63,6 @@
<Name>TimberWinR</Name> <Name>TimberWinR</Name>
</ProjectReference> </ProjectReference>
</ItemGroup> </ItemGroup>
<ItemGroup>
<Content Include="sampleconf.xml">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<SubType>Designer</SubType>
</Content>
</ItemGroup>
<Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" /> <Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
<!-- To modify your build process, add your task inside one of the targets below and uncomment it. <!-- To modify your build process, add your task inside one of the targets below and uncomment it.
Other similar extension points exist, see Microsoft.Common.targets. Other similar extension points exist, see Microsoft.Common.targets.

32
TimberWinR.ServiceHost/sampleconf.xml
View File

@@ -1,32 +0,0 @@
<?xml version="1.0" encoding="utf-8"?>
<TimberWinR>
<Inputs>
<WindowsEvents>
<Event source="System,Application" binaryFormat="PRINT" />
</WindowsEvents>
<Logs>
<Log name="Syslogs" location="C:\Logs1\*.log" />
</Logs>
<IISW3CLogs>
<IISW3CLog name="Default site" location="c:\inetpub\logs\LogFiles\W3SVC1\*" />
</IISW3CLogs>
</Inputs>
<Filters>
<Grok>
<!--Single Tag-->
<Match field="Text" value="%{SYSLOGLINE}" />
<AddTag>rn_%{RecordNumber}</AddTag>
<AddTag>bar</AddTag>
</Grok>
<Mutate>
<Rename oldName="TimeGenerated" newName="timestamp"/>
</Mutate>
<Date field="timestamp" target="@timestamp" convertToUTC="true">
<Pattern>MMM d HH:mm:ss</Pattern>
<Pattern>MMM dd HH:mm:ss</Pattern>
<Pattern>ISO8601</Pattern>
</Date>
</Filters>
</TimberWinR>

6
TimberWinR.UnitTests/TimberWinR.UnitTests.csproj
View File

@@ -62,12 +62,6 @@
<ItemGroup> <ItemGroup>
<Service Include="{82A7F48D-3B50-4B1E-B82E-3ADA8210C358}" /> <Service Include="{82A7F48D-3B50-4B1E-B82E-3ADA8210C358}" />
</ItemGroup> </ItemGroup>
<ItemGroup>
<Content Include="testconf.xml">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
<SubType>Designer</SubType>
</Content>
</ItemGroup>
<Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" /> <Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
<!-- To modify your build process, add your task inside one of the targets below and uncomment it. <!-- To modify your build process, add your task inside one of the targets below and uncomment it.
Other similar extension points exist, see Microsoft.Common.targets. Other similar extension points exist, see Microsoft.Common.targets.

27
TimberWinR.UnitTests/testconf.xml
View File

@@ -1,27 +0,0 @@
<?xml version="1.0" encoding="utf-8"?>
<TimberWinR>
<Inputs>
<WindowsEvents>
<Event source="System,Application" binaryFormat="PRINT" />
</WindowsEvents>
<Logs>
<Log name="First Set" location="C:\Logs1\*.log" />
<Log name="Second Set" location="C:\Logs2\*.log" recurse="0" splitLongLines="OFF" />
<Log name="Third Set" location="C:\Logs2\1.log,C:\Logs2\2.log" />
</Logs>
<IISW3CLogs>
<IISW3CLog name="Default site" location="c:\inetpub\logs\LogFiles\W3SVC1\*" />
</IISW3CLogs>
</Inputs>
<Filters>
<Grok>
<Match field="Text" value="%{IPAddress:ip1} %{IPAddress:ip2}" />
<AddField name="field1" value="%{foo}" />
<DropIfMatch value="true" />
<RemoveField value="ip1" />
</Grok>
</Filters>
</TimberWinR>

4
TimberWinR.sln
View File

@@ -16,11 +16,7 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "TimberWinR.UnitTests", "Tim
EndProject EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution Items", "{1B2F600B-2400-45B9-A28E-CFC391D9EFA9}" Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution Items", "{1B2F600B-2400-45B9-A28E-CFC391D9EFA9}"
ProjectSection(SolutionItems) = preProject ProjectSection(SolutionItems) = preProject
mdocs\DateFilter.md = mdocs\DateFilter.md
Filters.md = Filters.md
mdocs\GrokFilter.md = mdocs\GrokFilter.md
LICENSE.txt = LICENSE.txt LICENSE.txt = LICENSE.txt
mdocs\MutateFilter.md = mdocs\MutateFilter.md
Package.nuspec = Package.nuspec Package.nuspec = Package.nuspec
README.md = README.md README.md = README.md
WindowsEvents.md = WindowsEvents.md WindowsEvents.md = WindowsEvents.md

2
TimberWinR/Manager.cs
View File

@@ -28,7 +28,7 @@ namespace TimberWinR
listener.Shutdown(); listener.Shutdown();
} }
public Manager(string xmlConfigFile, string jsonConfigFile, CancellationToken cancelToken) public Manager(string jsonConfigFile, CancellationToken cancelToken)
{ {
Outputs = new List<OutputSender>(); Outputs = new List<OutputSender>();
Listeners = new List<InputListener>(); Listeners = new List<InputListener>();

5
TimberWinR/TimberWinR.csproj
View File

@@ -98,6 +98,11 @@
<None Include="configSchema.xsd"> <None Include="configSchema.xsd">
<SubType>Designer</SubType> <SubType>Designer</SubType>
</None> </None>
<None Include="mdocs\DateFilter.md" />
<None Include="mdocs\Filters.md" />
<None Include="mdocs\GrokFilter.md" />
<None Include="mdocs\MutateFilter.md" />
<None Include="mdocs\WindowsEvents.md" />
<None Include="packages.config" /> <None Include="packages.config" />
</ItemGroup> </ItemGroup>
<ItemGroup> <ItemGroup>

1
TimberWinR/mdocs/DateFilter.md Normal file
View File

@@ -0,0 +1 @@
# Date Filter

69
TimberWinR/mdocs/Filters.md Normal file
View File

@@ -0,0 +1,69 @@
# Filters
The following filters are provided.
| Filter | Description
| :---------------- |:-----------------------------------------------------------------------
| *[grok][4]* |Similar to the [logstash grok][1] filter
| *[date][5]* |Similar to the [logstash date][2] filter
| *[mutate][6]* |Similar to the [logstash mutate][3] filter
Example Input:
```json
"Filters": [
{
"grok": {
"condition": "[type] == \"Win32-Eventlog\"",
"match": [
"Message",
""
],
"remove_field": [
"ComputerName"
]
}
},
{
"grok": {
"match": [
"message",
"%{SYSLOGLINE}"
],
"add_field": [
"Hello", "from %{logsource}"
]
}
},
{
"date": {
"condition": "[type] == \"Win32-FileLog\"",
"match": [
"timestamp",
"MMM d HH:mm:sss",
"MMM dd HH:mm:ss"
],
"add_field": [
"UtcTimestamp"
],
"convertToUTC": true
}
},
{
"mutate": {
"_comment": "Custom Rules",
"rename": [
"ComputerName", "Host",
"host", "Host",
"message","Message",
"type","Type",
"SID", "Username"
]
}
}
]
```
[1]: http://logstash.net/docs/1.4.2/filters/grok
[2]: http://logstash.net/docs/1.4.2/filters/date
[3]: http://logstash.net/docs/1.4.2/filters/mutate
[4]: https://github.com/efontana/TimberWinR/blob/master/mdocs/GrokFilter.md
[5]: https://github.com/efontana/TimberWinR/blob/master/mdocs/DateFilter.md
[6]: https://github.com/efontana/TimberWinR/blob/master/mdocs/MutateFilter.md

155
TimberWinR/mdocs/GrokFilter.md Normal file
View File

@@ -0,0 +1,155 @@
# Grok Filter
The grok filter allows you to parse unstructured data into something structured and
queryable. The very same patterns used by logstash are supported in TimberWinR.
See [Grok Patterns](https://github.com/elasticsearch/logstash/blob/v1.4.2/patterns/grok-patterns)
## Grok Basics
Grok works by combining text patterns into something that matches your logs.
The syntax for a grok pattern is %{SYNTAX:SEMANTIC}
The SYNTAX is the name of the pattern that will match your text. For example, “3.44” will be matched by the NUMBER pattern and “55.3.244.1”
will be matched by the IP pattern. The syntax is how you match.
The SEMANTIC is the identifier you give to the piece of text being matched. For example, “3.44” could be the duration of an event, so you could
call it simply ‘duration’. Further, a string “55.3.244.1” might identify the ‘client’
making a request.
For the above example, your grok filter would look something like this:
%{NUMBER:duration} %{IP:client}
## Grok Operations
The following operations are allowed when mutating a field.
| Operation | Type | Description
| :---------------|:----------------|:-----------------------------------------------------------------------|
| *condition* | property:string |C# expression
| *match* | property:string |Required field must match before any subsequent grok operations are executed.
| *add_field* | property:array |If the filter is successful, add an arbitrary field to this event. Field names can be dynamic and include parts of the event using the %{field} syntax. This property must be specified in pairs.
| *remove_field* | property:array |If the filter is successful, remove arbitrary fields from this event. Field names can be dynamic and include parts of the event using the %{field} syntax.
| *add_tag* | property:array |If the filter is successful, add an arbitrary tag to this event. Tag names can be dynamic and include parts of the event using the %{field} syntax.
| *remove_tag* | property:array |If the filter is successful, remove arbitrary tags from this event. Field names can be dynamic and include parts of the event using the %{field} syntax.
## Operation Details
### match
The match field is required, the first argument is the field to inspect, and compare to the expression specified by the second
argument. In the below example, the message is spected to be something like this from a fictional sample log:
```
55.3.244.1 GET /index.html 15824 0.043
```
The pattern for this could be:
```
%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}
```
Given this configuration
```json
"Filters": [
{
"grok": {
"match": [
"message",
"%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}"
],
"add_tag": [
"http_log"
],
"add_field": [
"verb", "%{method}"
]
}
}
]
```
And if the message matches, then 6 fields would be added to the event:
1. client=55.3.244.1
2. method=GET
3. request=/index.html
4. bytes=15824
5. duration=0.043
6. verb=GET
And the following tag will be added
1. tag: { "http_log" }
### condition "C# expression"
If present, the condition must evaluate to true in order for the remaining operations to be performed. If there is no condition specified
then the operation(s) will be executed in order.
```json
"Filters": [
{
"grok": {
"condition": "[type] == \"Win32-EventLog\""
"add_field": [
"ComputerName", "%{Host}"
]
}
}
]
```
The above example will add a field ComputerName set to the value of Host only for Win32-EventLog types.
### add_field ["fieldName", "fieldValue", ...]
The fields must be in pairs with fieldName first and value second.
```json
"Filters": [
{
"grok": {
"add_field": [
"ComputerName", "Host",
"Username", "%{SID}"
]
}
}
]
```
### remove_field ["tag1", "tag2", ...]
Remove the fields. More than one field can be specified at a time.
```json
"Filters": [
{
"grok": {
"remove_tag": [
"static_tag1",
"Computer_%{Host}"
]
}
}
]
```
### add_tag ["tag1", "tag2", ...]
Adds the tag(s) to the tag array.
```json
"Filters": [
{
"grok": {
"add_tag": [
"foo_%{Host}",
"static_tag1"
]
}
}
]
```
### remove_tag ["tag1", "tag2", ...]
Remove the tag(s) to the tag array. More than one tag can be specified at a time.
```json
"Filters": [
{
"grok": {
"remove_tag": [
"static_tag1",
"Username"
]
}
}
]
```

77
TimberWinR/mdocs/MutateFilter.md Normal file
View File

@@ -0,0 +1,77 @@
# Mutate Filter
The mutate filter allows you to perform general mutations on fields. You can rename, remove, replace and modify fields in your events. This filter will automatically be applied to all inputs before sending to the outputs. If you want to make a
filter conditional, use the ***condition*** property to specify a legal C# expression.
## Mutate Operations
The following operations are allowed when mutating a field.
| Operation | Type | Description
| :-----------|:----------------|:-----------------------------------------------------------------------|
| *condition* | property:string |C# Expression
| *rename* | property:array |Rename one or more fields
| *replace* | property:array |Replace a field with a new value. The new value can include %{foo} strings to help you build a new value from other parts of the event.
| *split* | property:array |Separator between values of the "Strings" field.
## Details
### condition "C# expression"
If present, the condition must evaluate to true in order for the remaining operations to be performed. If there is no condition specified
then the operation(s) will be executed in order.
```json
"Filters": [
{
"mutate": {
"condition": "[type] == \"Win32-EventLog\""
"rename": [
"ComputerName", "Host"
]
}
}
]
```
The above example will rename ComputerName to Host only for Win32-EventLog types.
### rename ["oldname", "newname", ...]
The fields must be in pairs with oldname first and newname second.
```json
"Filters": [
{
"mutate": {
"rename": [
"ComputerName", "Host",
"host", "Host",
"message","Message",
"type","Type",
"SID", "Username"
]
}
}
]
```
### replace ["field", "newvalue", ...]
Replaces field with newvalue. The replacements must be described in pairs.
```json
"Filters": [
{
"mutate": {
"replace": [
"message", "%{source_host}: My new message"
]
}
}
]
```
### split
Split a field into an array of values. The first arguments is the fieldName and the second is the separator.
```json
"Filters": [
{
"mutate": {
"split": [
"InsertionStrings", "|"
]
}
}
]
```

33
TimberWinR/mdocs/WindowsEvents.md Normal file
View File

@@ -0,0 +1,33 @@
# Input: WindowsEvents
## Parameters
The following parameters are allowed when configuring WindowsEvents.
| Parameter | Type | Description | Legal Values | Default |
| :---------------- |:---------------| :----------------------------------------------------------------------- | :--------------------------- | :-- |
| *source* | property:string |Windows event logs | Application,System,Security | System |
| *binaryFormat* | property:string |Format of the "Data" binary field. | ASC,HEX,PRINT | **ASC** |
| *msgErrorMode* | property:string |Behavior when event messages or event category names cannot be resolved. |NULL,ERROR,MSG | **MSG** |
| *direction* | property:string |Format of the "Data" binary field. | FW,BW | **FW** |
| *stringsSep* | property:string |Separator between values of the "Strings" field. | any string | vertical bar |
| *fullEventCode* | property:bool |Return the full event ID code instead of the friendly code. | true,false | **false** |
| *fullText* | property:bool |Retrieve the full text message | true,false | **true** |
| *resolveSIDS* | property:bool |Resolve SID values into full account names | true,false | **true** |
| *formatMsg* | property:bool |Format the text message as a single line. | true,false | **true** |
Example Input:
```json
{
"TimberWinR": {
"Inputs": {
"WindowsEvents": [
{
"source": "System,Application",
"binaryFormat": "PRINT",
"resolveSIDS": true
}
]
}
}
}
```