Move redisHostIndex inc to a finally

It looks like this will just retry connections to the same host if new RedisClient throws.  Move _redisHostIndex++ to a finally.

Should we also be logging when these happen?

.gitignore

@@ -154,3 +154,4 @@ $RECYCLE.BIN/
 
 # Mac desktop service store files
 .DS_Store
+packages

LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright 2014 Vistaprint
+Copyright 2014 Cimpress
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -10,4 +10,4 @@ Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
-limitations under the License.
+limitations under the License.

Performance1.psess

@@ -0,0 +1,21 @@
+<VSPerformanceSession Version="1.00">
+  <Options>
+    <CollectionMethod>Sampling</CollectionMethod>
+    <AllocationMethod>None</AllocationMethod>
+    <AddReport>true</AddReport>
+    <UniqueReport>Timestamp</UniqueReport>
+    <SamplingMethod>Cycles</SamplingMethod>
+    <CycleCount>10000000</CycleCount>
+    <PageFaultCount>10</PageFaultCount>
+    <SysCallCount>10</SysCallCount>
+    <SamplingCounter PlatformID="00000000" CounterID="0000000000000000" ReloadValue="000000000000000a" />
+    <RelocateBinaries>false</RelocateBinaries>
+    <HardwareCounters EnableHWCounters="false" />
+  </Options>
+  <PreinstrumentEvent>
+    <InstrEventExclude>false</InstrEventExclude>
+  </PreinstrumentEvent>
+  <PostinstrumentEvent>
+    <InstrEventExclude>false</InstrEventExclude>
+  </PostinstrumentEvent>
+</VSPerformanceSession>

README.md

@@ -7,35 +7,105 @@ TimberWinR is a native .NET implementation utilizing Microsoft's [LogParser](htt
 no JVM/JRuby is required, and LogParser does all the heavy lifting.  TimberWinR collects
 the data from LogParser and ships it to Logstash via Redis (or can ship direcly to Elasticsearch)
 
+## Release Notes
+
+[View Version History](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/ReleaseNotes.md)
+
 ## Basics
 TimberWinR uses a configuration file to control how the logs are collected, filtered and shipped off.  
 These are broken down into:
  1. Inputs  (Collect data from different sources)
  2. Filters (Are applied to all Inputs)
- 3. Outputs (Currently ships only to Redis)
+ 3. Outputs (Redis, Elasticsearch or Stdout)
 
-## Input Formats
+### Support ###
+Please use the TimberWinR Google Group for discussion and support:
+
+https://groups.google.com/forum/#!forum/timberwinr
+
+Latest Build:
+
+![alt tag](https://ci.appveyor.com/api/projects/status/github/Cimpress-MCP/TimberWinR)
+
+## Inputs
 The current supported Input format sources are:
- 1. [Logs](https://github.com/efontana/TimberWinR/blob/master/TimberWinR/mdocs/Logs.md) (Files, a.k.a Tailing a file)
- 2. [Tcp](https://github.com/efontana/TimberWinR/blob/master/TimberWinR/mdocs/TcpInput.md) (listens on a port for JSON messages)
- 3. [IISW3C](https://github.com/efontana/TimberWinR/blob/master/TimberWinR/mdocs/IISW3CInput.md)(Internet Information Services W3C Format)
- 4. [WindowsEvents](https://github.com/efontana/TimberWinR/blob/master/TimberWinR/mdocs/WindowsEvents.md) (Windows Event Viewer)
- 5. [Stdin](https://github.com/efontana/TimberWinR/blob/master/TimberWinR/mdocs/StdinInput.md) (Standard Input for Debugging)
+ 1. [Logs](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/Logs.md) (Files, a.k.a Tailing a file)
+ 2. [Tcp](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/TcpInput.md) (listens on TCP port for JSON messages)
+ 3. [IISW3C](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/IISW3CInput.md)(Internet Information Services W3C Format)
+ 4. [WindowsEvents](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/WindowsEvents.md) (Windows Event Viewer)
+ 5. [Stdin](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/StdinInput.md) (Standard Input for Debugging)
+ 6. [W3C](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/W3CInput.md)(Internet Information Services W3C Advanced/Custom Format)
+ 7. [Udp](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/UdpInput.md) (listens for UDP on port for JSON messages)
+ 8. [TailFiles](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/TailFiles.md) (Tails log files efficiently *New*)
+
+## Codecs
+The current list of supported codecs are:
+ 1. [Multiline](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/Codec.md)
 
 ## Filters
 The current list of supported filters are:
- 1. [Grok](https://github.com/efontana/TimberWinR/blob/master/TimberWinR/mdocs/GrokFilter.md)
- 2. [Mutate](https://github.com/efontana/TimberWinR/blob/master/TimberWinR/mdocs/MutateFilter.md)
- 3. [Date](https://github.com/efontana/TimberWinR/blob/master/TimberWinR/mdocs/DateFilter.md)
+ 1. [Grok](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/GrokFilter.md)
+ 2. [Mutate](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/MutateFilter.md)
+ 3. [Date](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/DateFilter.md)
+ 4. [Json](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/JsonFilter.md)
+ 5. [GeoIP](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/GeoIPFilter.md)
+
+Note that there are now two syntaxes for filters, array and single, if you want more than one filter of the same
+type then you must use the array tag instead of the singular tag. i.e:
+
+A single Json filter using the single tag (this is only provided as a convienience, the array syntax is preferred)
+```json
+"Filters": [          
+    {
+        "json": {
+            "type": "Win32-FileLog",
+            "condition": "\"[logSource]\" == \"dev\"",
+            "source": "Text",
+            "add_field": [
+                "_index",
+                "dev-%{yyyy.MM.dd}"
+            ]
+        }
+    }
+]
+```
+
+Multiple Json filters must use the jsonFilters and array syntax, also mutateFilters, grokFilters, dateFilters, geoipFilters.
+```json
+"Filters": [
+    {
+        "jsonFilters": [
+            {
+                "type": "Win32-FileLog",
+                "condition": "\"[logSource]\" == \"dev\"",
+                "source": "Text",
+                "add_field": [
+                    "_index",
+                    "dev-%{yyyy.MM.dd}"
+                ]
+            },
+            {
+                "type": "Win32-FileLog",
+                "condition": "\"[logSource]\" == \"sta\"",
+                "source": "Text",
+                "add_field": [
+                    "_index",
+                    "sta-%{yyyy.MM.dd}"
+                ]
+            }
+        ]
+    }
+]
+```
 
 ## JSON
-Since TimberWinR only ships to Redis, the format generated by TimberWinR is JSON.  All fields referenced by TimberWinR can be
+Since TimberWinR only ships to Redis and Elasticsearch, the format generated by TimberWinR is JSON.  All fields referenced by TimberWinR can be
 represented as a JSON Property or Array.
 
-## Supported Output Formats
-1. [Redis](https://github.com/efontana/TimberWinR/blob/master/TimberWinR/mdocs/RedisOutput.md)
-2. [Elasticsearch](https://github.com/efontana/TimberWinR/blob/master/TimberWinR/mdocs/ElasticsearchOutput.md)
-3. [Stdout](https://github.com/efontana/TimberWinR/blob/master/TimberWinR/mdocs/StdoutOutput.md)
+## Outputs
+1. [Redis](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/RedisOutput.md)
+2. [Elasticsearch](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/ElasticsearchOutput.md)
+3. [Stdout](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/StdoutOutput.md)
 
 ## Sample Configuration
 TimberWinR reads a JSON configuration file, an example file is shown here:
@@ -54,7 +124,7 @@ TimberWinR reads a JSON configuration file, an example file is shown here:
     "Filters": [          
         {
             "grok": {
-                "condition": "[type] == \"Win32-Eventlog\"",
+                "condition": "\"[type]\" == \"Win32-Eventlog\"",
                 "match": [
                     "Message",
                     ""
@@ -83,12 +153,14 @@ This configuration:
  2. Filters: Removes the ComputerName field
  3. Sends the event to Redis services (server1.host.com, server2.host.com) in a shuffling manner (balanced).
 
-## Installation
-You must first install LogParser, then install TimberWinR.   Install LogParser from here:
+## Installation from Source
+If installing from source, you must first install LogParser, then install TimberWinR.   Install LogParser from here:
 
 [Install LogParser](http://www.microsoft.com/en-us/download/details.aspx?id=24659) from Microsoft.
 
-After installing, follow the remaining directions here.
+After installing, follow the remaining directions here.   If you install from [Chocolatey](https://chocolatey.org/packages/TimberWinR) then LogParser will automatically
+be installed first.
+
 ## Running Interactively
 You can run TimberWinR interactively when you are developing your JSON config file, to do so use the
 following options:
@@ -96,7 +168,13 @@ following options:
 TimberWinR.ServiceHost.exe -configFile:myconfig.json -logLevel:Debug
 ```
 
-## Installation as a Windows Service
+## Automatic Installation via Chocolatey
+
+[TimbeWinR Chocolatey](https://chocolatey.org/packages/TimberWinR)
+
+![alt tag](https://raw.github.com/Cimpress-MCP/TimberWinR/master/chocolatey.png)
+
+## Manual Installation as a Windows Service
 TimberWinR uses [TopShelf](http://topshelf-project.com/) to install as a service, so all the documentation
 for installing and configuring the service is show here [TopShelf Doc](http://docs.topshelf-project.com/en/latest/)
 
@@ -122,13 +200,16 @@ Alternatively you can use the Services Control Panel.
 TimberWinR.ServiceHost.exe [options]
 
 Options:
--logDir:        Specifies the directory where TimberWinR will write its log file TimberWinR.txt
-                Default is -logDir:"C:\logs"
--logLevel:      Specifies the logging level for TimberWinR
-                Legal Values: Trace|Debug|Info|Warn|Error|Fatal|Off
-                Default is -logDir:Info
--configFile:    Specifies the path to the JSON config file, or directory which contains .json file(s).
-                Default is -configFile:default.json
+-logDir:           Specifies the directory where TimberWinR will write its log file TimberWinR.txt
+                   Default is -logDir:"C:\logs"
+-logLevel:         Specifies the logging level for TimberWinR
+                   Legal Values: Trace|Debug|Info|Warn|Error|Fatal|Off
+                   Default is -logDir:Info
+-configFile:       Specifies the path to the JSON config file, or directory which contains .json file(s).
+                   Default is -configFile:default.json
+-diagnosticPort:   Specifies the diagnostic port which can be used to get a health check of the service.
+                   Default Port is 5141, A value of 0 will disable it.  Open a browser
+                      http://localhost:5141
 ```
 #### -configFile
 This may be a single .json file or a directory containing .json file(s).  If it is a directory, all
@@ -139,8 +220,8 @@ order on disk.
 If you really just want to try it out, grab the binary distribution, extract the .zip file
 into a directory, e.g.  C:\TimberWinR
 
-Grab the [JSON example file](https://github.com/efontana/TimberWinR/blob/master/TimberWinR.ServiceHost/default.json) and place it into C:\TimberWinR\default.json. 
-Edit the default.json file and change the Redis instance to match yours, replace 'tstlexiceapp006.vistaprint.svc' with the IP or DNS name
+Grab the [JSON example file](https://raw.githubusercontent.com/efontana/TimberWinR/master/TimberWinR.ServiceHost/default.json) and place it into C:\TimberWinR\default.json. 
+Edit the default.json file and change the Redis instance to match yours, replace 'tstlexiceapp006.mycompany.svc' with the IP or DNS name
 of the machine running redis.  Fire up the collector, enable the verbose debugging to see some Windows Events.
 
 ```
@@ -156,7 +237,26 @@ TimberWinR.ServiceHost.exe install --autostart
 TimberWinR.ServiceHost.exe start
 ```
 
-
-
+### Builds ###
+TimberWinR is distributed as an installable package via Chocolatey, and it is dependent on 
+
+[TimbeWinR Chocolatey](https://chocolatey.org/packages/TimberWinR)
+
+![alt tag](https://raw.github.com/Cimpress-MCP/TimberWinR/master/chocolatey.png)
+
+#### Notes ####
+After you install it via Chocolatey, you will have a folder:
+```
+C:\Program Files (x86)\TimberWinR
+```
+TimberWinR will be configured to read the file default.json located in the folder above.  You can edit this file
+to customize your installation.
+
+Use these commands to Stop/Start the service.
+
+```
+sc stop TimberWinR ; stop the service
+sc start TimberWinR; start the service
+```
 
 

TimberWinR.Builds/Readme.txt

@@ -0,0 +1,3 @@
+.MSI Packages Here
+------------------
+

TimberWinR.ExtractID/Program.cs

@@ -0,0 +1,107 @@
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Text;
+using Microsoft.Win32.SafeHandles;
+using System.Runtime.InteropServices;
+
+namespace TimberWinR.ExtractID
+{
+    class MsiHandle : SafeHandleMinusOneIsInvalid
+    {
+        public MsiHandle()
+            : base(true)
+        { }
+
+        protected override bool ReleaseHandle()
+        {
+            return NativeMethods.MsiCloseHandle(handle) == 0;
+        }
+    }
+
+    class NativeMethods
+    {
+        const string MsiDll = "Msi.dll";
+
+        [DllImport(MsiDll, CharSet = CharSet.Unicode, ExactSpelling = true)]
+        public extern static uint MsiOpenPackageW(string szPackagePath, out MsiHandle product);
+
+        [DllImport(MsiDll, ExactSpelling = true)]
+        public extern static uint MsiCloseHandle(IntPtr hAny);
+
+        [DllImport(MsiDll, CharSet = CharSet.Unicode, ExactSpelling = true)]
+        static extern uint MsiGetProductPropertyW(MsiHandle hProduct, string szProperty, StringBuilder value, ref int length);
+
+
+        [DllImport(MsiDll, ExactSpelling = true)]
+        public static extern int MsiSetInternalUI(int value, IntPtr hwnd);
+
+        public static uint MsiGetProductProperty(MsiHandle hProduct, string szProperty, out string value)
+        {
+            StringBuilder sb = new StringBuilder(1024);
+            int length = sb.Capacity;
+            uint err;
+            value = null;
+            if (0 == (err = MsiGetProductPropertyW(hProduct, szProperty, sb, ref length)))
+            {
+                sb.Length = length;
+                value = sb.ToString();
+                return 0;
+            }
+
+            return err;
+        }
+    }
+
+
+
+    class Program
+    {
+        [STAThread]
+        static int Main(string[] args)
+        {
+            if (args.Length < 2)
+            {
+                Console.Error.WriteLine("Expecting MSI and Tempolate file arguments");
+                return 1;
+            }
+
+            string msiDirectory = args[0];
+            string updateFile = args[1];
+            string newFile = args[2];
+          
+            string msiFile = Directory.GetFiles(msiDirectory, "TimberWinR*.msi").FirstOrDefault();
+
+            NativeMethods.MsiSetInternalUI(2, IntPtr.Zero); // Hide all UI. Without this you get a MSI dialog
+
+            MsiHandle msi;
+            uint err;
+            if (0 != (err = NativeMethods.MsiOpenPackageW(msiFile, out msi)))
+            {
+                Console.Error.WriteLine("Can't open MSI, error {0}", err);
+                return 1;
+            }
+
+            // Strings available in all MSIs
+            string productCode;
+            using (msi)
+            {
+                if (0 != NativeMethods.MsiGetProductProperty(msi, "ProductCode", out productCode))
+                    throw new InvalidOperationException("Can't obtain product code");
+
+                string contents = File.ReadAllText(args[1]);
+
+                contents = contents.Replace("${PROJECTGUID}", productCode);
+
+                File.WriteAllText(args[2], contents);
+
+                Console.WriteLine("Updated {0} ProductID: {1}", args[2], productCode);
+
+                return 0;
+            }
+
+            Console.Error.WriteLine("Failed for some reason");
+        }
+    }
+}

TimberWinR.ExtractID/Properties/AssemblyInfo.cs

@@ -0,0 +1,36 @@
+using System.Reflection;
+using System.Runtime.CompilerServices;
+using System.Runtime.InteropServices;
+
+// General Information about an assembly is controlled through the following 
+// set of attributes. Change these attribute values to modify the information
+// associated with an assembly.
+[assembly: AssemblyTitle("TimberWinR.ExtractID")]
+[assembly: AssemblyDescription("")]
+[assembly: AssemblyConfiguration("")]
+[assembly: AssemblyCompany("")]
+[assembly: AssemblyProduct("TimberWinR.ExtractID")]
+[assembly: AssemblyCopyright("Copyright ©  2014")]
+[assembly: AssemblyTrademark("")]
+[assembly: AssemblyCulture("")]
+
+// Setting ComVisible to false makes the types in this assembly not visible 
+// to COM components.  If you need to access a type in this assembly from 
+// COM, set the ComVisible attribute to true on that type.
+[assembly: ComVisible(false)]
+
+// The following GUID is for the ID of the typelib if this project is exposed to COM
+[assembly: Guid("cfb670ee-743d-49c7-b2bf-456bac3a88ef")]
+
+// Version information for an assembly consists of the following four values:
+//
+//      Major Version
+//      Minor Version 
+//      Build Number
+//      Revision
+//
+// You can specify all the values or you can default the Build and Revision Numbers 
+// by using the '*' as shown below:
+// [assembly: AssemblyVersion("1.0.*")]
+[assembly: AssemblyVersion("1.0.0.0")]
+[assembly: AssemblyFileVersion("1.0.0.0")]

TimberWinR.ExtractID/TimberWinR.ExtractID.csproj

@@ -0,0 +1,55 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="12.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <Import Project="$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props" Condition="Exists('$(MSBuildExtensionsPath)\$(MSBuildToolsVersion)\Microsoft.Common.props')" />
+  <PropertyGroup>
+    <Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
+    <Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
+    <ProjectGuid>{99096939-E9DD-4499-883D-4726745A5843}</ProjectGuid>
+    <OutputType>Exe</OutputType>
+    <AppDesignerFolder>Properties</AppDesignerFolder>
+    <RootNamespace>TimberWinR.ExtractID</RootNamespace>
+    <AssemblyName>TimberWinR.ExtractID</AssemblyName>
+    <TargetFrameworkVersion>v4.0</TargetFrameworkVersion>
+    <FileAlignment>512</FileAlignment>
+  </PropertyGroup>
+  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
+    <PlatformTarget>AnyCPU</PlatformTarget>
+    <DebugSymbols>true</DebugSymbols>
+    <DebugType>full</DebugType>
+    <Optimize>false</Optimize>
+    <OutputPath>bin\Debug\</OutputPath>
+    <DefineConstants>DEBUG;TRACE</DefineConstants>
+    <ErrorReport>prompt</ErrorReport>
+    <WarningLevel>4</WarningLevel>
+  </PropertyGroup>
+  <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
+    <PlatformTarget>AnyCPU</PlatformTarget>
+    <DebugType>pdbonly</DebugType>
+    <Optimize>true</Optimize>
+    <OutputPath>bin\Release\</OutputPath>
+    <DefineConstants>TRACE</DefineConstants>
+    <ErrorReport>prompt</ErrorReport>
+    <WarningLevel>4</WarningLevel>
+  </PropertyGroup>
+  <ItemGroup>
+    <Reference Include="System" />
+    <Reference Include="System.Core" />
+    <Reference Include="System.Xml.Linq" />
+    <Reference Include="System.Data.DataSetExtensions" />
+    <Reference Include="Microsoft.CSharp" />
+    <Reference Include="System.Data" />
+    <Reference Include="System.Xml" />
+  </ItemGroup>
+  <ItemGroup>
+    <Compile Include="Program.cs" />
+    <Compile Include="Properties\AssemblyInfo.cs" />
+  </ItemGroup>
+  <Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
+  <!-- To modify your build process, add your task inside one of the targets below and uncomment it. 
+       Other similar extension points exist, see Microsoft.Common.targets.
+  <Target Name="BeforeBuild">
+  </Target>
+  <Target Name="AfterBuild">
+  </Target>
+  -->
+</Project>

TimberWinR.ServiceHost/Program.cs

@@ -21,6 +21,9 @@ namespace TimberWinR.ServiceHost
 {
     internal class Program
     {
+        const string KeyPath = @"HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\TimberWinR";
+        const string KeyName = "ImagePath";
+        
         private static void Main(string[] args)
         {
             Arguments arguments = new Arguments();
@@ -35,7 +38,8 @@ namespace TimberWinR.ServiceHost
                     serviceConfigurator.WhenStarted(myService => myService.Start());
                     serviceConfigurator.WhenStopped(myService => myService.Stop());
                 });
-               
+
+                hostConfigurator.AddCommandLineDefinition("liveMonitor", c => arguments.LiveMonitor = bool.Parse(c.ToString()));
                 hostConfigurator.AddCommandLineDefinition("configFile", c => arguments.ConfigFile = c);
                 hostConfigurator.AddCommandLineDefinition("logLevel", c => arguments.LogLevel = c);
                 hostConfigurator.AddCommandLineDefinition("logDir", c => arguments.LogfileDir = c);
@@ -50,16 +54,14 @@ namespace TimberWinR.ServiceHost
                 hostConfigurator.SetServiceName("TimberWinR");
 
                 hostConfigurator.AfterInstall(() =>
-                {
-                    const string keyPath = @"HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\TimberWinR";
-                    const string keyName = "ImagePath";
-
-                    var currentValue = Registry.GetValue(keyPath, keyName, "").ToString();
+                { 
+                    var currentValue = Registry.GetValue(KeyPath, KeyName, "").ToString();
                     if (!string.IsNullOrEmpty(currentValue))
                     {
                         AddServiceParameter("-configFile", arguments.ConfigFile);
                         AddServiceParameter("-logLevel", arguments.LogLevel);
                         AddServiceParameter("-logDir", arguments.LogfileDir);
+                        AddServiceParameter("-liveMonitor", arguments.LiveMonitor);
                         if (arguments.DiagnosticPort > 0)
                             AddServiceParameter("-diagnosticPort", arguments.DiagnosticPort);
                     }
@@ -68,33 +70,37 @@ namespace TimberWinR.ServiceHost
         }
 
         private static void AddServiceParameter(string paramName, string value)
-        {
-            string keyPath = @"HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\TimberWinR";
-            string keyName = "ImagePath";
-
-            string currentValue = Registry.GetValue(keyPath, keyName, "").ToString();
+        {         
+            string currentValue = Registry.GetValue(KeyPath, KeyName, "").ToString();
 
             if (!string.IsNullOrEmpty(paramName) && !currentValue.Contains(string.Format("{0} ", paramName)))           
             {
                 currentValue += string.Format(" {0} \"{1}\"", paramName, value.Replace("\\\\", "\\"));               
-                Registry.SetValue(keyPath, keyName, currentValue);               
+                Registry.SetValue(KeyPath, KeyName, currentValue);               
             }
         }
 
         private static void AddServiceParameter(string paramName, int value)
-        {
-            string keyPath = @"HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\TimberWinR";
-            string keyName = "ImagePath";
-
-            string currentValue = Registry.GetValue(keyPath, keyName, "").ToString();
+        {    
+            string currentValue = Registry.GetValue(KeyPath, KeyName, "").ToString();
 
             if (!string.IsNullOrEmpty(paramName) && !currentValue.Contains(string.Format("{0}:", paramName)))
             {
                 currentValue += string.Format(" {0}:{1}", paramName, value);
-                Registry.SetValue(keyPath, keyName, currentValue);
+                Registry.SetValue(KeyPath, KeyName, currentValue);
             }
         }
 
+        private static void AddServiceParameter(string paramName, bool value)
+        {
+            string currentValue = Registry.GetValue(KeyPath, KeyName, "").ToString();
+
+            if (!string.IsNullOrEmpty(paramName) && !currentValue.Contains(string.Format("{0}:", paramName)))
+            {
+                currentValue += string.Format(" {0} \"{1}\"", paramName, value.ToString());
+                Registry.SetValue(KeyPath, KeyName, currentValue);
+            }
+        }
     }
 
     internal class Arguments
@@ -103,9 +109,10 @@ namespace TimberWinR.ServiceHost
         public string LogLevel { get; set; }
         public string LogfileDir { get; set; }
         public int DiagnosticPort { get; set; }
-
+        public bool LiveMonitor { get; set; }
         public Arguments()
         {
+            LiveMonitor = false;
             DiagnosticPort = 5141;
             ConfigFile = "default.json";
             LogLevel = "Info";
@@ -151,7 +158,7 @@ namespace TimberWinR.ServiceHost
         /// </summary>
         private void RunService()
         {
-            _manager = new TimberWinR.Manager(_args.ConfigFile, _args.LogLevel, _args.LogfileDir, _cancellationToken);
+            _manager = new TimberWinR.Manager(_args.ConfigFile, _args.LogLevel, _args.LogfileDir, _args.LiveMonitor, _cancellationToken);
             if (_args.DiagnosticPort > 0)
                 _diags = new Diagnostics.Diagnostics(_manager, _cancellationToken, _args.DiagnosticPort);
         }

TimberWinR.ServiceHost/Properties/AssemblyInfo.cs

@@ -10,7 +10,7 @@ using System.Runtime.InteropServices;
 [assembly: AssemblyConfiguration("")]
 [assembly: AssemblyCompany("")]
 [assembly: AssemblyProduct("TimberWinR.ServiceHost")]
-[assembly: AssemblyCopyright("Copyright ©  2014")]
+[assembly: AssemblyCopyright("Copyright © 2014-2015")]
 [assembly: AssemblyTrademark("")]
 [assembly: AssemblyCulture("")]
 
@@ -32,5 +32,5 @@ using System.Runtime.InteropServices;
 // You can specify all the values or you can default the Build and Revision Numbers 
 // by using the '*' as shown below:
 // [assembly: AssemblyVersion("1.0.*")]
-[assembly: AssemblyVersion("1.1.0.0")]
-[assembly: AssemblyFileVersion("1.1.0.0")]
+[assembly: AssemblyVersion("1.3.20.0")]
+[assembly: AssemblyFileVersion("1.3.20.0")]

TimberWinR.ServiceHost/TimberWinR.ServiceHost.csproj

@@ -32,6 +32,9 @@
     <ErrorReport>prompt</ErrorReport>
     <WarningLevel>4</WarningLevel>
   </PropertyGroup>
+  <PropertyGroup>
+    <ApplicationIcon>timberwinr.ico</ApplicationIcon>
+  </PropertyGroup>
   <ItemGroup>
     <Reference Include="RapidRegex.Core, Version=1.0.0.0, Culture=neutral, processorArchitecture=MSIL">
       <SpecificVersion>False</SpecificVersion>
@@ -62,6 +65,7 @@
     <Compile Include="Properties\AssemblyInfo.cs" />
   </ItemGroup>
   <ItemGroup>
+    <Content Include="timberwinr.ico" />
     <None Include="App.config" />
     <Content Include="config.json">
       <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
@@ -72,7 +76,9 @@
     <None Include="loopback.json">
       <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
     </None>
-    <None Include="packages.config" />
+    <None Include="packages.config">
+      <SubType>Designer</SubType>
+    </None>
   </ItemGroup>
   <ItemGroup>
     <ProjectReference Include="..\TimberWinR\TimberWinR.csproj">

TimberWinR.ServiceHost/config.json

@@ -23,7 +23,7 @@
         "Filters": [          
             {
                 "grok": {
-                    "condition": "[type] == \"Win32-Eventlog\"",
+                    "condition": "\"[type]\" == \"Win32-Eventlog\"",
                     "match": [
                         "Message",
                         ""
@@ -63,7 +63,7 @@
             },
             {
              "date":  {
-                    "condition": "[type] == \"Win32-FileLog\"",
+                    "condition": "\"[type]\" == \"Win32-FileLog\"",
                     "match": [
                         "timestamp",
                          "MMM  d HH:mm:sss",
@@ -90,7 +90,7 @@
                     "interval": 5000, 
                     "batch_count":  500,              
                     "host": [
-                        "tstlexiceapp006.vistaprint.svc"
+                        "tstlexiceapp006.mycompany.svc"
                     ]
                 }
             ],            
@@ -99,7 +99,7 @@
                     "threads":  1,   
                     "interval": 5000,                          
                     "host": [
-                        "tstlexiceapp003.vistaprint.svc"
+                        "tstlexiceapp003.mycompany.svc"
                     ]
                 }
             ]                           

TimberWinR.ServiceHost/default.json

@@ -1,30 +1,42 @@
 {
     "TimberWinR": {
-        "Inputs": {           
+        "Inputs": {
             "WindowsEvents": [
                 {
-                    "source": "System,Application",
+                    "source": "Application,System",
                     "binaryFormat": "PRINT",
                     "resolveSIDS": true
                 }
             ],
-             "Tcp": [
+            "Tcp": [
                 {
-                    "_comment": "Output from NLog",   
+                    "_comment": "Output from NLog",
                     "port": 5140
                 }
             ]
-        },      
+        },
+        "Filters": [
+            {
+                "grok": {
+                    "condition": "[EventTypeName] == \"Information Event\"",
+                    "match": [
+                        "Text",
+                        ""
+                    ],
+                    "drop": "true"
+                }
+            }
+        ],
         "Outputs": {
             "Redis": [
-                { 
-                    "_comment":  "Change the host to your Redis instance",   
-                    "port":  6379,                           
+                {
+                    "_comment": "Change the host to your Redis instance",
+                    "port": 6379,
                     "host": [
-                        "tstlexiceapp006.vistaprint.svc"
+                        "logaggregator.mycompany.svc"
                     ]
                 }
             ]
-        }       
+        }
     }
 }

TimberWinR.UnitTests/Configuration.cs

@@ -85,7 +85,7 @@ namespace TimberWinR.UnitTests
                         [{  
                             ""host"": 
                             [
-                                ""logaggregator.vistaprint.svc""
+                                ""logaggregator.mycompany.svc""
                             ]                   
 		                }]
 	                }

TimberWinR.UnitTests/GeoIPFilterTests.cs

@@ -0,0 +1,58 @@
+using System;
+using System.Collections.Generic;
+using System.Diagnostics;
+using System.Linq;
+using System.Text;
+using System.Threading.Tasks;
+using Newtonsoft.Json;
+using NUnit.Framework;
+using TimberWinR.Parser;
+using Newtonsoft.Json.Linq;
+
+namespace TimberWinR.UnitTests
+{
+    [TestFixture]
+    public class GeoIPFilterTests
+    {       
+        [Test]
+        public void TestDropConditions()
+        {
+            JObject jsonInputLine1 = new JObject
+            {               
+                {"type", "Win32-FileLog"},
+                {"IP", "8.8.8.8"}              
+            };
+           
+
+            string jsonFilter = @"{  
+            ""TimberWinR"":{        
+                ""Filters"":[  
+	                {  
+		            ""geoip"":{  
+		                ""type"": ""Win32-FileLog"",
+		                ""target"": ""mygeoip"",
+                        ""source"": ""IP""            
+		            }
+	              }]
+                }
+            }";
+          
+            // Positive Tests
+            Configuration c = Configuration.FromString(jsonFilter);
+            GeoIP jf = c.Filters.First() as GeoIP;
+            Assert.IsTrue(jf.Apply(jsonInputLine1));
+
+            JObject stuff = jsonInputLine1["mygeoip"] as JObject;
+            Assert.IsNotNull(stuff);
+          
+            Assert.AreEqual("8.8.8.8", stuff["ip"].ToString());
+            Assert.AreEqual("US", stuff["country_code2"].ToString());
+            Assert.AreEqual("United States", stuff["country_name"].ToString());
+            Assert.AreEqual("CA", stuff["region_name"].ToString());
+            Assert.AreEqual("Mountain View", stuff["city_name"].ToString());
+            Assert.AreEqual("California", stuff["real_region_name"].ToString());
+            Assert.AreEqual(37.386f, (float)stuff["latitude"]);
+            Assert.AreEqual(-122.0838f, (float) stuff["longitude"]);
+        }
+    }
+}

TimberWinR.UnitTests/GrokFilterTests.cs

@@ -22,7 +22,7 @@ namespace TimberWinR.UnitTests
                 {"Index", 7},
                 {"Text", null},
                 {"type", "Win32-FileLog"},
-                {"ComputerName", "dev.vistaprint.net"}
+                {"ComputerName", "dev.mycompany.net"}
             };
 
             string grokJson = @"{  
@@ -30,7 +30,7 @@ namespace TimberWinR.UnitTests
                 ""Filters"":[  
 	                {  
 		            ""grok"":{  
-		                ""condition"": ""[type] == \""Win32-FileLog\"""",
+		                ""type"": ""Win32-FileLog"",
 		                ""match"":[  
 			                ""Text"",
 			                """"
@@ -55,7 +55,7 @@ namespace TimberWinR.UnitTests
             Assert.IsTrue(grok.Apply(json));
 
             // Verify host field added
-            Assert.AreEqual(json["host"].ToString(), "dev.vistaprint.net");
+            Assert.AreEqual(json["host"].ToString(), "dev.mycompany.net");
 
             // Verify two tags added
             Assert.AreEqual(json["tags"][0].ToString(), "rn_7");
@@ -69,9 +69,9 @@ namespace TimberWinR.UnitTests
             {
                 {"LogFilename", @"C:\\Logs1\\test1.log"},
                 {"Index", 7},
-                {"Text", null},
+                {"Text", "crap"},
                 {"type", "Win32-FileLog"},
-                {"ComputerName", "dev.vistaprint.net"}
+                {"ComputerName", "dev.mycompany.net"}
             };
 
             string grokJson = @"{  
@@ -79,10 +79,10 @@ namespace TimberWinR.UnitTests
                 ""Filters"":[  
 	                {  
 		            ""grok"":{  
-		                ""condition"": ""[type] == \""Win32-FileLog\"""",
+		                ""condition"": ""\""[type]\"" == \""Win32-FileLog\"""",
 		                ""match"":[  
 			                ""Text"",
-			                """"
+			                ""crap""
 		                ],
                         ""remove_field"":[  
                             ""Index"",
@@ -106,6 +106,108 @@ namespace TimberWinR.UnitTests
             Assert.IsNull(json["LogFilename"]);        
         }
 
+        [Test]
+        public void TestDropConditions()
+        {
+            JObject json1 = new JObject
+            {
+                {"LogFilename", @"C:\\Logs1\\test1.log"},
+                {"EventType", 1},
+                {"Index", 7},
+                {"Text", null},
+                {
+                    "tags", new JArray
+                    {
+                        "tag1",
+                        "tag2"
+                    }
+                },
+                {"type", "Win32-FileLog"},
+                {"ComputerName", "dev.mycompany.net"}
+            };
+
+            JObject json2 = new JObject
+            {
+                {"LogFilename", @"C:\\Logs1\\test1.log"},
+                {"EventType", 2},
+                {"Index", 7},
+                {"Text", null},
+                {
+                    "tags", new JArray
+                    {
+                        "tag1",
+                        "tag2"
+                    }
+                },
+                {"type", "Win32-FileLog"},
+                {"ComputerName", "dev.mycompany.net"}
+            };
+
+
+            string grokJsonDropIf1 = @"{  
+            ""TimberWinR"":{        
+                ""Filters"":[  
+	                {  
+		            ""grok"":{                          
+		                ""condition"": ""[EventType] == 1"",
+                        ""drop"": ""true"",
+		                ""match"":[  
+			                ""Text"",
+			                """"
+		                ],
+                        ""remove_tag"":[  
+                            ""tag1""                          
+                        ]		               
+		            }
+	                }]
+                }
+            }";
+
+            string grokJsonDropIfNot1 = @"{  
+            ""TimberWinR"":{        
+                ""Filters"":[  
+	                {  
+		            ""grok"":{                          
+		                ""condition"": ""[EventType] != 1"",
+                        ""drop"": ""true"",
+		                ""match"":[  
+			                ""Text"",
+			                """"
+		                ],
+                        ""remove_tag"":[  
+//                            ""tag1""                          
+                        ]		               
+		            }
+	                }]
+                }
+            }";
+
+
+            // Positive Tests
+            Configuration c = Configuration.FromString(grokJsonDropIf1);
+            Grok grok = c.Filters.First() as Grok;
+            Assert.IsFalse(grok.Apply(json1));
+
+            c = Configuration.FromString(grokJsonDropIf1);
+            grok = c.Filters.First() as Grok;
+            Assert.IsTrue(grok.Apply(json2));
+
+            // Negative Tests
+            c = Configuration.FromString(grokJsonDropIfNot1);
+            grok = c.Filters.First() as Grok;
+            Assert.IsFalse(grok.Apply(json2));
+
+            c = Configuration.FromString(grokJsonDropIfNot1);
+            grok = c.Filters.First() as Grok;
+            Assert.IsTrue(grok.Apply(json1));
+
+
+
+
+        }
+
+
+
         [Test]
         public void TestConditions()
         {
@@ -121,7 +223,7 @@ namespace TimberWinR.UnitTests
                     }
                 },
                 {"type", "Win32-FileLog"},
-                {"ComputerName", "dev.vistaprint.net"}
+                {"ComputerName", "dev.mycompany.net"}
             };
 
             string grokJson1 = @"{  
@@ -129,7 +231,7 @@ namespace TimberWinR.UnitTests
                 ""Filters"":[  
 	                {  
 		            ""grok"":{  
-		                ""condition"": ""[type] == \""Win32-FileLog\"""",
+		                ""condition"": ""\""[type]\"" == \""Win32-FileLog\"""",
 		                ""match"":[  
 			                ""Text"",
 			                """"
@@ -147,7 +249,7 @@ namespace TimberWinR.UnitTests
                 ""Filters"":[  
 	                {  
 		            ""grok"":{  
-		                ""condition"": ""[type].Contains(\""Win32-FileLog\"")"",
+		                ""condition"": ""\""[type]\"".Contains(\""Win32-FileLog\"")"",
 		                ""match"":[  
 			                ""Text"",
 			                """"
@@ -166,7 +268,7 @@ namespace TimberWinR.UnitTests
                 ""Filters"":[  
 	                {  
 		            ""grok"":{  
-		                ""condition"": ""[type].Contains(\""Win32-Filelog\"")"",
+		                ""condition"": ""\""[type]\"".Contains(\""Win32-Filelog\"")"",
 		                ""match"":[  
 			                ""Text"",
 			                """"
@@ -191,7 +293,7 @@ namespace TimberWinR.UnitTests
             // Negative Test
             c = Configuration.FromString(grokJson3);
             grok = c.Filters.First() as Grok;
-            Assert.IsFalse(grok.Apply(json));
+            Assert.IsTrue(grok.Apply(json));
         }
 
         [Test]
@@ -209,7 +311,7 @@ namespace TimberWinR.UnitTests
                     }
                 },
                 {"type", "Win32-FileLog"},
-                {"ComputerName", "dev.vistaprint.net"}
+                {"ComputerName", "dev.mycompany.net"}
             };
 
             string grokJson = @"{  
@@ -217,7 +319,7 @@ namespace TimberWinR.UnitTests
                 ""Filters"":[  
 	                {  
 		            ""grok"":{  
-		                ""condition"": ""[type] == \""Win32-FileLog\"""",
+		                ""condition"": ""\""[type]\"" == \""Win32-FileLog\"""",
 		                ""match"":[  
 			                ""Text"",
 			                """"
@@ -248,7 +350,7 @@ namespace TimberWinR.UnitTests
                 ""Filters"":[  
 	                {  
 		            ""grok"":{  
-		                ""condition"": ""[type] == \""Win32-FileLog\"""",
+		                ""condition"": ""\""[type]\"" == \""Win32-FileLog\"""",
 		                ""match"":[  
 			                ""Text"",
 			                """"
@@ -274,7 +376,7 @@ namespace TimberWinR.UnitTests
                 ""Filters"":[  
 	                {  
 		            ""grok"":{  
-		                ""condition"": ""[type] == \""Win32-FileLog\"""",
+		                ""condition"": ""\""[type]\"" == \""Win32-FileLog\"""",
 		                ""match"":[  
 			                ""Text"",
 			                """"
@@ -300,7 +402,7 @@ namespace TimberWinR.UnitTests
                 ""Filters"":[  
 	                {  
 		            ""grok"":{  
-		                ""condition"": ""[type] == \""Win32-FileLog\"""",
+		                ""condition"": ""\""[type]\"" == \""Win32-FileLog\"""",
 		                ""match"":[  
 			                ""Text"",
 			                """"
@@ -330,7 +432,7 @@ namespace TimberWinR.UnitTests
                 ""Filters"":[  
 	                {  
 		            ""grok"":{  
-		                ""condition"": ""[type] == \""Win32-FileLog\"""",
+		                ""condition"": ""\""[type]\"" == \""Win32-FileLog\"""",
 		                ""match"":[  
 			                ""Text"",
 			                """"

TimberWinR.UnitTests/Inputs/IisW3CRowReaderTests.cs

@@ -0,0 +1,60 @@
+namespace TimberWinR.UnitTests.Inputs
+{
+    using System;
+    using System.Collections.Generic;
+
+    using Interop.MSUtil;
+
+    using Moq;
+
+    using NUnit.Framework;
+
+    using TimberWinR.Inputs;
+    using TimberWinR.Parser;
+
+    [TestFixture]
+    public class IisW3CRowReaderTests : TestBase
+    {
+        private IisW3CRowReader reader;
+
+        public override void Setup()
+        {
+            base.Setup();
+            var fields = new List<Field>
+                             {
+                                 new Field("date", "DateTime"),
+                                 new Field("time", "DateTime"),
+                                 new Field("uri")
+                             };
+            this.reader = new IisW3CRowReader(fields);
+
+            var recordset = this.GetRecordsetMock();
+            this.reader.ReadColumnMap(recordset.Object);
+        }
+
+        [Test]
+        public void GivenValidRowAddsTimestampColumn()
+        {
+            var record = this.MockRepository.Create<ILogRecord>();
+            record.Setup(x => x.getValue("date")).Returns(new DateTime(2014, 11, 30));
+            record.Setup(x => x.getValue("time")).Returns(new DateTime(1, 1, 1, 18, 45, 37, 590));
+            record.Setup(x => x.getValue("uri")).Returns("http://somedomain.com/someurl");
+
+            var json = this.reader.ReadToJson(record.Object);
+
+            Assert.AreEqual("2014-11-30T18:45:37.000Z", json["@timestamp"].ToString());
+            Assert.AreEqual("http://somedomain.com/someurl", json["uri"].ToString());
+        }
+
+        private Mock<ILogRecordset> GetRecordsetMock()
+        {
+            var recordset = this.MockRepository.Create<ILogRecordset>();
+            recordset.Setup(x => x.getColumnCount()).Returns(3);
+
+            recordset.Setup(x => x.getColumnName(0)).Returns("date");
+            recordset.Setup(x => x.getColumnName(1)).Returns("time");
+            recordset.Setup(x => x.getColumnName(2)).Returns("uri");
+            return recordset;
+        }
+    }
+}

TimberWinR.UnitTests/JsonFilterTests.cs

@@ -0,0 +1,81 @@
+using System;
+using System.Collections.Generic;
+using System.Diagnostics;
+using System.Linq;
+using System.Text;
+using System.Threading.Tasks;
+using Newtonsoft.Json;
+using NUnit.Framework;
+using TimberWinR.Parser;
+using Newtonsoft.Json.Linq;
+
+namespace TimberWinR.UnitTests
+{
+    [TestFixture]
+    public class JsonFilterTests
+    {       
+        [Test]
+        public void TestDropConditions()
+        {
+            JObject jsonInputLine1 = new JObject
+            {               
+                {"type", "Win32-FileLog"},
+                {"ComputerName", "dev.mycompany.net"},
+                {"Text", "{\"Email\":\"james@example.com\",\"Active\":true,\"CreatedDate\":\"2013-01-20T00:00:00Z\",\"Roles\":[\"User\",\"Admin\"]}"}
+            };
+
+            JObject jsonInputLine2 = new JObject
+            {               
+                {"type", "Win32-FileLog"},
+                {"ComputerName", "dev.mycompany.net"},
+                {"Text", "{\"Email\":\"james@example.com\",\"Active\":true,\"CreatedDate\":\"2013-01-20T00:00:00Z\",\"Roles\":[\"User\",\"Admin\"]}"}
+            };
+
+
+            string jsonFilter = @"{  
+            ""TimberWinR"":{        
+                ""Filters"":[  
+	                {  
+		            ""json"":{  
+		                ""type"": ""Win32-FileLog"",
+		                ""target"": ""stuff"",
+                        ""source"": ""Text""            
+		            }
+	              }]
+                }
+            }";
+
+            string jsonFilterNoTarget = @"{  
+            ""TimberWinR"":{        
+                ""Filters"":[  
+	                {  
+		            ""json"":{  
+		                ""type"": ""Win32-FileLog"",		               
+                        ""source"": ""Text""            
+		            }
+	              }]
+                }
+            }";
+
+            // Positive Tests
+            Configuration c = Configuration.FromString(jsonFilter);
+            Json jf = c.Filters.First() as Json;
+            Assert.IsTrue(jf.Apply(jsonInputLine1));
+
+            JObject stuff = jsonInputLine1["stuff"] as JObject;
+            Assert.IsNotNull(stuff);
+
+            // 4 fields, Email, Active, CreatedDate, Roles
+            Assert.AreEqual(4, stuff.Count);
+
+            // Now, merge it into the root (starts as 3 fields, ends up with 7 fields)
+            Assert.AreEqual(3, jsonInputLine2.Count);
+            c = Configuration.FromString(jsonFilterNoTarget);
+            jf = c.Filters.First() as Json;
+            Assert.IsTrue(jf.Apply(jsonInputLine2));
+            JObject nostuff = jsonInputLine2["stuff"] as JObject;
+            Assert.IsNull(nostuff);
+            Assert.AreEqual(6, jsonInputLine2.Count);            
+        }
+    }
+}

TimberWinR.UnitTests/Multiline1.txt

@@ -0,0 +1,13 @@
+multiline1 \
+ml1_1 \
+ml1_2 \
+ml1_2 
+singleline1
+singleline2
+multiline2 \
+ml2_1 \
+ml2_2
+multiline3 \
+ml3_1 \
+ml3_2
+singleline3

TimberWinR.UnitTests/Multiline2.txt

@@ -0,0 +1,19 @@
+2015-01-07 13:14:26,572 TEST DEBUG [THREAD : 25] - Sending message to TServer - tcp://10.1111.11.111:1111
+'RequestAttachUserData' ('30')
+message attributes:
+AttributeConnID [long] = 00890
+AttributeReferenceID [int] = 88
+AttributeThisDN [str] = "2214"
+AttributeUserData [bstr] = KVList: 
+		'ActivityID' [str] = "1-XXXXXX"
+2015-01-07 13:14:26,574 TEST DEBUG [THREAD : 25] - Writing message RequestAttachUserData in 'proxy1' via '.StatePrimary proxy: proxy1'
+2015-01-07 13:14:26,575 TEST DEBUG [THREAD : 25] - sending RequestAttachUserData to Test.Platform.Commons.Connection.CommonConnection
+2015-01-07 13:20:31,665 TEST DEBUG [THREAD : SelectorThread] - Proxy got message 'EventOnHook' ('87')
+message attributes:
+AttributeEventSequenceNumber [long] = 4899493
+Time            = ComplexClass(TimeStamp):
+	AttributeTimeinuSecs [int] = 573000
+	AttributeTimeinSecs [int] = 1420644031
+AttributeThisDN [str] = "2214"
+. Processing with  state .StatePrimary proxy: proxy1
+2015-01-07 14:14:26,666 TEST DEBUG [THREAD : 25] - sending RequestAttachUserData to Test.Platform.Commons.Connection.CommonConnection

TimberWinR.UnitTests/MultilineTests.cs

@@ -0,0 +1,119 @@
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Text;
+using System.Linq;
+using System.Threading;
+using System.Threading.Tasks;
+using Newtonsoft.Json;
+using NUnit.Framework;
+using TimberWinR.Inputs;
+using TimberWinR.Parser;
+using Newtonsoft.Json.Linq;
+using System.Diagnostics;
+
+namespace TimberWinR.UnitTests
+{
+    [TestFixture]
+    public class MultilineTests
+    {
+     //   [Test(Description = "Test using next")]
+        public void TestMultiline1()
+        {
+            using (StreamReader sr = new StreamReader("Multiline1.txt"))
+            {
+                List<JObject> events = new List<JObject>();
+
+                Console.SetIn(sr);
+
+                Stdin sin = new Stdin();
+
+                sin.Codec = new Codec();
+                sin.Codec.Pattern = "\\\\$";
+                sin.Codec.What = Codec.WhatType.next;
+                sin.Codec.Type = Codec.CodecType.multiline;
+
+                var cancelTokenSource = new CancellationTokenSource();
+
+                using (var syncHandle = new ManualResetEventSlim())
+                {
+                    try
+                    {
+                        StdinListener sl = new StdinListener(sin, cancelTokenSource.Token);
+
+                        sl.OnMessageRecieved += o =>
+                        {
+                            events.Add(o);
+                            if (events.Count >= 6)
+                                cancelTokenSource.Cancel();
+                        };
+
+                        if (!cancelTokenSource.Token.IsCancellationRequested)
+                            syncHandle.Wait(TimeSpan.FromSeconds(10000), cancelTokenSource.Token);
+                    }
+                    catch (OperationCanceledException oex)
+                    {
+                    }
+                }
+
+                Assert.AreEqual(events.Count, 6);
+                Assert.AreEqual(events[0]["message"].ToString(), "multiline1 \\\nml1_1 \\\nml1_2 \\\nml1_2 ");
+                Assert.AreEqual(events[1]["message"].ToString(), "singleline1");
+                Assert.AreEqual(events[2]["message"].ToString(), "singleline2");
+                Assert.AreEqual(events[3]["message"].ToString(), "multiline2 \\\nml2_1 \\\nml2_2");
+                Assert.AreEqual(events[4]["message"].ToString(), "multiline3 \\\nml3_1 \\\nml3_2");
+                Assert.AreEqual(events[5]["message"].ToString(), "singleline3");
+            }
+        }
+
+    //    [Test(Description = "Test using previous")]
+        public void TestMultiline2()
+        {
+            using (StreamReader sr = new StreamReader("Multiline2.txt"))
+            {
+                List<JObject> events = new List<JObject>();
+
+                Console.SetIn(sr);
+
+                Stdin sin = new Stdin();
+
+                sin.Codec = new Codec();
+                sin.Codec.Pattern = "^(\\d{4}-\\d{2}-\\d{2}\\s\\d{2}:\\d{2}:\\d{2},\\d{3})(.*)$";              
+                sin.Codec.What = Codec.WhatType.previous;
+                sin.Codec.Type = Codec.CodecType.multiline;
+                sin.Codec.Negate = true;
+
+                var cancelTokenSource = new CancellationTokenSource();
+
+                using (var syncHandle = new ManualResetEventSlim())
+                {
+                    try
+                    {
+                        StdinListener sl = new StdinListener(sin, cancelTokenSource.Token);
+
+                        sl.OnMessageRecieved += o =>
+                        {
+                            events.Add(o);
+                            if (events.Count >= 4)
+                                cancelTokenSource.Cancel();
+                        };
+
+                        if (!cancelTokenSource.Token.IsCancellationRequested)
+                            syncHandle.Wait(TimeSpan.FromSeconds(10000), cancelTokenSource.Token);
+                    }
+                    catch (OperationCanceledException oex)
+                    {
+                    }
+                }
+
+                Assert.AreEqual(events.Count, 4);
+                Assert.AreEqual(events[0]["message"].ToString(), "2015-01-07 13:14:26,572 TEST DEBUG [THREAD : 25] - Sending message to TServer - tcp://10.1111.11.111:1111\n'RequestAttachUserData' ('30')\nmessage attributes:\nAttributeConnID [long] = 00890\nAttributeReferenceID [int] = 88\nAttributeThisDN [str] = \"2214\"\nAttributeUserData [bstr] = KVList: \n\t\t'ActivityID' [str] = \"1-XXXXXX\"");
+                Assert.AreEqual(events[1]["message"].ToString(), "2015-01-07 13:14:26,574 TEST DEBUG [THREAD : 25] - Writing message RequestAttachUserData in 'proxy1' via '.StatePrimary proxy: proxy1'");
+                Assert.AreEqual(events[2]["message"].ToString(), "2015-01-07 13:14:26,575 TEST DEBUG [THREAD : 25] - sending RequestAttachUserData to Test.Platform.Commons.Connection.CommonConnection");
+                Assert.AreEqual(events[3]["message"].ToString(), "2015-01-07 13:20:31,665 TEST DEBUG [THREAD : SelectorThread] - Proxy got message 'EventOnHook' ('87')\nmessage attributes:\nAttributeEventSequenceNumber [long] = 4899493\nTime            = ComplexClass(TimeStamp):\n\tAttributeTimeinuSecs [int] = 573000\n\tAttributeTimeinSecs [int] = 1420644031\nAttributeThisDN [str] = \"2214\"\n. Processing with  state .StatePrimary proxy: proxy1");
+            }
+        }
+
+    }
+}

TimberWinR.UnitTests/Parser/ElasticsearchOutputTests.cs

@@ -0,0 +1,55 @@
+namespace TimberWinR.UnitTests.Parser
+{
+    using System;
+
+    using Newtonsoft.Json.Linq;
+
+    using NUnit.Framework;
+
+    using TimberWinR.Parser;
+
+    public class ElasticsearchOutputTests
+    {
+        private ElasticsearchOutput parser;
+
+        [SetUp]
+        public void Setup()
+        {
+            this.parser = new ElasticsearchOutput();
+        }
+
+        [Test]
+        public void Given_no_index_returns_default_index_name()
+        {
+            this.parser.Index = "someindex";
+            var json = new JObject();
+
+            var result = this.parser.GetIndexName(json);
+
+            Assert.AreEqual("someindex", result);
+        }
+
+        [Test]
+        public void Given_index_with_date_format_and_timestamp_returns_name_by_timestamp()
+        {
+            this.parser.Index = "someindex-%{yyyy.MM.dd}";
+            var json = new JObject();
+            json.Add(new JProperty("@timestamp", "2011-11-30T18:45:32.450Z"));
+
+            var result = this.parser.GetIndexName(json);
+
+            Assert.AreEqual("someindex-2011.11.30", result);
+        }
+
+        [Test]
+        public void Given_index_with_date_format_and_no_timestamp_returns_name_by_current_date()
+        {
+            this.parser.Index = "someindex-%{yyyy.MM.dd}";
+            var json = new JObject();
+
+            var result = this.parser.GetIndexName(json);
+
+            Assert.AreEqual("someindex-" + DateTime.UtcNow.ToString("yyyy.MM.dd"), result);
+        }
+    }
+}

TimberWinR.UnitTests/TailFileTests.cs

@@ -0,0 +1,90 @@
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Text;
+using System.Linq;
+using System.Threading;
+using System.Threading.Tasks;
+using Newtonsoft.Json;
+using NUnit.Framework;
+using TimberWinR.Inputs;
+using TimberWinR.Parser;
+using Newtonsoft.Json.Linq;
+using System.Diagnostics;
+
+namespace TimberWinR.UnitTests
+{
+    [TestFixture]
+    public class TailFileTests
+    {
+        [Test]
+        public void TestTailFile()
+        {
+            List<JObject> events = new List<JObject>();
+
+            if (File.Exists(".timberwinrdb"))
+                File.Delete(".timberwinrdb");
+
+            var mgr = new Manager();
+            mgr.LogfileDir = ".";
+
+            var tf = new TailFile();
+            var cancelTokenSource = new CancellationTokenSource();
+            tf.Location = "TestTailFile1.log";
+
+   
+            if (File.Exists(tf.Location))
+                File.Delete(tf.Location);
+
+            try
+            {
+                var listener = new TailFileListener(tf, cancelTokenSource.Token);
+
+                listener.OnMessageRecieved += o =>
+                {
+                    events.Add(o);
+                    if (events.Count >= 100)
+                        cancelTokenSource.Cancel();
+                };
+
+                GenerateLogFile(tf.Location);
+
+                bool createdFile = false;
+                while (!listener.Stop && !cancelTokenSource.IsCancellationRequested)
+                {
+                    Thread.Sleep(100);
+                    if (!createdFile)
+                    {
+                        GenerateLogFile(tf.Location);
+                        createdFile = true;
+                    }
+                }
+            }
+            catch (OperationCanceledException oex)
+            {
+                Console.WriteLine("Done!");
+            }
+            catch (Exception ex)
+            {
+                Console.WriteLine(ex.ToString());
+
+            }
+            finally
+            {
+                Assert.AreEqual(100, events.Count);
+            }
+        }
+
+        private static void GenerateLogFile(string fileName)
+        {
+            using (System.IO.StreamWriter file = new System.IO.StreamWriter(fileName))
+            {
+                for (int i = 0; i < 100; i++)
+                {
+                    file.WriteLine("Log Line Number {0}", i);
+                }
+            }
+        }
+    }
+}

TimberWinR.UnitTests/TestBase.cs

@@ -0,0 +1,23 @@
+namespace TimberWinR.UnitTests
+{
+    using Moq;
+
+    using NUnit.Framework;
+
+    public class TestBase
+    {
+        public MockRepository MockRepository { get; private set; }
+
+        [SetUp]
+        public virtual void Setup()
+        {
+            this.MockRepository = new MockRepository(MockBehavior.Default);
+        }
+
+        [TearDown]
+        public virtual void TearDown()
+        {
+            this.MockRepository.VerifyAll();
+        }
+    }
+}

TimberWinR.UnitTests/TimberWinR.UnitTests.csproj

@@ -33,6 +33,14 @@
     <Prefer32Bit>false</Prefer32Bit>
   </PropertyGroup>
   <ItemGroup>
+    <Reference Include="Interop.MSUtil, Version=1.0.0.0, Culture=neutral, processorArchitecture=MSIL">
+      <SpecificVersion>False</SpecificVersion>
+      <EmbedInteropTypes>False</EmbedInteropTypes>
+      <HintPath>..\TimberWinR\lib\com-logparser\Interop.MSUtil.dll</HintPath>
+    </Reference>
+    <Reference Include="Moq">
+      <HintPath>..\packages\Moq.4.2.1409.1722\lib\net40\Moq.dll</HintPath>
+    </Reference>
     <Reference Include="Newtonsoft.Json, Version=6.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed, processorArchitecture=MSIL">
       <SpecificVersion>False</SpecificVersion>
       <HintPath>..\packages\Newtonsoft.Json.6.0.4\lib\net40\Newtonsoft.Json.dll</HintPath>
@@ -52,8 +60,15 @@
   <ItemGroup>
     <Compile Include="Configuration.cs" />
     <Compile Include="DateFilterTests.cs" />
+    <Compile Include="GeoIPFilterTests.cs" />
+    <Compile Include="Inputs\IisW3CRowReaderTests.cs" />
+    <Compile Include="JsonFilterTests.cs" />
     <Compile Include="GrokFilterTests.cs" />
+    <Compile Include="MultilineTests.cs" />
+    <Compile Include="Parser\ElasticsearchOutputTests.cs" />
     <Compile Include="Properties\AssemblyInfo.cs" />
+    <Compile Include="TailFileTests.cs" />
+    <Compile Include="TestBase.cs" />
   </ItemGroup>
   <ItemGroup>
     <ProjectReference Include="..\TimberWinR\TimberWinR.csproj">
@@ -69,6 +84,14 @@
       <SubType>Designer</SubType>
     </None>
   </ItemGroup>
+  <ItemGroup>
+    <Content Include="Multiline2.txt">
+      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
+    </Content>
+    <Content Include="Multiline1.txt">
+      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
+    </Content>
+  </ItemGroup>
   <Import Project="$(MSBuildToolsPath)\Microsoft.CSharp.targets" />
   <!-- To modify your build process, add your task inside one of the targets below and uncomment it. 
        Other similar extension points exist, see Microsoft.Common.targets.

TimberWinR.UnitTests/packages.config

@@ -1,5 +1,6 @@
 <?xml version="1.0" encoding="utf-8"?>
 <packages>
+  <package id="Moq" version="4.2.1409.1722" targetFramework="net40" />
   <package id="Newtonsoft.Json" version="6.0.4" targetFramework="net40" />
   <package id="NUnit" version="2.6.3" targetFramework="net40" />
 </packages>

TimberWinR.sln

@@ -17,11 +17,23 @@ EndProject
 Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution Items", "{1B2F600B-2400-45B9-A28E-CFC391D9EFA9}"
 	ProjectSection(SolutionItems) = preProject
 		appveyor.yml = appveyor.yml
+		chocolatey.png = chocolatey.png
+		chocolateyInstall.ps1.template = chocolateyInstall.ps1.template
+		chocolateyUninstall.ps1.guid = chocolateyUninstall.ps1.guid
+		chocolateyUninstall.ps1.template = chocolateyUninstall.ps1.template
+		chocolateyUninstall.ps1.template.orig = chocolateyUninstall.ps1.template.orig
 		LICENSE.txt = LICENSE.txt
+		Performance1.psess = Performance1.psess
 		README.md = README.md
+		timberwinr.nuspec.template = timberwinr.nuspec.template
 	EndProjectSection
 EndProject
 Project("{930C7802-8A8C-48F9-8165-68863BCCD9DD}") = "TimberWinR.Wix", "TimberWix\TimberWinR.Wix.wixproj", "{82A39B31-61EC-468D-AA71-0D949AC6528F}"
+	ProjectSection(ProjectDependencies) = postProject
+		{99096939-E9DD-4499-883D-4726745A5843} = {99096939-E9DD-4499-883D-4726745A5843}
+	EndProjectSection
+EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "TimberWinR.ExtractID", "TimberWinR.ExtractID\TimberWinR.ExtractID.csproj", "{99096939-E9DD-4499-883D-4726745A5843}"
 EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
@@ -75,8 +87,21 @@ Global
 		{82A39B31-61EC-468D-AA71-0D949AC6528F}.Release|Mixed Platforms.Build.0 = Release|x86
 		{82A39B31-61EC-468D-AA71-0D949AC6528F}.Release|x86.ActiveCfg = Release|x86
 		{82A39B31-61EC-468D-AA71-0D949AC6528F}.Release|x86.Build.0 = Release|x86
+		{99096939-E9DD-4499-883D-4726745A5843}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{99096939-E9DD-4499-883D-4726745A5843}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{99096939-E9DD-4499-883D-4726745A5843}.Debug|Mixed Platforms.ActiveCfg = Debug|Any CPU
+		{99096939-E9DD-4499-883D-4726745A5843}.Debug|Mixed Platforms.Build.0 = Debug|Any CPU
+		{99096939-E9DD-4499-883D-4726745A5843}.Debug|x86.ActiveCfg = Debug|Any CPU
+		{99096939-E9DD-4499-883D-4726745A5843}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{99096939-E9DD-4499-883D-4726745A5843}.Release|Any CPU.Build.0 = Release|Any CPU
+		{99096939-E9DD-4499-883D-4726745A5843}.Release|Mixed Platforms.ActiveCfg = Release|Any CPU
+		{99096939-E9DD-4499-883D-4726745A5843}.Release|Mixed Platforms.Build.0 = Release|Any CPU
+		{99096939-E9DD-4499-883D-4726745A5843}.Release|x86.ActiveCfg = Release|Any CPU
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE
 	EndGlobalSection
+	GlobalSection(Performance) = preSolution
+		HasPerformanceSessions = true
+	EndGlobalSection
 EndGlobal

TimberWinR/Configuration.cs

@@ -4,6 +4,8 @@ using System.Data.Odbc;
 using System.Diagnostics;
 using System.Linq;
 using System.Reflection;
+using System.Threading;
+using System.Threading.Tasks;
 using System.Text;
 using System.Xml;
 using System.Xml.Linq;
@@ -16,6 +18,7 @@ using Newtonsoft.Json.Linq;
 using TimberWinR.Inputs;
 using TimberWinR.Filters;
 
+
 using NLog;
 using TimberWinR.Parser;
 using Topshelf.Configurators;
@@ -25,7 +28,12 @@ using WindowsEvent = TimberWinR.Parser.WindowsEvent;
 namespace TimberWinR
 {
     public class Configuration
-    {
+    {       
+        private CancellationToken _cancelToken;
+        private bool _stopService;
+        private FileSystemWatcher _dirWatcher;
+        private Manager _manager;
+ 
         private List<WindowsEvent> _events = new List<WindowsEvent>();
         public IEnumerable<WindowsEvent> Events
         {
@@ -55,13 +63,25 @@ namespace TimberWinR
         public IEnumerable<Tcp> Tcps
         {
             get { return _tcps; }
-        }       
+        }
+
+        private List<Udp> _udps = new List<Udp>();
+        public IEnumerable<Udp> Udps
+        {
+            get { return _udps; }
+        }     
 
         private List<Log> _logs = new List<Log>();
         public IEnumerable<Log> Logs
         {
             get { return _logs; }
-        }       
+        }
+
+        private List<TailFile> _tails = new List<TailFile>();
+        public IEnumerable<TailFile> TailFiles
+        {
+            get { return _tails; }
+        }    
 
         private List<IISW3CLog> _iisw3clogs = new List<IISW3CLog>();
 
@@ -70,6 +90,12 @@ namespace TimberWinR
             get { return _iisw3clogs; }
         }
 
+        private List<W3CLog> _w3clogs = new List<W3CLog>();
+
+        public IEnumerable<W3CLog> W3C
+        {
+            get { return _w3clogs; }
+        }
 
         private List<Stdin> _stdins = new List<Stdin>();
         public IEnumerable<Stdin> Stdins
@@ -84,11 +110,88 @@ namespace TimberWinR
             get { return _filters; }
         }
 
+        private void MonitorDirectory(string directoryToWatch, CancellationToken cancelToken, Manager manager)
+        {
+            _manager = manager;
+            _cancelToken = cancelToken;
+            if (_dirWatcher == null)
+            {
+                _dirWatcher = new FileSystemWatcher();
+                _dirWatcher.Path = directoryToWatch;
+                _dirWatcher.NotifyFilter = NotifyFilters.LastAccess | NotifyFilters.LastWrite | NotifyFilters.FileName | NotifyFilters.DirectoryName;
+                // Only watch json files.
+                _dirWatcher.Filter = "*.json";
+                _dirWatcher.Created += DirWatcherOnCreated;
+                _dirWatcher.Changed += DirWatcherOnChanged;
+                _dirWatcher.Renamed += DirWatcherOnRenamed;
+                _dirWatcher.EnableRaisingEvents = true;
+            }       
+        }
 
-        public static Configuration FromDirectory(string jsonDirectory)
+        private void DirWatcherOnRenamed(object sender, RenamedEventArgs e)
+        {
+            // The Renamed file could be a different name from .json
+            FileInfo fi = new FileInfo(e.FullPath);
+            if (fi.Extension == ".json")
+            {
+                LogManager.GetCurrentClassLogger().Info("File: OnRenamed " + e.FullPath + " " + e.ChangeType);
+                ProcessNewJson(e.FullPath);
+            }
+        }
+
+        private void DirWatcherOnCreated(object sender, FileSystemEventArgs e)
+        {
+            FileInfo fi = new FileInfo(e.FullPath);
+            if (fi.Extension == ".json")
+            {
+                LogManager.GetCurrentClassLogger().Info("File: OnCreated " + e.FullPath + " " + e.ChangeType);
+                ProcessNewJson(e.FullPath);
+            }
+        }
+
+        private void DirWatcherOnChanged(object sender, FileSystemEventArgs e)
+        {
+            FileInfo fi = new FileInfo(e.FullPath);
+            if (fi.Extension == ".json")
+            {
+                // Specify what is done when a file is changed, created, or deleted.
+                LogManager.GetCurrentClassLogger()
+                    .Info("File: OnChanged " + e.ChangeType.ToString() + " " + e.FullPath + " " + e.ChangeType);
+                ProcessNewJson(e.FullPath);
+            }
+        }
+      
+        private void ProcessNewJson(string fileName)
+        {
+            try
+            {
+                Configuration c = new Configuration();
+                var config = Configuration.FromFile(fileName, c);
+                _manager.ProcessConfiguration(_cancelToken, config);
+
+            }
+            catch (Exception ex)
+            {
+                LogManager.GetCurrentClassLogger().Error(ex);                         
+            }
+        }
+
+        private void ShutdownDirectoryMonitor()
+        {
+            _stopService = true;
+            _dirWatcher.EnableRaisingEvents = false;
+            LogManager.GetCurrentClassLogger().Info("Stopping Directory Monitor");            
+        }
+
+        private void DirectoryWatcher(string directoryToWatch)
+        {
+            LogManager.GetCurrentClassLogger().Info("Starting Directory Monitor {0}", directoryToWatch);                       
+        }
+
+        public static Configuration FromDirectory(string jsonDirectory, CancellationToken cancelToken, Manager manager)
         {
             Configuration c = null;
-
+      
             foreach (string jsonConfFile in Directory.GetFiles(jsonDirectory, "*.json"))
             {
                 if (!string.IsNullOrEmpty(jsonConfFile))
@@ -97,6 +200,10 @@ namespace TimberWinR
                 }
             }
 
+            // Startup Directory Monitor
+            if (manager.LiveMonitor)           
+                c.MonitorDirectory(jsonDirectory, cancelToken, manager);            
+
             return c;
         }
 
@@ -127,29 +234,35 @@ namespace TimberWinR
             if (x.TimberWinR.Inputs != null)
             {
                 if (x.TimberWinR.Inputs.WindowsEvents != null)
-                    c._events = x.TimberWinR.Inputs.WindowsEvents.ToList();
+                    c._events.AddRange(x.TimberWinR.Inputs.WindowsEvents.ToList());
+                if (x.TimberWinR.Inputs.W3CLogs != null)
+                    c._w3clogs.AddRange(x.TimberWinR.Inputs.W3CLogs.ToList());
                 if (x.TimberWinR.Inputs.IISW3CLogs != null)
-                    c._iisw3clogs = x.TimberWinR.Inputs.IISW3CLogs.ToList();
+                    c._iisw3clogs.AddRange(x.TimberWinR.Inputs.IISW3CLogs.ToList());
                 if (x.TimberWinR.Inputs.Stdins != null)
-                    c._stdins = x.TimberWinR.Inputs.Stdins.ToList();
+                    c._stdins.AddRange(x.TimberWinR.Inputs.Stdins.ToList());
                 if (x.TimberWinR.Inputs.Logs != null)
-                    c._logs = x.TimberWinR.Inputs.Logs.ToList();
+                    c._logs.AddRange(x.TimberWinR.Inputs.Logs.ToList());
+                if (x.TimberWinR.Inputs.TailFiles != null)
+                    c._tails.AddRange(x.TimberWinR.Inputs.TailFiles.ToList());
                 if (x.TimberWinR.Inputs.Tcps != null)
-                    c._tcps = x.TimberWinR.Inputs.Tcps.ToList();
+                    c._tcps.AddRange(x.TimberWinR.Inputs.Tcps.ToList());
+                if (x.TimberWinR.Inputs.Udps != null)
+                    c._udps.AddRange(x.TimberWinR.Inputs.Udps.ToList());
             }
 
             if (x.TimberWinR.Outputs != null)
             {
                 if (x.TimberWinR.Outputs.Redis != null)
-                    c._redisOutputs = x.TimberWinR.Outputs.Redis.ToList();
+                    c._redisOutputs.AddRange(x.TimberWinR.Outputs.Redis.ToList());
                 if (x.TimberWinR.Outputs.Elasticsearch != null)
-                    c._elasticsearchOutputs = x.TimberWinR.Outputs.Elasticsearch.ToList();
+                    c._elasticsearchOutputs.AddRange(x.TimberWinR.Outputs.Elasticsearch.ToList());
                 if (x.TimberWinR.Outputs.Stdout != null)
-                    c._stdoutOutputs = x.TimberWinR.Outputs.Stdout.ToList();
-            }         
+                    c._stdoutOutputs.AddRange(x.TimberWinR.Outputs.Stdout.ToList());
+            }
 
             if (x.TimberWinR.Filters != null)
-                c._filters = x.TimberWinR.AllFilters.ToList();
+                c._filters.AddRange(x.TimberWinR.AllFilters.ToList());
 
             c.Validate(c);
 
@@ -184,6 +297,7 @@ namespace TimberWinR
             _elasticsearchOutputs = new List<ElasticsearchOutput>();
             _stdoutOutputs = new List<StdoutOutput>();
             _tcps = new List<Tcp>();
+            _udps = new List<Udp>();
         }
 
         public static Object GetPropValue(String name, Object obj)

TimberWinR/Diagnostics/Diagnostics.cs

@@ -1,6 +1,8 @@
 using System;
 using System.Collections.Generic;
 using System.Linq;
+using System.Reflection;
+using System.Runtime.Remoting.Messaging;
 using System.Text;
 using System.Threading;
 using System.Net;
@@ -20,10 +22,7 @@ namespace TimberWinR.Diagnostics
         public int Port { get; set; }
         public Manager Manager { get; set; }
 
-        private readonly System.Net.Sockets.TcpListener _tcpListenerV4;
-        private readonly System.Net.Sockets.TcpListener _tcpListenerV6;
-        private Thread _listenThreadV4;
-        private Thread _listenThreadV6;
+        private HttpListener web;
 
         public Diagnostics(Manager manager, CancellationToken cancelToken, int port = 5141)
         {
@@ -33,15 +32,79 @@ namespace TimberWinR.Diagnostics
 
             LogManager.GetCurrentClassLogger().Info("Diagnostic(v4/v6) on Port {0} Ready", Port);
 
-            _tcpListenerV6 = new System.Net.Sockets.TcpListener(IPAddress.IPv6Any, Port);
-            _tcpListenerV4 = new System.Net.Sockets.TcpListener(IPAddress.Any, Port);
+            var hl = new Thread(new ParameterizedThreadStart(HttpListen));
+            hl.Start(null);
+        }
 
-            _listenThreadV4 = new Thread(new ParameterizedThreadStart(ListenForClients));
-            _listenThreadV4.Start(_tcpListenerV4);
+        void processRequest()
+        {
+            var result = web.BeginGetContext(DiagnosticCallback, web);
+            result.AsyncWaitHandle.WaitOne();
+        }
 
-            _listenThreadV6 = new Thread(new ParameterizedThreadStart(ListenForClients));
-            _listenThreadV6.Start(_tcpListenerV6);
+        private Assembly GetAssemblyByName(string name)
+        {
+            return AppDomain.CurrentDomain.GetAssemblies().
+                   SingleOrDefault(assembly => assembly.GetName().Name == name);
+        }
 
+
+        private void DiagnosticCallback(IAsyncResult result)
+        {       
+            if (web == null)         
+                return;
+
+            var context = web.EndGetContext(result);
+            var response = context.Response;
+
+            JObject json = new JObject(
+                new JProperty("timberwinr",
+                    new JObject(
+                        new JProperty("version", GetAssemblyByName("TimberWinR.ServiceHost").GetName().Version.ToString()),
+                        new JProperty("messages", Manager.NumMessages),
+                        new JProperty("startedon", Manager.StartedOn),
+                        new JProperty("configfile", Manager.JsonConfig),
+                        new JProperty("logdir", Manager.LogfileDir),
+                        new JProperty("logginglevel", LogManager.GlobalThreshold.ToString()),
+                        new JProperty("inputs",
+                            new JArray(
+                                from i in Manager.Listeners
+                                select new JObject(i.ToJson()))),
+                        new JProperty("filters",
+                            new JArray(
+                                from f in Manager.Config.Filters
+                                select new JObject(f.ToJson()))),
+                        new JProperty("outputs",
+                            new JArray(
+                                from o in Manager.Outputs
+                                select new JObject(o.ToJson()))))));
+
+            response.StatusCode = (int)HttpStatusCode.OK;
+            response.StatusDescription = HttpStatusCode.OK.ToString();
+            byte[] buffer = Encoding.UTF8.GetBytes(json.ToString());
+            response.ContentLength64 = buffer.Length;
+            response.OutputStream.Write(buffer, 0, buffer.Length);
+            response.OutputStream.Close();
+        }
+
+
+        private void HttpListen(object o)
+        {
+            web = new HttpListener();
+            try
+            {
+                web.Prefixes.Add(string.Format("http://*:{0}/", Port));
+                web.Start();
+
+                while (web != null && web.IsListening)
+                {                  
+                    processRequest();                  
+                }               
+            }
+            catch (Exception ex)
+            {
+                LogManager.GetCurrentClassLogger().Error("Diagnostic Listener Error: {0}", ex.ToString());
+            }           
         }
 
         private void ListenForClients(object olistener)
@@ -49,7 +112,7 @@ namespace TimberWinR.Diagnostics
             var listener = olistener as System.Net.Sockets.TcpListener;
 
             listener.Start();
-          
+
             while (!CancelToken.IsCancellationRequested)
             {
                 try
@@ -71,11 +134,13 @@ namespace TimberWinR.Diagnostics
             }
         }
 
+
         private void HandleNewClient(object client)
         {
             var tcpClient = (TcpClient)client;
             NetworkStream clientStream = null;
 
+            Console.WriteLine("Handle new diag client: {0}, {1}", tcpClient.Connected, tcpClient.Client.RemoteEndPoint.ToString());
             try
             {
                 using (clientStream = tcpClient.GetStream())
@@ -87,17 +152,22 @@ namespace TimberWinR.Diagnostics
                                 new JProperty("messages", Manager.NumMessages),
                                 new JProperty("startedon", Manager.StartedOn),
                                 new JProperty("configfile", Manager.JsonConfig),
-                                new JProperty("logdir", Manager.LogfileDir),                              
+                                new JProperty("logdir", Manager.LogfileDir),
                                 new JProperty("logginglevel", LogManager.GlobalThreshold.ToString()),
                                 new JProperty("inputs",
                                     new JArray(
-                                        from i in Manager.Listeners                                    
-                                        select new JObject(i.ToJson()))),             
+                                        from i in Manager.Listeners
+                                        select new JObject(i.ToJson()))),
+                                new JProperty("filters",
+                                    new JArray(
+                                        from f in Manager.Config.Filters
+                                        select new JObject(f.ToJson()))),
                                 new JProperty("outputs",
                                     new JArray(
                                         from o in Manager.Outputs
                                         select new JObject(o.ToJson()))))));
-                                                                                 
+
+
 
                     sw.WriteLine(json.ToString());
                     sw.Flush();
@@ -106,15 +176,27 @@ namespace TimberWinR.Diagnostics
             }
             catch (Exception ex)
             {
-                LogManager.GetCurrentClassLogger().Error("Tcp Exception", ex);
+                LogManager.GetCurrentClassLogger().Error(ex);
             }
 
         }
 
         public void Shutdown()
         {
-            _tcpListenerV4.Stop();
-            _tcpListenerV6.Stop();
+            
+            try
+            {
+                if (web != null && web.IsListening)
+                {
+                    LogManager.GetCurrentClassLogger().Info("Shutting down diagnostics listener");
+                    web.Close();
+                    web = null;
+                }
+            }
+            catch (Exception ex)
+            {
+                LogManager.GetCurrentClassLogger().Error(ex);
+            }
         }
 
     }

TimberWinR/Filters/DateFilter.cs

@@ -16,8 +16,15 @@ namespace TimberWinR.Parser
     {                               
         public override bool Apply(JObject json)
         {
+            if (!string.IsNullOrEmpty(Type))
+            {
+                JToken json_type = json["type"];
+                if (json_type != null && json_type.ToString() != Type)
+                    return true; // Filter does not apply.
+            }
+
             if (Condition != null && !EvaluateCondition(json, Condition))
-                return false;
+                return true;
 
             if (Matches(json))
             {                
@@ -25,7 +32,19 @@ namespace TimberWinR.Parser
             }
            
             return true;
-        }   
+        }
+
+        public override JObject ToJson()
+        {
+            JObject json = new JObject(
+                 new JProperty("date",
+                     new JObject(
+                         new JProperty("condition", Condition),
+                         new JProperty("type", Type),
+                         new JProperty("addfields", AddField)                     
+                         )));
+            return json;
+        }
 
         // copy_field "field1" -> "field2"
         private void AddFields(Newtonsoft.Json.Linq.JObject json)

TimberWinR/Filters/GeoIPFilter.cs

@@ -0,0 +1,188 @@
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Reflection;
+using System.Text;
+using System.Xml.Linq;
+using Newtonsoft.Json;
+using Newtonsoft.Json.Linq;
+using MaxMind.GeoIP2;
+using MaxMind.Db;
+using MaxMind.GeoIP2.Exceptions;
+using NLog;
+
+namespace TimberWinR.Parser
+{
+    public partial class GeoIP : LogstashFilter
+    {
+        private string DatabaseFileName { get; set; }
+        private DatabaseReader dr;
+        public override JObject ToJson()
+        {
+            JObject json = new JObject(
+               new JProperty("geoip",
+                   new JObject(
+                       new JProperty("source", Source),
+                       new JProperty("type", Type),
+                       new JProperty("condition", Condition),
+                       new JProperty("target", Target)
+                       )));
+            return json;
+        }
+
+        public GeoIP()
+        {
+            Target = "geoip";
+            DatabaseFileName = Path.Combine(AssemblyDirectory, "GeoLite2City.mmdb");
+            dr = new DatabaseReader(DatabaseFileName);
+        }
+
+        private static string AssemblyDirectory
+        {
+            get
+            {
+                string codeBase = Assembly.GetExecutingAssembly().CodeBase;
+                UriBuilder uri = new UriBuilder(codeBase);
+                string path = Uri.UnescapeDataString(uri.Path);
+                return Path.GetDirectoryName(path);
+            }
+        }
+
+        public override bool Apply(JObject json)
+        {
+            if (!string.IsNullOrEmpty(Type))
+            {
+                JToken json_type = json["type"];
+                if (json_type != null && json_type.ToString() != Type)
+                    return true; // Filter does not apply.
+            }
+
+            if (Condition != null)
+            {
+                var expr = EvaluateCondition(json, Condition);
+                if (!expr)
+                    return true;
+            }
+
+            var source = json[Source];
+
+            if (source != null && !string.IsNullOrEmpty(source.ToString()))
+            {
+                try
+                {                  
+                    var l = dr.City(source.ToString());
+                    if (l != null)
+                    {
+                        JObject geo_json = new JObject(
+                            new JProperty(Target,
+                                new JObject(
+                                    new JProperty("ip", source.ToString()),
+                                    new JProperty("country_code2", l.Country.IsoCode),                                   
+                                    new JProperty("country_name", l.Country.Name),
+                                    new JProperty("continent_code", l.Continent.Code),
+                                    new JProperty("region_name", l.MostSpecificSubdivision.IsoCode),
+                                    new JProperty("city_name", l.City.Name),
+                                    new JProperty("postal_code", l.Postal.Code),
+                                    new JProperty("latitude", l.Location.Latitude),
+                                    new JProperty("longitude", l.Location.Longitude),
+                                    new JProperty("dma_code", l.Location.MetroCode),                                   
+                                    new JProperty("timezone", l.Location.TimeZone),
+                                    new JProperty("real_region_name", l.MostSpecificSubdivision.Name),
+                                    new JProperty("location",
+                                        new JArray(l.Location.Longitude, l.Location.Latitude)
+                                        ))));
+                                   
+                        json.Merge(geo_json, new JsonMergeSettings
+                        {
+                            MergeArrayHandling = MergeArrayHandling.Union
+                        });                       
+                    }
+                    else
+                    {
+                        json["_geoiperror"] = string.Format("IP Address not found: {0}", source.ToString());
+                    }
+                }
+                catch (Exception ex)
+                {
+                    json["_geoiperror"] = string.Format("IP Address not found: {0} ({1})", source.ToString(), ex.ToString());
+                    return true;
+                }
+            }
+
+            AddFields(json);
+            AddTags(json);
+            RemoveFields(json);
+            RemoveTags(json);      
+
+            return true;
+        }
+
+        private void AddFields(Newtonsoft.Json.Linq.JObject json)
+        {
+            if (AddField != null && AddField.Length > 0)
+            {
+                for (int i = 0; i < AddField.Length; i += 2)
+                {
+                    string fieldName = ExpandField(AddField[i], json);
+                    string fieldValue = ExpandField(AddField[i + 1], json);
+                    AddOrModify(json, fieldName, fieldValue);
+                }
+            }
+        }
+
+        private void RemoveFields(Newtonsoft.Json.Linq.JObject json)
+        {
+            if (RemoveField != null && RemoveField.Length > 0)
+            {
+                for (int i = 0; i < RemoveField.Length; i++)
+                {
+                    string fieldName = ExpandField(RemoveField[i], json);
+                    RemoveProperties(json, new string[] { fieldName });
+                }
+            }
+        }
+
+        private void AddTags(Newtonsoft.Json.Linq.JObject json)
+        {
+            if (AddTag != null && AddTag.Length > 0)
+            {
+                for (int i = 0; i < AddTag.Length; i++)
+                {
+                    string value = ExpandField(AddTag[i], json);
+
+                    JToken tags = json["tags"];
+                    if (tags == null)
+                        json.Add("tags", new JArray(value));
+                    else
+                    {
+                        JArray a = tags as JArray;
+                        a.Add(value);
+                    }
+                }
+            }
+        }
+
+        private void RemoveTags(Newtonsoft.Json.Linq.JObject json)
+        {
+            if (RemoveTag != null && RemoveTag.Length > 0)
+            {
+                JToken tags = json["tags"];
+                if (tags != null)
+                {
+                    List<JToken> children = tags.Children().ToList();
+                    for (int i = 0; i < RemoveTag.Length; i++)
+                    {
+                        string tagName = ExpandField(RemoveTag[i], json);
+                        foreach (JToken token in children)
+                        {
+                            if (token.ToString() == tagName)
+                                token.Remove();
+                        }
+                    }
+                }
+            }
+        }
+
+    }
+}

TimberWinR/Filters/GrokFilter.cs

@@ -30,22 +30,65 @@ namespace TimberWinR.Parser
         }
     }
 
+
+
     public partial class Grok : LogstashFilter
     {
+        public override JObject ToJson()
+        {
+            string field = Match[0];
+            string expr = Match[1];
+
+            JObject json = new JObject(
+                 new JProperty("grok",
+                     new JObject(
+                         new JProperty("match", field),
+                         new JProperty("expr", expr),
+                         new JProperty("condition", Condition),
+                         new JProperty("addfields", AddField),
+                         new JProperty("addtags", AddTag),
+                         new JProperty("type", Type),
+                         new JProperty("removefields", RemoveField),
+                         new JProperty("removetag", RemoveTag)
+                         )));
+            return json;
+        }
+
+        // Returns: true - Filter does not apply or has been applied successfully
+        // Returns: false - Drop this object
         public override bool Apply(JObject json)
         {
-            if (Condition != null && !EvaluateCondition(json, Condition))
-                return false;
+            if (!string.IsNullOrEmpty(Type))
+            {
+                JToken json_type = json["type"];
+                if (json_type != null && json_type.ToString() != Type)
+                    return true; // Filter does not apply.
+            }
 
             if (Matches(json))
             {
+                if (Condition != null)
+                {
+                    var expr = EvaluateCondition(json, Condition);
+                    if (expr)
+                    {
+                        if (DropIfMatch)
+                            return false; // drop this one
+                    }
+                    else
+                        return true;
+                }
+
+                if (DropIfMatch)
+                    return false;
+
                 AddFields(json);
-                AddTags(json);               
+                AddTags(json);
                 RemoveFields(json);
                 RemoveTags(json);
-                return true;
             }
-            return false;
+
+            return true;
         }
 
         private bool Matches(Newtonsoft.Json.Linq.JObject json)
@@ -73,7 +116,10 @@ namespace TimberWinR.Parser
                         return true; // Yes!
                     }
                 }
-                return true; // Empty field is no match
+                if (string.IsNullOrEmpty(expr))
+                    return true; // Empty field is no match
+                else
+                    return false;
             }
             return false; // Not specified is failure
         }
@@ -91,8 +137,6 @@ namespace TimberWinR.Parser
             }
         }
 
-      
-
         private void RemoveFields(Newtonsoft.Json.Linq.JObject json)
         {
             if (RemoveField != null && RemoveField.Length > 0)
@@ -132,17 +176,17 @@ namespace TimberWinR.Parser
                 JToken tags = json["tags"];
                 if (tags != null)
                 {
-                    List<JToken> children = tags.Children().ToList();                                      
+                    List<JToken> children = tags.Children().ToList();
                     for (int i = 0; i < RemoveTag.Length; i++)
                     {
-                        string tagName = ExpandField(RemoveTag[i], json);       
-                        foreach(JToken token in children)
+                        string tagName = ExpandField(RemoveTag[i], json);
+                        foreach (JToken token in children)
                         {
                             if (token.ToString() == tagName)
                                 token.Remove();
                         }
-                    }                   
-                }               
+                    }
+                }
             }
         }
     }

TimberWinR/Filters/JsonFilter.cs

@@ -0,0 +1,184 @@
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Text;
+using System.Xml.Linq;
+using Newtonsoft.Json;
+using Newtonsoft.Json.Linq;
+using NLog;
+
+namespace TimberWinR.Parser
+{
+    public partial class Json : LogstashFilter
+    {
+        public Json()
+        {
+            RemoveSource = true;
+        }
+        public override JObject ToJson()
+        {
+            JObject json = new JObject(
+               new JProperty("json",
+                   new JObject(
+                       new JProperty("condition", Condition),
+                       new JProperty("source", Source),
+                       new JProperty("promote", Source),
+                       new JProperty("target", Target),
+                       new JProperty("type", Type),
+                       new JProperty("addfields", AddField),
+                       new JProperty("addtags", AddTag),
+                       new JProperty("removefields", RemoveField),
+                       new JProperty("removetag", RemoveTag)
+                       )));
+            return json;
+        }
+
+        public override bool Apply(JObject json)
+        {
+            if (!string.IsNullOrEmpty(Type))
+            {
+                JToken json_type = json["type"];
+                if (json_type != null && json_type.ToString() != Type)
+                    return true; // Filter does not apply.
+            }
+
+            if (Condition != null)
+            {
+                var expr = EvaluateCondition(json, Condition);
+                if (!expr)
+                    return true;
+            }
+
+            var source = json[Source];
+            if (source == null)
+                return true;
+
+            var jsonOrig = source.ToString();
+
+            if (!string.IsNullOrEmpty(source.ToString()))
+            {
+                try
+                {
+                    JObject subJson;
+
+                    if (Target != null && !string.IsNullOrEmpty(Target))
+                    {
+                        subJson = new JObject();
+                        subJson[Target] = JObject.Parse(source.ToString());
+                    }
+                    else
+                        subJson = JObject.Parse(source.ToString());
+
+                    if (!string.IsNullOrEmpty(Promote))
+                    {
+                        var promotedJson = subJson[Promote];
+                        RemoveProperties(subJson, new string[] { Promote });
+
+                        subJson.Merge(promotedJson, new JsonMergeSettings
+                        {
+                            MergeArrayHandling = MergeArrayHandling.Replace
+                        });
+                    }
+
+                    json.Merge(subJson, new JsonMergeSettings
+                    {
+                        MergeArrayHandling = MergeArrayHandling.Replace
+                    });
+
+                    if (Rename != null && Rename.Length > 0)
+                    {
+                        string oldName = ExpandField(Rename[0], json);
+                        string newName = ExpandField(Rename[1], json);
+                        RenameProperty(json, oldName, newName);
+                    }
+
+                    if (RemoveSource)
+                    {
+                        RemoveProperties(json, new string[] { Source });
+                    }
+
+                }
+                catch (Exception ex)
+                {
+                    LogManager.GetCurrentClassLogger().Error(ex);
+                    return true;
+                }
+            }
+
+            AddFields(json);
+            AddTags(json);
+            RemoveFields(json);
+            RemoveTags(json);
+
+            return true;
+        }
+
+        private void AddFields(Newtonsoft.Json.Linq.JObject json)
+        {
+            if (AddField != null && AddField.Length > 0)
+            {
+                for (int i = 0; i < AddField.Length; i += 2)
+                {
+                    string fieldName = ExpandField(AddField[i], json);
+                    string fieldValue = ExpandField(AddField[i + 1], json);
+                    AddOrModify(json, fieldName, fieldValue);
+                }
+            }
+        }
+
+        private void RemoveFields(Newtonsoft.Json.Linq.JObject json)
+        {
+            if (RemoveField != null && RemoveField.Length > 0)
+            {
+                for (int i = 0; i < RemoveField.Length; i++)
+                {
+                    string fieldName = ExpandField(RemoveField[i], json);
+                    RemoveProperties(json, new string[] { fieldName });
+                }
+            }
+        }
+
+        private void AddTags(Newtonsoft.Json.Linq.JObject json)
+        {
+            if (AddTag != null && AddTag.Length > 0)
+            {
+                for (int i = 0; i < AddTag.Length; i++)
+                {
+                    string value = ExpandField(AddTag[i], json);
+
+                    JToken tags = json["tags"];
+                    if (tags == null)
+                        json.Add("tags", new JArray(value));
+                    else
+                    {
+                        JArray a = tags as JArray;
+                        a.Add(value);
+                    }
+                }
+            }
+        }
+
+        private void RemoveTags(Newtonsoft.Json.Linq.JObject json)
+        {
+            if (RemoveTag != null && RemoveTag.Length > 0)
+            {
+                JToken tags = json["tags"];
+                if (tags != null)
+                {
+                    List<JToken> children = tags.Children().ToList();
+                    for (int i = 0; i < RemoveTag.Length; i++)
+                    {
+                        string tagName = ExpandField(RemoveTag[i], json);
+                        foreach (JToken token in children)
+                        {
+                            if (token.ToString() == tagName)
+                                token.Remove();
+                        }
+                    }
+                }
+            }
+        }
+
+    }
+}

TimberWinR/Filters/MutateFilter.cs

@@ -11,17 +11,56 @@ namespace TimberWinR.Parser
 { 
      public partial class Mutate : LogstashFilter
      {
+         public override JObject ToJson()
+         {
+             JObject json = new JObject(
+                new JProperty("mutate",
+                    new JObject(
+                        new JProperty("condition", Condition),
+                        new JProperty("splits", Split),
+                        new JProperty("type", Type),
+                        new JProperty("remove", Remove),
+                        new JProperty("rename", Rename),
+                        new JProperty("replace", Replace)                       
+                        )));
+             return json;
+         }
+
          public override bool Apply(JObject json)
          {
-             if (Condition != null && !EvaluateCondition(json, Condition))
-                 return false;
+             if (!string.IsNullOrEmpty(Type))
+             {
+                 JToken json_type = json["type"];
+                 if (json_type != null && json_type.ToString() != Type)
+                     return true; // Filter does not apply.
+             }
+
+             if (Condition != null)
+             {
+                 var expr = EvaluateCondition(json, Condition);
+                 if (!expr)
+                    return true;
+             }
 
              ApplySplits(json);
+             ApplyRemoves(json);
              ApplyRenames(json);
              ApplyReplace(json);
              return true;
          }
 
+         private void ApplyRemoves(JObject json)
+         {
+             if (Remove != null && Remove.Length > 0)
+             {
+                 for (int i = 0; i < Remove.Length; i += 1)
+                 {
+                     string name = ExpandField(Remove[i], json);
+                     RemoveProperty(json, name);
+                 }
+             }
+         }
+
          private void ApplyRenames(JObject json)
          {
              if (Rename != null && Rename.Length > 0)

TimberWinR/Inputs/IISW3CInputListener.cs

@@ -1,15 +1,10 @@
 using System;
 using System.Collections.Generic;
 using System.Linq;
-using System.Security.AccessControl;
-using System.Text;
 using System.Threading;
 using System.Threading.Tasks;
-using System.IO;
-using Interop.MSUtil;
 
 using Newtonsoft.Json.Linq;
-using Newtonsoft.Json.Serialization;
 using NLog;
 using LogQuery = Interop.MSUtil.LogQueryClassClass;
 using IISW3CLogInputFormat = Interop.MSUtil.COMIISW3CInputContextClassClass;
@@ -20,22 +15,31 @@ namespace TimberWinR.Inputs
 {
     public class IISW3CInputListener : InputListener
     {
-        private int _pollingIntervalInSeconds = 1;
-        private TimberWinR.Parser.IISW3CLog _arguments;
+        private readonly int _pollingIntervalInSeconds;
+        private readonly Parser.IISW3CLog _arguments;
         private long _receivedMessages;
+        public bool Stop { get; set; }
+        private IisW3CRowReader rowReader;      
 
-        public IISW3CInputListener(TimberWinR.Parser.IISW3CLog arguments, CancellationToken cancelToken, int pollingIntervalInSeconds = 1)
+        public IISW3CInputListener(Parser.IISW3CLog arguments, CancellationToken cancelToken, int pollingIntervalInSeconds = 5)
             : base(cancelToken, "Win32-IISLog")
-        {
+        {          
             _arguments = arguments;
             _receivedMessages = 0;
             _pollingIntervalInSeconds = pollingIntervalInSeconds;
-            var task = new Task(IISW3CWatcher, cancelToken);
-            task.Start();
+            this.rowReader = new IisW3CRowReader(_arguments.Fields);
+
+            foreach (string loc in _arguments.Location.Split(','))
+            {
+                string hive = loc.Trim();
+                Task.Factory.StartNew(() => IISW3CWatcher(loc));
+            }
         }
 
         public override void Shutdown()
         {
+            Stop = true;
+            LogManager.GetCurrentClassLogger().Info("Shutting Down {0}", InputType);
             base.Shutdown();
         }
 
@@ -56,18 +60,18 @@ namespace TimberWinR.Inputs
             return json;
         }
 
-
-        private void IISW3CWatcher()
+        private void IISW3CWatcher(string location)
         {
+            LogManager.GetCurrentClassLogger().Info("IISW3Listener Ready For {0}", location);
+
             var oLogQuery = new LogQuery();
 
             var iFmt = new IISW3CLogInputFormat()
             {
                 codepage = _arguments.CodePage,
-                consolidateLogs = _arguments.ConsolidateLogs,
+                consolidateLogs = true,
                 dirTime = _arguments.DirTime,
                 dQuotes = _arguments.DoubleQuotes,
-                iCheckpoint = CheckpointFileName,
                 recurse = _arguments.Recurse,
                 useDoubleQuotes = _arguments.DoubleQuotes
             };
@@ -75,58 +79,75 @@ namespace TimberWinR.Inputs
             if (_arguments.MinDateMod.HasValue)
                 iFmt.minDateMod = _arguments.MinDateMod.Value.ToString("yyyy-MM-dd hh:mm:ss");
 
-            // Create the query
-            var query = string.Format("SELECT * FROM {0}", _arguments.Location);
+            Dictionary<string, Int64> logFileMaxRecords = new Dictionary<string, Int64>();
 
-            var firstQuery = true;
-            // Execute the query
-            while (!CancelToken.IsCancellationRequested)
+            using (var syncHandle = new ManualResetEventSlim())
             {
-                try
+                // Execute the query
+                while (!Stop)
                 {
-                    var rs = oLogQuery.Execute(query, iFmt);
-                    Dictionary<string, int> colMap = new Dictionary<string, int>();
-                    for (int col = 0; col < rs.getColumnCount(); col++)
+                    // Execute the query
+                    if (!CancelToken.IsCancellationRequested)
                     {
-                        string colName = rs.getColumnName(col);
-                        colMap[colName] = col;
-                    }
-
-                    // Browse the recordset
-                    for (; !rs.atEnd(); rs.moveNext())
-                    {
-                        // We want to "tail" the log, so skip the first query results.
-                        if (!firstQuery)
+                        try
                         {
-                            var record = rs.getRecord();
-                            var json = new JObject();
-                            foreach (var field in _arguments.Fields)
-                            {
-                                if (!colMap.ContainsKey(field.Name))
-                                    continue;
+                            oLogQuery = new LogQuery();
 
-                                object v = record.getValue(field.Name);
-                                if (field.DataType == typeof(DateTime))
+                            var qfiles = string.Format("SELECT Distinct [LogFilename] FROM {0}", location);
+                            var rsfiles = oLogQuery.Execute(qfiles, iFmt);
+                            for (; !rsfiles.atEnd(); rsfiles.moveNext())
+                            {
+                                var record = rsfiles.getRecord();
+                                string fileName = record.getValue("LogFilename") as string;
+                                if (!logFileMaxRecords.ContainsKey(fileName))
                                 {
-                                    DateTime dt = DateTime.Parse(v.ToString());
-                                    json.Add(new JProperty(field.Name, dt));
+                                    var qcount = string.Format("SELECT max(LogRow) as MaxRecordNumber FROM {0}",
+                                        fileName);
+                                    var rcount = oLogQuery.Execute(qcount, iFmt);
+                                    var qr = rcount.getRecord();
+                                    var lrn = (Int64) qr.getValueEx("MaxRecordNumber");
+                                    logFileMaxRecords[fileName] = lrn;
                                 }
-                                else
-                                    json.Add(new JProperty(field.Name, v));
                             }
-                            ProcessJson(json);
-                            _receivedMessages++;
+
+                            foreach (string fileName in logFileMaxRecords.Keys.ToList())
+                            {
+                                var lastRecordNumber = logFileMaxRecords[fileName];
+                                var query = string.Format("SELECT * FROM '{0}' Where LogRow > {1}", fileName,
+                                    lastRecordNumber);
+
+                                var rs = oLogQuery.Execute(query, iFmt);
+                                rowReader.ReadColumnMap(rs);
+
+                                // Browse the recordset
+                                for (; !rs.atEnd(); rs.moveNext())
+                                {
+                                    var record = rs.getRecord();
+                                    var json = rowReader.ReadToJson(record);
+                                    ProcessJson(json);
+                                    _receivedMessages++;
+                                    var lrn = (Int64) record.getValueEx("LogRow");
+                                    logFileMaxRecords[fileName] = lrn;
+                                    record = null;
+                                    json = null;
+                                }
+                                // Close the recordset
+                                rs.close();
+                                GC.Collect();
+                            }
+                            if (!Stop)
+                                syncHandle.Wait(TimeSpan.FromSeconds(_pollingIntervalInSeconds), CancelToken);
                         }
+                        catch (OperationCanceledException oce)
+                        {
+                            break;
+                        }
+                        catch (Exception ex)
+                        {
+                            LogManager.GetCurrentClassLogger().Error(ex);
+                        }                       
                     }
-                    // Close the recordset
-                    rs.close();
                 }
-                catch (Exception ex)
-                {
-                    LogManager.GetCurrentClassLogger().Error(ex);
-                }
-                firstQuery = false;
-                System.Threading.Thread.Sleep(_pollingIntervalInSeconds * 1000);
             }
 
             Finished();

TimberWinR/Inputs/IISW3CRowReader.cs

@@ -0,0 +1,69 @@
+namespace TimberWinR.Inputs
+{
+    using System;
+    using System.Collections.Generic;
+
+    using Interop.MSUtil;
+
+    using Newtonsoft.Json.Linq;
+
+    using TimberWinR.Parser;
+
+    public class IisW3CRowReader
+    {
+        private readonly List<Field> fields;
+        private IDictionary<string, int> columnMap;
+
+        public IisW3CRowReader(List<Field> fields)
+        {
+            this.fields = fields;
+        }
+
+        public JObject ReadToJson(ILogRecord row)
+        {
+            var json = new JObject();
+            foreach (var field in this.fields)
+            {
+                if (this.columnMap.ContainsKey(field.Name))
+                {
+                    object v = row.getValue(field.Name);
+                    if (field.DataType == typeof(DateTime))
+                    {
+                        DateTime dt = DateTime.Parse(v.ToString());
+                        json.Add(new JProperty(field.Name, dt));
+                    }
+                    else
+                    {
+                        json.Add(new JProperty(field.Name, v));
+                    }
+                }
+            }
+
+            AddTimestamp(json);
+
+            return json;
+        }
+
+        public void ReadColumnMap(ILogRecordset rs)
+        {
+            this.columnMap = new Dictionary<string, int>();
+            for (int col = 0; col < rs.getColumnCount(); col++)
+            {
+                string colName = rs.getColumnName(col);
+                this.columnMap[colName] = col;
+            }
+        }
+
+        private static void AddTimestamp(JObject json)
+        {
+            if (json["date"] != null && json["time"] != null)
+            {
+                var date = DateTime.Parse(json["date"].ToString());
+                var time = DateTime.Parse(json["time"].ToString());
+                date = new DateTime(date.Year, date.Month, date.Day, time.Hour, time.Minute, time.Second, time.Millisecond);
+
+                json.Add(new JProperty("@timestamp", date.ToString("yyyy-MM-ddTHH:mm:ss.fffZ")));
+            }
+        }
+    }
+}

TimberWinR/Inputs/InputListener.cs

@@ -18,7 +18,7 @@ namespace TimberWinR.Inputs
         private string _typeName;
         public AutoResetEvent FinishedEvent { get; set; }
         public string CheckpointFileName { get; set; }
-
+      
         public string InputType
         {
             get { return _typeName; }
@@ -58,11 +58,17 @@ namespace TimberWinR.Inputs
 
         public void Finished()
         {
+            LogManager.GetCurrentClassLogger().Info("Signaling Event Shutdown {0}", InputType);
             FinishedEvent.Set();
+            LogManager.GetCurrentClassLogger().Info("Finished signaling Shutdown {0}", InputType);
         }
         public virtual void Shutdown()
         {
+            LogManager.GetCurrentClassLogger().Info("Shutting Down {0}", InputType);
+
             FinishedEvent.WaitOne();
+
+            LogManager.GetCurrentClassLogger().Info("Finished Wait For {0}", InputType);
             try
             {
                 if (File.Exists(CheckpointFileName))
@@ -70,7 +76,7 @@ namespace TimberWinR.Inputs
             }
             catch (Exception ex)
             {
-                LogManager.GetCurrentClassLogger().Error("Error Deleting Checkpoint File", ex);
+                LogManager.GetCurrentClassLogger().Error(ex);
             }          
         }
 

TimberWinR/Inputs/LogsFileDatabase.cs

@@ -0,0 +1,207 @@
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Text;
+using Newtonsoft.Json;
+using NLog;
+using TimberWinR.Parser;
+
+
+namespace TimberWinR.Inputs
+{
+    //
+    // Maintain persistent state for Log files (to be used across restarts)
+    //
+    public class LogsFileDatabase
+    {
+        private static readonly object _locker = new object();
+        private List<LogsFileDatabaseEntry> Entries { get; set; }
+        private string DatabaseDirectory { get; set; }
+        public string DatabaseFileName
+        {
+            get { return Path.Combine(DatabaseDirectory, ".timberwinrdb"); }
+        }
+
+        public static Manager Manager { get; set; }
+
+        private static LogsFileDatabase instance;
+
+        private bool ExistingFile(string logName)
+        {
+            lock (_locker)
+            {
+                return ExistingFileTest(logName);
+            }
+        }
+
+        private LogsFileDatabaseEntry FindFile(string logName)
+        {
+            lock (_locker)
+            {
+                var existingEntry = (from e in Entries where e.FileName == logName select e).FirstOrDefault();
+                return existingEntry;
+            }
+        }
+        private bool ExistingFileTest(string logName)
+        {
+            var existingEntry = (from e in Entries where e.FileName == logName select e).FirstOrDefault();
+            return existingEntry != null;
+        }
+
+        private void RemoveFileEntry(string logName)
+        {
+            lock (_locker)
+            {
+                var existingEntry = (from e in Entries where e.FileName == logName select e).FirstOrDefault();
+                if (existingEntry != null)
+                {
+                    Entries.Remove(existingEntry);
+                    WriteDatabaseFileNoLock();
+                }
+            }
+        }
+
+        private LogsFileDatabaseEntry AddFileEntry(string logName)
+        {
+            var de = new LogsFileDatabaseEntry();
+            lock (_locker)
+            {
+                de.NewFile = true;
+                var fi = new FileInfo(logName);
+                de.FileName = logName;
+                de.Size = fi.Length;
+                de.SampleTime = DateTime.UtcNow;
+                de.CreationTimeUtc = fi.CreationTimeUtc;
+                Entries.Add(de);
+                WriteDatabaseFileNoLock();
+            }
+            return de;
+        }
+
+        public static LogsFileDatabaseEntry LookupLogFile(string logName)
+        {
+            LogsFileDatabaseEntry dbe = Instance.FindFile(logName);
+            if (dbe == null)
+                dbe = Instance.AddFileEntry(logName);
+            else
+                dbe.NewFile = false;
+    
+            return dbe;
+        }
+
+        public static void Update(LogsFileDatabaseEntry dbe)
+        {
+            Instance.UpdateEntry(dbe);
+        }
+
+        private void UpdateEntry(LogsFileDatabaseEntry dbe)
+        {
+            lock(_locker)
+            {
+                var fi = new FileInfo(dbe.FileName);
+                dbe.CreationTimeUtc = fi.CreationTimeUtc;
+                dbe.SampleTime = DateTime.UtcNow;
+                dbe.Size = fi.Length;
+                WriteDatabaseFileNoLock();
+            }
+            
+        }
+        public static LogsFileDatabase Instance
+        {
+            get
+            {
+                if (instance == null)
+                {
+                    instance = new LogsFileDatabase(Manager.LogfileDir);
+                    lock (_locker)
+                    {
+                        if (!Directory.Exists(instance.DatabaseDirectory))
+                            Directory.CreateDirectory(instance.DatabaseDirectory);
+                        // If it exists, read the current state, otherwise create an empty database.
+                        if (File.Exists(instance.DatabaseFileName))
+                            instance.ReadDatabaseNoLock();
+                        else
+                            instance.WriteDatabaseFileNoLock();
+                    }
+                }
+                return instance;
+            }
+        }
+
+        private void ReadDatabaseNoLock()
+        {
+            try
+            {
+                var serializer = new JsonSerializer();
+                if (File.Exists(DatabaseFileName))
+                    Entries =
+                        JsonConvert.DeserializeObject<List<LogsFileDatabaseEntry>>(File.ReadAllText(DatabaseFileName));
+            }
+            catch (Exception ex)
+            {
+                LogManager.GetCurrentClassLogger()
+                    .Error("Error reading database '{0}': {1}", DatabaseFileName, ex.ToString());
+                try
+                {
+                    if (File.Exists(DatabaseFileName))
+                        File.Delete(DatabaseFileName);
+                    LogManager.GetCurrentClassLogger().Info("Creating New Database '{0}'", DatabaseFileName);
+                    WriteDatabaseLock();
+                }
+                catch (Exception ex2)
+                {
+                    LogManager.GetCurrentClassLogger().Info("Error Creating New Database '{0}': {1}", DatabaseFileName, ex2.ToString());
+                }              
+            }
+        }
+        private void WriteDatabaseFileNoLock()
+        {
+            try
+            {
+                File.WriteAllText(DatabaseFileName, JsonConvert.SerializeObject(instance.Entries), Encoding.UTF8);
+            }
+            catch (Exception ex)
+            {
+                LogManager.GetCurrentClassLogger()
+                    .Error("Error saving database '{0}': {1}", DatabaseFileName, ex.ToString());
+            }
+        }
+
+
+        private void ReadDatabaseLock()
+        {
+            lock (_locker)
+            {
+                ReadDatabaseNoLock();
+            }
+
+        }
+        private void WriteDatabaseLock()
+        {
+            lock (_locker)
+            {
+                WriteDatabaseFileNoLock();
+            }
+        }
+
+        private LogsFileDatabase(string databaseDirectory)
+        {
+            DatabaseDirectory = databaseDirectory;
+            Entries = new List<LogsFileDatabaseEntry>();
+        }
+
+    }
+
+    public class LogsFileDatabaseEntry
+    {
+        [JsonIgnore]
+        public bool NewFile { get; set; }
+        public string FileName { get; set; }
+        public Int64 MaxRecords { get; set; }
+        public DateTime CreationTimeUtc { get; set; }
+        public DateTime SampleTime { get; set; }
+        public long Size { get; set; }       
+    }
+
+}

TimberWinR/Inputs/LogsListener.cs

@@ -1,19 +1,24 @@
 using System;
 using System.Collections.Generic;
 using System.Linq;
+using System.Net.Configuration;
+using System.Runtime.InteropServices;
 using System.Text;
+using System.Text.RegularExpressions;
 using System.Threading;
 using System.Threading.Tasks;
 using System.IO;
 using Interop.MSUtil;
-
+using Newtonsoft.Json;
 using Newtonsoft.Json.Linq;
 using Newtonsoft.Json.Serialization;
+
 using NLog;
 
 using LogQuery = Interop.MSUtil.LogQueryClassClass;
 using TextLineInputFormat = Interop.MSUtil.COMTextLineInputContextClass;
 using LogRecordSet = Interop.MSUtil.ILogRecordset;
+using TimberWinR.Parser;
 
 namespace TimberWinR.Inputs
 {
@@ -25,111 +30,334 @@ namespace TimberWinR.Inputs
         private int _pollingIntervalInSeconds;
         private TimberWinR.Parser.Log _arguments;
         private long _receivedMessages;
-       
-        public LogsListener(TimberWinR.Parser.Log arguments, CancellationToken cancelToken, int pollingIntervalInSeconds = 3)
+        private Dictionary<string, Int64> _logFileMaxRecords;
+        private Dictionary<string, DateTime> _logFileCreationTimes;
+        private Dictionary<string, DateTime> _logFileSampleTimes;
+        private Dictionary<string, long> _logFileSizes;
+        private Codec _codec;
+        private List<string> _multiline { get; set; }
+
+        public bool Stop { get; set; }
+
+        public LogsListener(TimberWinR.Parser.Log arguments, CancellationToken cancelToken)
             : base(cancelToken, "Win32-FileLog")
         {
+            Stop = false;
+
+            _codec = arguments.Codec;
+            _logFileMaxRecords = new Dictionary<string, Int64>();
+            _logFileCreationTimes = new Dictionary<string, DateTime>();
+            _logFileSampleTimes = new Dictionary<string, DateTime>();
+            _logFileSizes = new Dictionary<string, long>();
+
             _receivedMessages = 0;
             _arguments = arguments;
-            _pollingIntervalInSeconds = pollingIntervalInSeconds;
-            var task = new Task(FileWatcher, cancelToken);
-            task.Start();
+            _pollingIntervalInSeconds = arguments.Interval;
+
+            foreach (string srcFile in _arguments.Location.Split(','))
+            {
+                string file = srcFile.Trim();
+                Task.Factory.StartNew(() => FileWatcher(file));
+            }
         }
 
         public override void Shutdown()
         {
+            LogManager.GetCurrentClassLogger().Info("Shutting Down {0}", InputType);
+            Stop = true;
             base.Shutdown();
         }
 
         public override JObject ToJson()
         {
+
             JObject json = new JObject(
                 new JProperty("log",
                     new JObject(
                         new JProperty("messages", _receivedMessages),
+                        new JProperty("type", InputType),
                         new JProperty("location", _arguments.Location),
+                        new JProperty("logSource", _arguments.LogSource),
                         new JProperty("codepage", _arguments.CodePage),
-                        new JProperty("splitLongLines", _arguments.SplitLongLines),                     
-                        new JProperty("recurse", _arguments.Recurse)
+                        new JProperty("splitLongLines", _arguments.SplitLongLines),
+                        new JProperty("recurse", _arguments.Recurse),
+
+                        new JProperty("files",
+                            new JArray(from f in _logFileMaxRecords.Keys
+                                       select new JValue(f))),
+                        new JProperty("fileSampleTimes",
+                            new JArray(from f in _logFileSampleTimes.Values
+                                       select new JValue(f))),
+                        new JProperty("fileSizes",
+                            new JArray(from f in _logFileSizes.Values
+                                       select new JValue(f))),
+                        new JProperty("fileIndices",
+                            new JArray(from f in _logFileMaxRecords.Values
+                                       select new JValue(f))),
+                        new JProperty("fileCreationDates",
+                            new JArray(from f in _logFileCreationTimes.Values
+                                       select new JValue(f)))
                         )));
+
+
+            if (_codec != null)
+            {
+                var cp = new JProperty("codec",
+                    new JArray(
+                        new JObject(
+                            new JProperty("type", _codec.Type.ToString()),
+                            new JProperty("what", _codec.What.ToString()),
+                            new JProperty("negate", _codec.Negate),
+                            new JProperty("multilineTag", _codec.MultilineTag),
+                            new JProperty("pattern", _codec.Pattern))));
+                json.Add(cp);
+            }
+
+
             return json;
         }
 
-        private void FileWatcher()
-        {                      
+        // return true to cancel codec
+        private void applyMultilineCodec(string msg)
+        {
+            if (_codec.Re == null)
+                _codec.Re = new Regex(_codec.Pattern);
+
+            Match match = _codec.Re.Match(msg);
+
+            bool isMatch = (match.Success && !_codec.Negate) || (!match.Success && _codec.Negate);
+
+            switch (_codec.What)
+            {
+                case Codec.WhatType.previous:
+                    if (isMatch)
+                    {
+                        if (_multiline == null)
+                            _multiline = new List<string>();
+
+                        _multiline.Add(msg);
+                    }
+                    else // No Match
+                    {
+                        if (_multiline != null)
+                        {
+                            string single = string.Join("\n", _multiline.ToArray());
+                            _multiline = null;
+                            JObject jo = new JObject();
+                            jo["message"] = single;
+                            jo.Add("tags", new JArray(_codec.MultilineTag));
+                            AddDefaultFields(jo);
+                            ProcessJson(jo);
+                            _receivedMessages++;
+                        }
+                        _multiline = new List<string>();
+                        _multiline.Add(msg);
+                    }
+                    break;
+                case Codec.WhatType.next:
+                    if (isMatch)
+                    {
+                        if (_multiline == null)
+                            _multiline = new List<string>();
+                        _multiline.Add(msg);
+                    }
+                    else // No match
+                    {
+                        if (_multiline != null)
+                        {
+                            _multiline.Add(msg);
+                            string single = string.Join("\n", _multiline.ToArray());
+                            _multiline = null;
+                            JObject jo = new JObject();
+                            jo["message"] = single;
+                            jo.Add("tags", new JArray(_codec.MultilineTag));
+                            AddDefaultFields(jo);
+                            ProcessJson(jo);
+                            _receivedMessages++;
+                        }
+                        else
+                        {
+                            JObject jo = new JObject();
+                            jo["message"] = msg;
+                            AddDefaultFields(jo);
+                            ProcessJson(jo);
+                            _receivedMessages++;
+                        }
+                    }
+                    break;
+            }
+        }
+
+
+        private void FileWatcher(string fileToWatch)
+        {
             var iFmt = new TextLineInputFormat()
             {
                 iCodepage = _arguments.CodePage,
                 splitLongLines = _arguments.SplitLongLines,
-                iCheckpoint = CheckpointFileName,
                 recurse = _arguments.Recurse
             };
 
-              // Create the query
-            var query = string.Format("SELECT * FROM {0}", _arguments.Location);          
+            Dictionary<string, string> _fnfmap = new Dictionary<string, string>();
 
-            var firstQuery = true;
-            // Execute the query
-            while (!CancelToken.IsCancellationRequested)
+            using (var syncHandle = new ManualResetEventSlim())
             {
-                var oLogQuery = new LogQuery();
-                try
-                {                   
-                    var rs = oLogQuery.Execute(query, iFmt);
-                    Dictionary<string, int> colMap = new Dictionary<string, int>();
-                    for (int col=0; col<rs.getColumnCount(); col++)
+                // Execute the query
+                while (!Stop)
+                {
+                    var oLogQuery = new LogQuery();
+                    if (!CancelToken.IsCancellationRequested)
                     {
-                        string colName = rs.getColumnName(col);
-                        colMap[colName] = col;
-                    }
-
-                    // Browse the recordset
-                    for (; !rs.atEnd(); rs.moveNext())
-                    {
-                        // We want to "tail" the log, so skip the first query results.
-                        if (!firstQuery)
+                        try
                         {
-                            var record = rs.getRecord();
-                            var json = new JObject();
-                            foreach (var field in _arguments.Fields)
+                            var qfiles = string.Format("SELECT Distinct [LogFilename] FROM {0}", fileToWatch);
+                            var rsfiles = oLogQuery.Execute(qfiles, iFmt);
+                            for (; !rsfiles.atEnd(); rsfiles.moveNext())
                             {
-                                if (!colMap.ContainsKey(field.Name))
-                                    continue;
+                                var record = rsfiles.getRecord();
+                                string logName = record.getValue("LogFilename") as string;
+                                FileInfo fi = new FileInfo(logName);
 
-                                object v = record.getValue(field.Name);
-                                if (field.DataType == typeof (DateTime))
+                                if (!fi.Exists)
                                 {
-                                    DateTime dt = DateTime.Parse(v.ToString());
-                                    json.Add(new JProperty(field.Name, dt));
+                                    _logFileCreationTimes.Remove(logName);
+                                    _logFileMaxRecords.Remove(logName);
+                                    _logFileSizes.Remove(logName);
                                 }
-                                else
-                                    json.Add(new JProperty(field.Name, v));
+
+                                _logFileSampleTimes[logName] = DateTime.UtcNow;
+
+                                DateTime creationTime = fi.CreationTimeUtc;
+                                bool logHasRolled = (_logFileCreationTimes.ContainsKey(logName) &&
+                                                     creationTime > _logFileCreationTimes[logName]) ||
+                                                    (_logFileSizes.ContainsKey(logName) &&
+                                                     fi.Length < _logFileSizes[logName]);
+
+
+                                if (!_logFileMaxRecords.ContainsKey(logName) || logHasRolled)
+                                {
+                                    _logFileCreationTimes[logName] = creationTime;
+                                    _logFileSizes[logName] = fi.Length;
+                                    var qcount = string.Format("SELECT max(Index) as MaxRecordNumber FROM {0}", logName);
+                                    var rcount = oLogQuery.Execute(qcount, iFmt);
+                                    var qr = rcount.getRecord();
+                                    var lrn = (Int64)qr.getValueEx("MaxRecordNumber");
+                                    if (logHasRolled)
+                                    {
+                                        LogManager.GetCurrentClassLogger().Info("Log {0} has rolled", logName);
+                                        lrn = 0;
+                                    }
+                                    _logFileMaxRecords[logName] = lrn;
+                                }
+
+                                _logFileSizes[logName] = fi.Length;
                             }
-                            string msg = json["Text"].ToString();
-                            if (!string.IsNullOrEmpty(msg))
+                            rsfiles.close();
+                            foreach (string fileName in _logFileMaxRecords.Keys.ToList())
                             {
-                                ProcessJson(json);
-                                _receivedMessages++;
+                                var lastRecordNumber = _logFileMaxRecords[fileName];
+                                var query = string.Format("SELECT * FROM {0} where Index > {1}", fileName,
+                                    lastRecordNumber);
+
+                                var rs = oLogQuery.Execute(query, iFmt);
+                                Dictionary<string, int> colMap = new Dictionary<string, int>();
+                                for (int col = 0; col < rs.getColumnCount(); col++)
+                                {
+                                    string colName = rs.getColumnName(col);
+                                    colMap[colName] = col;
+                                }
+
+                                // Browse the recordset
+                                for (; !rs.atEnd(); rs.moveNext())
+                                {
+                                    var record = rs.getRecord();
+
+                                    var json = new JObject();
+                                    foreach (var field in _arguments.Fields)
+                                    {
+                                        if (!colMap.ContainsKey(field.Name))
+                                            continue;
+
+                                        if (json["logSource"] == null)
+                                        {
+                                            if (string.IsNullOrEmpty(_arguments.LogSource))
+                                                json.Add(new JProperty("logSource", fileName));
+                                            else
+                                                json.Add(new JProperty("logSource", _arguments.LogSource));
+                                        }
+
+                                        object v = record.getValue(field.Name);
+                                        if (field.DataType == typeof(DateTime))
+                                        {
+                                            DateTime dt = DateTime.Parse(v.ToString());
+                                            json.Add(new JProperty(field.Name, dt));
+                                        }
+                                        else
+                                            json.Add(new JProperty(field.Name, v));
+                                    }
+                                    string msg = json["Text"].ToString();
+                                    if (!string.IsNullOrEmpty(msg))
+                                    {
+                                        if (_codec != null && _codec.Type == Codec.CodecType.multiline)
+                                            applyMultilineCodec(msg);
+                                        else
+                                        {
+                                            ProcessJson(json);
+                                            _receivedMessages++;
+                                        }
+                                    }
+
+                                    var lrn = (Int64)record.getValueEx("Index");
+                                    _logFileMaxRecords[fileName] = lrn;
+                                    GC.Collect();
+                                }
+
+                                colMap.Clear();
+                                // Close the recordset
+                                rs.close();
+                                rs = null;
+                                GC.Collect();
+                            }
+                        }
+                        catch (FileNotFoundException fnfex)
+                        {
+                            string fn = fnfex.FileName;
+
+                            if (!_fnfmap.ContainsKey(fn))
+                                LogManager.GetCurrentClassLogger().Warn(fnfex.Message);
+
+                            _fnfmap[fn] = fn;
+                        }
+                        catch (OperationCanceledException oce)
+                        {
+                            break;
+                        }
+                        catch (Exception ex)
+                        {
+                            LogManager.GetCurrentClassLogger().Error(ex);
+                        }
+                        finally
+                        {
+                            try
+                            {
+                                oLogQuery = null;
+                                // Sleep 
+                                if (!Stop)
+                                    syncHandle.Wait(TimeSpan.FromSeconds(_pollingIntervalInSeconds), CancelToken);
+                            }
+                            catch (OperationCanceledException oce)
+                            {                               
+                            }
+                            catch (Exception ex1)
+                            {
+                                LogManager.GetCurrentClassLogger().Warn(ex1);
                             }
                         }
                     }
-                    // Close the recordset
-                    rs.close();
-                    rs = null;
                 }
-                catch (Exception ex)
-                {
-                    LogManager.GetCurrentClassLogger().Error(ex);
-                }
-                finally
-                {
-                    oLogQuery = null;
-                }
-                firstQuery = false;
-                System.Threading.Thread.Sleep(_pollingIntervalInSeconds * 1000);
+                Finished();
             }
-
-            Finished();
-        }       
+        }
     }
 }
+

TimberWinR/Inputs/StdinListener.cs

@@ -1,21 +1,27 @@
 using System;
 using System.Collections.Generic;
+using System.Diagnostics;
 using System.Linq;
 using System.Text;
+using System.Text.RegularExpressions;
 using System.Threading;
 using Newtonsoft.Json;
 using Newtonsoft.Json.Linq;
 using NLog;
+using TimberWinR.Parser;
 
 namespace TimberWinR.Inputs
 {
     public class StdinListener : InputListener
     {
         private Thread _listenThread;
+        private Codec _codec;
+        private List<string> _multiline { get; set; }
 
-        public StdinListener(CancellationToken cancelToken)
+        public StdinListener(TimberWinR.Parser.Stdin arguments, CancellationToken cancelToken)
             : base(cancelToken, "Win32-Console")
         {
+            _codec = arguments.Codec;
             _listenThread = new Thread(new ThreadStart(ListenToStdin));
             _listenThread.Start();
         }
@@ -23,12 +29,28 @@ namespace TimberWinR.Inputs
         public override JObject ToJson()
         {
             JObject json = new JObject(
-                new JProperty("stdin", "enabled"));                 
+                new JProperty("stdin", "enabled"));
+
+            
+            if (_codec != null)
+            {
+                var cp = new JProperty("codec",
+                    new JArray(
+                        new JObject(
+                            new JProperty("type", _codec.Type.ToString()),
+                            new JProperty("what", _codec.What.ToString()),
+                            new JProperty("negate", _codec.Negate),
+                            new JProperty("multilineTag", _codec.MultilineTag),
+                            new JProperty("pattern", _codec.Pattern))));
+                json.Add(cp);              
+            }
+
             return json;
         }
 
         public override void Shutdown()
-        {           
+        {
+            LogManager.GetCurrentClassLogger().Info("Shutting Down {0}", InputType);
             base.Shutdown();
         }
 
@@ -37,20 +59,92 @@ namespace TimberWinR.Inputs
             LogManager.GetCurrentClassLogger().Info("StdIn Ready");
 
             while (!CancelToken.IsCancellationRequested)
-            {                       
+            {
                 string line = Console.ReadLine();
                 if (line != null)
                 {
                     string msg = ToPrintable(line);
-                    JObject jo = new JObject();
-                    jo["message"] = msg;
-                    AddDefaultFields(jo);                                    
-                    ProcessJson(jo);
-                }
-                else              
-                    break;              
+
+                    if (_codec != null && _codec.Type == Codec.CodecType.multiline)
+                        applyMultilineCodec(msg);
+                    else
+                    {
+                        JObject jo = new JObject();
+                        jo["message"] = msg;
+                        AddDefaultFields(jo);
+                        ProcessJson(jo);
+                    }
+                }              
             }
             Finished();
         }
+
+        // return true to cancel codec
+        private void applyMultilineCodec(string msg)
+        {
+            if (_codec.Re == null)
+                _codec.Re = new Regex(_codec.Pattern);
+
+            Match match = _codec.Re.Match(msg);
+
+            bool isMatch = (match.Success && !_codec.Negate) || (!match.Success && _codec.Negate);
+
+            switch (_codec.What)
+            {
+                case Codec.WhatType.previous:
+                    if (isMatch)
+                    {
+                        if (_multiline == null)
+                            _multiline = new List<string>();
+
+                        _multiline.Add(msg);
+                    }
+                    else // No Match
+                    {
+                        if (_multiline != null)
+                        {
+                            string single = string.Join("\n", _multiline.ToArray());
+                            _multiline = null;
+                            JObject jo = new JObject();
+                            jo["message"] = single;
+                            jo.Add("tags", new JArray(_codec.MultilineTag));
+                            AddDefaultFields(jo);
+                            ProcessJson(jo);                          
+                        }
+                        _multiline = new List<string>();
+                        _multiline.Add(msg);
+                    }
+                    break;
+                case Codec.WhatType.next:
+                    if (isMatch)
+                    {
+                        if (_multiline == null)
+                            _multiline = new List<string>();
+                        _multiline.Add(msg);
+                    }
+                    else // No match
+                    {
+                        if (_multiline != null)
+                        {
+                            _multiline.Add(msg);
+                            string single = string.Join("\n", _multiline.ToArray());
+                            _multiline = null;
+                            JObject jo = new JObject();
+                            jo["message"] = single;
+                            jo.Add("tags", new JArray(_codec.MultilineTag));
+                            AddDefaultFields(jo);
+                            ProcessJson(jo);                          
+                        }
+                        else
+                        {
+                            JObject jo = new JObject();
+                            jo["message"] = msg;
+                            AddDefaultFields(jo);
+                            ProcessJson(jo);                          
+                        }
+                    }
+                    break;
+            }
+        }
     }
 }

TimberWinR/Inputs/TailFileListener.cs

@@ -0,0 +1,328 @@
+using System;
+using System.Collections.Generic;
+using System.Diagnostics;
+using System.Linq;
+using System.Net.Configuration;
+using System.Runtime.InteropServices;
+using System.Text;
+using System.Text.RegularExpressions;
+using System.Threading;
+using System.Threading.Tasks;
+using System.IO;
+using Interop.MSUtil;
+using Newtonsoft.Json;
+using Newtonsoft.Json.Linq;
+using Newtonsoft.Json.Serialization;
+
+using NLog;
+using NLog.LayoutRenderers;
+using TimberWinR.Parser;
+
+namespace TimberWinR.Inputs
+{
+    /// <summary>
+    /// Tail a file.
+    /// </summary>
+    public class TailFileListener : InputListener
+    {
+        private int _pollingIntervalInSeconds;
+        private TimberWinR.Parser.TailFile _arguments;
+        private long _receivedMessages;
+        private Dictionary<string, Int64> _logFileMaxRecords;
+        private Dictionary<string, DateTime> _logFileCreationTimes;
+        private Dictionary<string, DateTime> _logFileSampleTimes;
+        private Dictionary<string, long> _logFileSizes;
+        private Codec _codec;
+        private List<string> _multiline { get; set; }
+
+        public bool Stop { get; set; }
+
+        public TailFileListener(TimberWinR.Parser.TailFile arguments, CancellationToken cancelToken)
+            : base(cancelToken, "Win32-TailLog")
+        {
+            Stop = false;
+
+            _codec = arguments.Codec;
+            _logFileMaxRecords = new Dictionary<string, Int64>();
+            _logFileCreationTimes = new Dictionary<string, DateTime>();
+            _logFileSampleTimes = new Dictionary<string, DateTime>();
+            _logFileSizes = new Dictionary<string, long>();
+
+            _receivedMessages = 0;
+            _arguments = arguments;
+            _pollingIntervalInSeconds = arguments.Interval;
+
+            foreach (string srcFile in _arguments.Location.Split(','))
+            {
+                string file = srcFile.Trim();
+                Task.Factory.StartNew(() => TailFileWatcher(file));
+            }
+        }
+
+        public override void Shutdown()
+        {
+            LogManager.GetCurrentClassLogger().Info("Shutting Down {0}", InputType);
+            Stop = true;
+            base.Shutdown();
+        }
+
+        public override JObject ToJson()
+        {
+            JObject json = new JObject(
+                new JProperty("log",
+                    new JObject(
+                        new JProperty("messages", _receivedMessages),
+                        new JProperty("type", InputType),
+                        new JProperty("location", _arguments.Location),
+                        new JProperty("logSource", _arguments.LogSource),
+                        new JProperty("recurse", _arguments.Recurse),
+
+                        new JProperty("files",
+                            new JArray(from f in _logFileMaxRecords.Keys
+                                       select new JValue(f))),
+                        new JProperty("fileSampleTimes",
+                            new JArray(from f in _logFileSampleTimes.Values
+                                       select new JValue(f))),
+                        new JProperty("fileSizes",
+                            new JArray(from f in _logFileSizes.Values
+                                       select new JValue(f))),
+                        new JProperty("fileIndices",
+                            new JArray(from f in _logFileMaxRecords.Values
+                                       select new JValue(f))),
+                        new JProperty("fileCreationDates",
+                            new JArray(from f in _logFileCreationTimes.Values
+                                       select new JValue(f)))
+                        )));
+
+
+            if (_codec != null)
+            {
+                var cp = new JProperty("codec",
+                    new JArray(
+                        new JObject(
+                            new JProperty("type", _codec.Type.ToString()),
+                            new JProperty("what", _codec.What.ToString()),
+                            new JProperty("negate", _codec.Negate),
+                            new JProperty("multilineTag", _codec.MultilineTag),
+                            new JProperty("pattern", _codec.Pattern))));
+                json.Add(cp);
+            }
+
+
+            return json;
+        }
+
+        // return true to cancel codec
+        private void applyMultilineCodec(string msg)
+        {
+            if (_codec.Re == null)
+                _codec.Re = new Regex(_codec.Pattern);
+
+            Match match = _codec.Re.Match(msg);
+
+            bool isMatch = (match.Success && !_codec.Negate) || (!match.Success && _codec.Negate);
+
+            switch (_codec.What)
+            {
+                case Codec.WhatType.previous:
+                    if (isMatch)
+                    {
+                        if (_multiline == null)
+                            _multiline = new List<string>();
+
+                        _multiline.Add(msg);
+                    }
+                    else // No Match
+                    {
+                        if (_multiline != null)
+                        {
+                            string single = string.Join("\n", _multiline.ToArray());
+                            _multiline = null;
+                            JObject jo = new JObject();
+                            jo["message"] = single;
+                            jo.Add("tags", new JArray(_codec.MultilineTag));
+                            AddDefaultFields(jo);
+                            ProcessJson(jo);
+                            _receivedMessages++;
+                        }
+                        _multiline = new List<string>();
+                        _multiline.Add(msg);
+                    }
+                    break;
+                case Codec.WhatType.next:
+                    if (isMatch)
+                    {
+                        if (_multiline == null)
+                            _multiline = new List<string>();
+                        _multiline.Add(msg);
+                    }
+                    else // No match
+                    {
+                        if (_multiline != null)
+                        {
+                            _multiline.Add(msg);
+                            string single = string.Join("\n", _multiline.ToArray());
+                            _multiline = null;
+                            JObject jo = new JObject();
+                            jo["message"] = single;
+                            jo.Add("tags", new JArray(_codec.MultilineTag));
+                            AddDefaultFields(jo);
+                            ProcessJson(jo);
+                            _receivedMessages++;
+                        }
+                        else
+                        {
+                            JObject jo = new JObject();
+                            jo["message"] = msg;
+                            AddDefaultFields(jo);
+                            ProcessJson(jo);
+                            _receivedMessages++;
+                        }
+                    }
+                    break;
+            }
+        }
+
+        private void TailFileContents(string fileName, long offset)
+        {
+            using (StreamReader reader = new StreamReader(new FileStream(fileName,
+                     FileMode.Open, FileAccess.Read, FileShare.ReadWrite)))
+            {
+                //start at the end of the file
+                long lastMaxOffset = offset;
+
+                //if the file size has not changed, idle
+                if (reader.BaseStream.Length == lastMaxOffset)
+                    return;
+
+                //seek to the last max offset
+                reader.BaseStream.Seek(lastMaxOffset, SeekOrigin.Begin);
+
+                //read out of the file until the EOF
+                string line = "";
+                long lineOffset = 0;
+                while ((line = reader.ReadLine()) != null)
+                {
+                    if (string.IsNullOrEmpty(line))
+                        continue;
+
+                    long index = lastMaxOffset + lineOffset;
+                    string text = line;
+                    string logFileName = fileName;
+                    var json = new JObject();
+
+                    if (json["logSource"] == null)
+                    {
+                        if (string.IsNullOrEmpty(_arguments.LogSource))
+                            json.Add(new JProperty("logSource", fileName));
+                        else
+                            json.Add(new JProperty("logSource", _arguments.LogSource));
+                    }
+                    json["Text"] = line;
+                    json["Index"] = index;
+                    json["LogFileName"] = fileName;
+
+                    if (_codec != null && _codec.Type == Codec.CodecType.multiline)
+                        applyMultilineCodec(line);
+                    else
+                    {
+                        ProcessJson(json);
+                        Interlocked.Increment(ref _receivedMessages);
+                    }
+                    lineOffset += line.Length;
+                    // Console.WriteLine("File: {0}:{1}: {2}", fileName, reader.BaseStream.Position, line);
+                }
+                //update the last max offset
+                lastMaxOffset = reader.BaseStream.Position;
+            }
+        }
+        // One thread for each kind of file to watch, i.e. "*.log,*.txt" would be two separate
+        // threads.
+        private void TailFileWatcher(string fileToWatch)
+        {
+            Dictionary<string, string> _fnfmap = new Dictionary<string, string>();
+
+            using (var syncHandle = new ManualResetEventSlim())
+            {
+                // Execute the query
+                while (!Stop && !CancelToken.IsCancellationRequested)
+                {
+                    try
+                    {
+                        if (!CancelToken.IsCancellationRequested)
+                        {
+                            string path = Path.GetDirectoryName(fileToWatch);
+                            string name = Path.GetFileName(fileToWatch);
+                            if (string.IsNullOrEmpty(path))
+                                path = ".";
+
+                            // Ok, we have a potential file filter here as 'fileToWatch' could be foo.log or *.log
+
+                            SearchOption so = SearchOption.TopDirectoryOnly;
+                            if (_arguments.Recurse == -1)
+                                so = SearchOption.AllDirectories;
+
+                            foreach (string fileName in Directory.GetFiles(path, name, so))
+                            {
+                                var dbe = LogsFileDatabase.LookupLogFile(fileName);
+                                FileInfo fi = new FileInfo(dbe.FileName);
+                                //LogManager.GetCurrentClassLogger().Info("Located File: {0}, New: {1}", dbe.FileName, dbe.NewFile);
+                                long length = fi.Length;
+                                bool logHasRolled = false;
+                                if (fi.Length < dbe.Size || fi.CreationTimeUtc != dbe.CreationTimeUtc)
+                                {
+                                    LogManager.GetCurrentClassLogger().Info("Log has Rolled: {0}", dbe.FileName);
+                                    logHasRolled = true;
+                                }
+                                bool processWholeFile = logHasRolled || dbe.NewFile;
+                                if (processWholeFile)
+                                {
+                                    LogManager.GetCurrentClassLogger().Info("Process Whole File: {0}", dbe.FileName);
+                                    TailFileContents(dbe.FileName, 0);
+                                }
+                                else
+                                {
+                                    TailFileContents(dbe.FileName, dbe.Size);
+                                }
+                                LogsFileDatabase.Update(dbe);
+                            }
+                        }
+                    }
+                    catch (FileNotFoundException fnfex)
+                    {
+                        string fn = fnfex.FileName;
+
+                        if (!_fnfmap.ContainsKey(fn))
+                            LogManager.GetCurrentClassLogger().Warn(fnfex.Message);
+                        _fnfmap[fn] = fn;
+                    }
+                    catch (OperationCanceledException oce)
+                    {
+                        break;
+                    }
+                    catch (Exception ex)
+                    {
+                        LogManager.GetCurrentClassLogger().Error(ex);
+                    }
+                    finally
+                    {
+                        try
+                        {
+                            if (!Stop)
+                                syncHandle.Wait(TimeSpan.FromSeconds(_pollingIntervalInSeconds), CancelToken);
+                        }
+                        catch (OperationCanceledException)
+                        {
+                        }
+                        catch (Exception ex1)
+                        {
+                            LogManager.GetCurrentClassLogger().Warn(ex1);
+                        }                      
+                    }
+                }
+            }
+            Finished();
+        }
+    }
+}
+

TimberWinR/Inputs/TcpInputListener.cs

@@ -1,11 +1,9 @@
 using System;
-using System.Collections.Generic;
 using System.IO;
-using System.Linq;
-using System.Text;
 using System.Threading;
 using System.Net;
 using System.Net.Sockets;
+using Newtonsoft.Json;
 using Newtonsoft.Json.Linq;
 using NLog;
 
@@ -36,9 +34,10 @@ namespace TimberWinR.Inputs
             : base(cancelToken, "Win32-Tcp")
         {
             _port = port;
-          
+
             LogManager.GetCurrentClassLogger().Info("Tcp Input(v4/v6) on Port {0} Ready", _port);
 
+            _receivedMessages = 0;
 
             _tcpListenerV6 = new System.Net.Sockets.TcpListener(IPAddress.IPv6Any, port);
             _tcpListenerV4 = new System.Net.Sockets.TcpListener(IPAddress.Any, port);
@@ -53,6 +52,8 @@ namespace TimberWinR.Inputs
 
         public override void Shutdown()
         {
+            LogManager.GetCurrentClassLogger().Info("Shutting Down {0}", InputType);
+
             this._tcpListenerV4.Stop();
             this._tcpListenerV6.Stop();
 
@@ -67,7 +68,7 @@ namespace TimberWinR.Inputs
 
             listener.Start();
 
-          
+
             while (!CancelToken.IsCancellationRequested)
             {
                 try
@@ -92,37 +93,38 @@ namespace TimberWinR.Inputs
         private void HandleNewClient(object client)
         {
             var tcpClient = (TcpClient)client;
-            NetworkStream clientStream = null;
 
             try
             {
-                clientStream = tcpClient.GetStream();
-                var stream = new StreamReader(clientStream);
-                string line;
-                while ((line = stream.ReadLine()) != null)
+                NetworkStream clientStream = tcpClient.GetStream();
+                using (var stream = new StreamReader(clientStream))
                 {
-                    try
+                    //assume a continuous stream of JSON objects
+                    using (var reader = new JsonTextReader(stream) { SupportMultipleContent = true })
                     {
-                        JObject json = JObject.Parse(line);
-                        ProcessJson(json);
-                        _receivedMessages++;
+                        while (reader.Read())
+                        {
+                            if (CancelToken.IsCancellationRequested) break;
+                            try
+                            {
+                                JObject json = JObject.Load(reader);
+                                ProcessJson(json);
+                                _receivedMessages++;
+                            }
+                            catch (Exception ex)
+                            {
+                                LogManager.GetCurrentClassLogger().Warn(ex);
+                            }
+
+                        }
                     }
-                    catch (Exception ex)
-                    {
-                        LogManager.GetCurrentClassLogger().Error(ex);
-                    }
-                    if (CancelToken.IsCancellationRequested)
-                        break;
                 }
             }
             catch (Exception ex)
             {
-                LogManager.GetCurrentClassLogger().Error("Tcp Exception", ex);
+                LogManager.GetCurrentClassLogger().Error(ex);
             }
 
-            if (clientStream != null)
-                clientStream.Close();
-
             tcpClient.Close();
             Finished();
         }

TimberWinR/Inputs/UdpInputListener.cs

@@ -0,0 +1,108 @@
+using System;
+using System.IO;
+using System.Text;
+using System.Threading;
+using System.Net;
+using System.Net.Sockets;
+using Newtonsoft.Json;
+using Newtonsoft.Json.Linq;
+using NLog;
+
+namespace TimberWinR.Inputs
+{
+    public class UdpInputListener : InputListener
+    {
+        private readonly System.Net.Sockets.UdpClient _udpListener;
+        private IPEndPoint groupV4;
+        private IPEndPoint groupV6;
+
+        private Thread _listenThreadV4;
+        private Thread _listenThreadV6;
+
+        private readonly int _port;
+        private long _receivedMessages;
+        private long _parsedErrors;
+
+        private struct listenProfile
+        {
+            public IPEndPoint endPoint;
+            public UdpClient client;
+        }
+
+        public override JObject ToJson()
+        {
+            JObject json = new JObject(
+                new JProperty("udp",
+                    new JObject(
+                        new JProperty("port", _port),
+                        new JProperty("errors", _parsedErrors),
+                        new JProperty("messages", _receivedMessages)
+                        )));
+
+            return json;
+        }
+
+        public UdpInputListener(CancellationToken cancelToken, int port = 5140)
+            : base(cancelToken, "Win32-Udp")
+        {
+            _port = port;
+
+            LogManager.GetCurrentClassLogger().Info("Udp Input on Port {0} Ready", _port);
+
+            _receivedMessages = 0;
+
+            _udpListener = new System.Net.Sockets.UdpClient(port);
+
+            _listenThreadV4 = new Thread(new ParameterizedThreadStart(StartListener));
+            _listenThreadV4.Start(new listenProfile() { endPoint = groupV4, client = _udpListener });
+
+            _listenThreadV6 = new Thread(new ParameterizedThreadStart(StartListener));
+            _listenThreadV6.Start(new listenProfile() { endPoint = groupV6, client = _udpListener });
+        }
+
+
+        public override void Shutdown()
+        {
+            LogManager.GetCurrentClassLogger().Info("Shutting Down {0}", InputType);
+            _udpListener.Close();
+            Finished();
+            base.Shutdown();
+        }
+
+
+        private void StartListener(object useProfile)
+        {
+            var profile = (listenProfile)useProfile;
+            string lastMessage = "";
+            try
+            {
+                while (!CancelToken.IsCancellationRequested)
+                {
+                    try
+                    {
+                        byte[] bytes = profile.client.Receive(ref profile.endPoint);  
+                        var data = Encoding.UTF8.GetString(bytes, 0, bytes.Length);
+                        lastMessage = data;
+                        JObject json = JObject.Parse(data);
+                        ProcessJson(json);
+                        _receivedMessages++;
+                    }
+                    catch (Exception ex1)
+                    {
+                        LogManager.GetCurrentClassLogger().Warn("Bad JSON: {0}", lastMessage);
+                        LogManager.GetCurrentClassLogger().Warn(ex1);
+                        _parsedErrors++;
+                    }
+                }
+                _udpListener.Close();
+            }
+            catch (Exception ex)
+            {
+                if (!CancelToken.IsCancellationRequested)
+                    LogManager.GetCurrentClassLogger().Error(ex);
+            }
+
+            Finished();
+        }
+    }
+}

TimberWinR/Inputs/W3CInputListener.cs

@@ -0,0 +1,171 @@
+using System;
+using System.Collections.Generic;
+using System.ComponentModel;
+using System.Linq;
+using System.Security.AccessControl;
+using System.Text;
+using System.Threading;
+using System.Threading.Tasks;
+using System.IO;
+using Interop.MSUtil;
+
+using Newtonsoft.Json.Linq;
+using Newtonsoft.Json.Serialization;
+using NLog;
+using TimberWinR.Parser;
+using LogQuery = Interop.MSUtil.LogQueryClassClass;
+using W3CLogInputFormat = Interop.MSUtil.COMW3CInputContextClassClass;
+using LogRecordSet = Interop.MSUtil.ILogRecordset;
+
+
+namespace TimberWinR.Inputs
+{
+    public class W3CInputListener : InputListener
+    {
+        private readonly int _pollingIntervalInSeconds;
+        private readonly TimberWinR.Parser.W3CLog _arguments;
+        private long _receivedMessages;
+        public bool Stop { get; set; }
+
+        public W3CInputListener(TimberWinR.Parser.W3CLog arguments, CancellationToken cancelToken, int pollingIntervalInSeconds = 5)
+            : base(cancelToken, "Win32-W3CLog")
+        {
+            _arguments = arguments;
+            _receivedMessages = 0;
+            _pollingIntervalInSeconds = pollingIntervalInSeconds;
+            foreach (string loc in _arguments.Location.Split(','))
+            {
+                string hive = loc.Trim();
+                Task.Factory.StartNew(() => IISW3CWatcher(loc));
+            }
+        }
+
+        public override void Shutdown()
+        {
+            Stop = true;
+            LogManager.GetCurrentClassLogger().Info("Shutting Down {0}", InputType);
+            base.Shutdown();
+        }
+
+        public override JObject ToJson()
+        {
+            JObject json = new JObject(
+                new JProperty("iisw3c",
+                    new JObject(
+                        new JProperty("messages", _receivedMessages),
+                        new JProperty("location", _arguments.Location),
+                        new JProperty("codepage", _arguments.CodePage),
+                        new JProperty("separator", _arguments.Separator),
+                        new JProperty("dQuotes", _arguments.DoubleQuotes),
+                        new JProperty("dtLines", _arguments.DtLines)
+                        )));
+            return json;
+        }
+
+
+        private void IISW3CWatcher(string location)
+        {
+            LogManager.GetCurrentClassLogger().Info("IISW3Listener Ready For {0}", location);
+
+            var oLogQuery = new LogQuery();
+
+            var iFmt = new W3CLogInputFormat()
+            {
+                codepage = _arguments.CodePage,
+                iCodepage = _arguments.CodePage,
+                doubleQuotedStrings = _arguments.DoubleQuotes,
+                detectTypesLines = _arguments.DtLines,
+                dQuotes = _arguments.DoubleQuotes,
+                separator = _arguments.Separator
+            };
+
+            Dictionary<string, Int64> logFileMaxRecords = new Dictionary<string, Int64>();
+            using (var syncHandle = new ManualResetEventSlim())
+            {
+                // Execute the query
+                while (!Stop)
+                {
+                    // Execute the query
+                    if (!CancelToken.IsCancellationRequested)
+                    {
+                        try
+                        {
+                            oLogQuery = new LogQuery();
+
+                            var qfiles = string.Format("SELECT Distinct [LogFilename] FROM {0}", location);
+                            var rsfiles = oLogQuery.Execute(qfiles, iFmt);
+                            for (; !rsfiles.atEnd(); rsfiles.moveNext())
+                            {
+                                var record = rsfiles.getRecord();
+                                string fileName = record.getValue("LogFilename") as string;
+                                if (!logFileMaxRecords.ContainsKey(fileName))
+                                {
+                                    var qcount = string.Format("SELECT max(RowNumber) as MaxRecordNumber FROM {0}",
+                                        fileName);
+                                    var rcount = oLogQuery.Execute(qcount, iFmt);
+                                    var qr = rcount.getRecord();
+                                    var lrn = (Int64)qr.getValueEx("MaxRecordNumber");
+                                    logFileMaxRecords[fileName] = lrn;
+                                }
+                            }
+
+
+                            foreach (string fileName in logFileMaxRecords.Keys.ToList())
+                            {
+                                var lastRecordNumber = logFileMaxRecords[fileName];
+                                var query = string.Format(
+                                    "SELECT * FROM '{0}' Where RowNumber > {1} order by RowNumber", fileName,
+                                    lastRecordNumber);
+                                var rs = oLogQuery.Execute(query, iFmt);
+                                var colMap = new Dictionary<string, int>();
+                                for (int col = 0; col < rs.getColumnCount(); col++)
+                                {
+                                    string colName = rs.getColumnName(col);
+                                    colMap[colName] = col;
+                                }
+
+                                // Browse the recordset
+                                for (; !rs.atEnd(); rs.moveNext())
+                                {
+                                    var record = rs.getRecord();
+                                    var json = new JObject();
+                                    foreach (var field in colMap.Keys)
+                                    {
+                                        object v = record.getValue(field);
+                                        if (field == "date" || field == "time")
+                                        {
+                                            DateTime dt = DateTime.Parse(v.ToString());
+                                            json.Add(new JProperty(field, dt));
+                                        }
+                                        else
+                                            json.Add(new JProperty(field, v));
+                                    }
+                                    ProcessJson(json);
+                                    _receivedMessages++;
+                                    var lrn = (Int64)record.getValueEx("RowNumber");
+                                    logFileMaxRecords[fileName] = lrn;
+                                    record = null;
+                                    json = null;
+                                }
+                                // Close the recordset
+                                rs.close();
+                            }
+                            if (!Stop)
+                                syncHandle.Wait(TimeSpan.FromSeconds(_pollingIntervalInSeconds), CancelToken);
+                        }
+                        catch (OperationCanceledException oce)
+                        {
+                            break;
+                        }
+                        catch (Exception ex)
+                        {
+                            LogManager.GetCurrentClassLogger().Error(ex);
+                        }
+                    }
+                }
+            }
+
+            Finished();
+        }
+    }
+}

TimberWinR/Inputs/WindowsEvtInputListener.cs

@@ -1,6 +1,7 @@
 using System;
 using System.Collections.Generic;
 using System.Linq;
+using System.Runtime.InteropServices;
 using System.Security.AccessControl;
 using System.Text;
 using System.Threading;
@@ -25,18 +26,29 @@ namespace TimberWinR.Inputs
         private int _pollingIntervalInSeconds = 1;
         private TimberWinR.Parser.WindowsEvent _arguments;
         private long _receivedMessages;
+        private List<Thread> _tasks { get; set; }
+        public bool Stop { get; set; }
 
-        public WindowsEvtInputListener(TimberWinR.Parser.WindowsEvent arguments, CancellationToken cancelToken, int pollingIntervalInSeconds = 1)
+        public WindowsEvtInputListener(TimberWinR.Parser.WindowsEvent arguments, CancellationToken cancelToken)
             : base(cancelToken, "Win32-Eventlog")
         {
             _arguments = arguments;
-            _pollingIntervalInSeconds = pollingIntervalInSeconds;
-            var task = new Task(EventWatcher, cancelToken);
-            task.Start();
+            _pollingIntervalInSeconds = arguments.Interval;
+            _tasks = new List<Thread>();
+
+            foreach (string eventHive in _arguments.Source.Split(','))
+            {
+                string hive = eventHive.Trim();
+                var thread = new Thread(new ParameterizedThreadStart(EventWatcher));
+                _tasks.Add(thread);
+                thread.Start(eventHive);
+            }
         }
 
         public override void Shutdown()
         {
+            Stop = true;
+            LogManager.GetCurrentClassLogger().Info("Shutting Down {0}", InputType);           
             base.Shutdown();
         }
 
@@ -48,6 +60,7 @@ namespace TimberWinR.Inputs
                         new JProperty("messages", _receivedMessages),
                         new JProperty("binaryFormat", _arguments.BinaryFormat.ToString()),
                         new JProperty("direction", _arguments.Direction.ToString()),
+                        new JProperty("interval", _arguments.Interval),
                         new JProperty("formatMsg", _arguments.FormatMsg),
                         new JProperty("fullEventCode", _arguments.FullEventCode),
                         new JProperty("fullText", _arguments.FullText),
@@ -59,9 +72,11 @@ namespace TimberWinR.Inputs
             return json;
         }
 
-        private void EventWatcher()
+        private void EventWatcher(object ploc)
         {
-            var oLogQuery = new LogQuery();
+            string location = ploc.ToString();
+
+            LogQuery oLogQuery = new LogQuery();
 
             LogManager.GetCurrentClassLogger().Info("WindowsEvent Input Listener Ready");
 
@@ -75,55 +90,93 @@ namespace TimberWinR.Inputs
                 fullText = _arguments.FullText,
                 msgErrorMode = _arguments.MsgErrorMode.ToString(),
                 stringsSep = _arguments.StringsSep,
-                resolveSIDs = _arguments.ResolveSIDS,
-                iCheckpoint = CheckpointFileName,
+                resolveSIDs = _arguments.ResolveSIDS
             };
 
+            oLogQuery = null;
 
-            // Create the query
-            var query = string.Format("SELECT * FROM {0}", _arguments.Source);
+            Dictionary<string, Int64> logFileMaxRecords = new Dictionary<string, Int64>();
 
-            var firstQuery = true;
-            // Execute the query
-            while (!CancelToken.IsCancellationRequested)
+            using (var syncHandle = new ManualResetEventSlim())
             {
-                try
+                // Execute the query
+                while (!Stop)
                 {
-                    var rs = oLogQuery.Execute(query, iFmt);
-                    // Browse the recordset
-                    for (; !rs.atEnd(); rs.moveNext())
+                    // Execute the query
+                    if (!CancelToken.IsCancellationRequested)
                     {
-                        // We want to "tail" the log, so skip the first query results.
-                        if (!firstQuery)
-                        {
-                            var record = rs.getRecord();
-                            var json = new JObject();
-                            foreach (var field in _arguments.Fields)
+                        try
+                        {                          
+                            oLogQuery = new LogQuery();
+
+                            var qfiles = string.Format("SELECT Distinct [EventLog] FROM {0}", location);
+                            var rsfiles = oLogQuery.Execute(qfiles, iFmt);
+                            for (; !rsfiles.atEnd(); rsfiles.moveNext())
                             {
-                                object v = record.getValue(field.Name);
-                                if (field.Name == "Data")
-                                    v = ToPrintable(v.ToString());
-                                json.Add(new JProperty(field.Name, v));
+                                var record = rsfiles.getRecord();
+                                string logName = record.getValue("EventLog") as string;
+                                if (!logFileMaxRecords.ContainsKey(logName))
+                                {
+                                    var qcount = string.Format("SELECT max(RecordNumber) as MaxRecordNumber FROM {0}",
+                                        logName);
+                                    var rcount = oLogQuery.Execute(qcount, iFmt);
+                                    var qr = rcount.getRecord();
+                                    var lrn = (Int64)qr.getValueEx("MaxRecordNumber");
+                                    logFileMaxRecords[logName] = lrn;
+                                }
                             }
 
-                            ProcessJson(json);
-                            _receivedMessages++;
+
+                            foreach (string fileName in logFileMaxRecords.Keys.ToList())
+                            {
+                                var lastRecordNumber = logFileMaxRecords[fileName];
+                                var query = string.Format("SELECT * FROM {0} where RecordNumber > {1}", location,
+                                    lastRecordNumber);
+
+                                var rs = oLogQuery.Execute(query, iFmt);
+                                // Browse the recordset
+                                for (; !rs.atEnd(); rs.moveNext())
+                                {
+
+                                    var record = rs.getRecord();
+                                    var json = new JObject();
+                                    foreach (var field in _arguments.Fields)
+                                    {
+                                        object v = record.getValue(field.Name);
+                                        if (field.Name == "Data")
+                                            v = ToPrintable(v.ToString());
+                                        json.Add(new JProperty(field.Name, v));
+                                    }
+
+                                    var lrn = (Int64)record.getValueEx("RecordNumber");
+                                    logFileMaxRecords[fileName] = lrn;
+
+                                    record = null;
+                                    ProcessJson(json);
+                                    _receivedMessages++;
+                                    json = null;
+
+                                }
+                                // Close the recordset
+                                rs.close();
+                                rs = null;
+                                GC.Collect();
+                            }
+                            if (!Stop)
+                                syncHandle.Wait(TimeSpan.FromSeconds(_pollingIntervalInSeconds), CancelToken);
+                        }
+                        catch (OperationCanceledException oce)
+                        {
+                            break;
+                        }
+                        catch (Exception ex)
+                        {
+                            LogManager.GetCurrentClassLogger().Error(ex);
                         }
                     }
-                    // Close the recordset
-                    rs.close();
-                    firstQuery = false;
                 }
-                catch (Exception ex)
-                {
-                    LogManager.GetCurrentClassLogger().Error("WindowsEventListener", ex);
-                    firstQuery = true;
-                    oLogQuery = new LogQuery();
-                }
-                System.Threading.Thread.Sleep(_pollingIntervalInSeconds * 1000);
+                Finished();
             }
-
-            Finished();
         }
     }
 }

TimberWinR/Manager.cs

@@ -1,5 +1,7 @@
 using System.IO;
 using System.Net.Sockets;
+using System.Reflection;
+using Newtonsoft.Json;
 using NLog;
 using NLog.Config;
 using NLog.Targets;
@@ -10,6 +12,7 @@ using System.Text;
 using TimberWinR.Inputs;
 using TimberWinR.Outputs;
 using System.Threading;
+using Newtonsoft.Json.Linq;
 
 namespace TimberWinR
 {
@@ -21,12 +24,16 @@ namespace TimberWinR
         public Configuration Config { get; set; }
         public List<OutputSender> Outputs { get; set; }
         public List<TcpInputListener> Tcps { get; set; }
-        public List<InputListener> Listeners { get; set;  }
+        public List<TcpInputListener> Udps { get; set; }
+        public List<InputListener> Listeners { get; set; }
+        public bool LiveMonitor { get; set; }
+
         public DateTime StartedOn { get; set; }
         public string JsonConfig { get; set; }
         public string LogfileDir { get; set; }
 
-        public int NumConnections {
+        public int NumConnections
+        {
             get { return numConnections; }
         }
 
@@ -38,34 +45,47 @@ namespace TimberWinR
         private static int numConnections;
         private static int numMessages;
 
-     
+
         public void Shutdown()
         {
             LogManager.GetCurrentClassLogger().Info("Shutting Down");
 
             foreach (InputListener listener in Listeners)
                 listener.Shutdown();
+
+            LogManager.GetCurrentClassLogger().Info("Completed ShutDown");
         }
 
 
         public void IncrementMessageCount(int count = 1)
-        {            
+        {
             Interlocked.Add(ref numMessages, count);
         }
-    
-        public Manager(string jsonConfigFile, string logLevel, string logfileDir, CancellationToken cancelToken)
-        {
-            StartedOn = DateTime.UtcNow;
 
-            JsonConfig = jsonConfigFile;
+        public Manager()
+        {
+            LogsFileDatabase.Manager = this;                   
+        }
+
+        public Manager(string jsonConfigFile, string logLevel, string logfileDir, bool liveMonitor, CancellationToken cancelToken)
+        {
+            LogsFileDatabase.Manager = this;           
+  
+            StartedOn = DateTime.UtcNow;
+            LiveMonitor = liveMonitor;
+
+            var vfi = new FileInfo(jsonConfigFile);
+
+            JsonConfig = vfi.FullName;
             LogfileDir = logfileDir;
 
+
             numMessages = 0;
             numConnections = 0;
 
-            Outputs = new List<OutputSender>();           
+            Outputs = new List<OutputSender>();
             Listeners = new List<InputListener>();
-           
+
             var loggingConfiguration = new LoggingConfiguration();
 
             // Create our default targets
@@ -80,107 +100,182 @@ namespace TimberWinR
             loggingConfiguration.LoggingRules.Add(new LoggingRule("*", LogLevel.Trace, coloredConsoleTarget));
             // LogLevel.Debug means has to be at least Debug to show up in logfile
             loggingConfiguration.LoggingRules.Add(new LoggingRule("*", LogLevel.Debug, fileTarget));
-          
+
             LogManager.Configuration = loggingConfiguration;
             LogManager.EnableLogging();
 
             LogManager.GlobalThreshold = LogLevel.FromString(logLevel);
 
+            LogManager.GetCurrentClassLogger()
+                .Info("TimberWinR Version {0}", GetAssemblyByName("TimberWinR.ServiceHost").GetName().Version.ToString());
 
-            // Is it a directory?
-            if (Directory.Exists(jsonConfigFile))
+
+            LogManager.GetCurrentClassLogger()
+                .Info("Database Directory: {0}", LogsFileDatabase.Instance.DatabaseFileName);
+
+            try
             {
-                DirectoryInfo di = new DirectoryInfo(jsonConfigFile);
-                LogManager.GetCurrentClassLogger().Info("Initialized, Reading Configurations From {0}", di.FullName);
-                Config = Configuration.FromDirectory(jsonConfigFile);              
+                // Is it a directory?
+                if (Directory.Exists(jsonConfigFile))
+                {
+                    DirectoryInfo di = new DirectoryInfo(jsonConfigFile);
+                    LogManager.GetCurrentClassLogger().Info("Initialized, Reading Configurations From {0}", di.FullName);
+                    Config = Configuration.FromDirectory(jsonConfigFile, cancelToken, this);
+                }
+                else
+                {
+                    var fi = new FileInfo(jsonConfigFile);
+
+                    LogManager.GetCurrentClassLogger().Info("Initialized, Reading Configurations From File: {0}", fi.FullName);
+
+                    if (!fi.Exists)
+                        throw new FileNotFoundException("Missing config file", jsonConfigFile);
+
+                    LogManager.GetCurrentClassLogger().Info("Initialized, Reading Config: {0}", fi.FullName);
+                    Config = Configuration.FromFile(jsonConfigFile);
+                }
             }
-            else
+            catch (JsonSerializationException jse)
             {
-                var fi = new FileInfo(jsonConfigFile);
-
-                LogManager.GetCurrentClassLogger().Info("Initialized, Reading Configurations From File: {0}", fi.FullName);
-               
-                if (!fi.Exists)
-                    throw new FileNotFoundException("Missing config file", jsonConfigFile);
-
-                LogManager.GetCurrentClassLogger().Info("Initialized, Reading Config: {0}", fi.FullName);
-                Config = Configuration.FromFile(jsonConfigFile);
+                LogManager.GetCurrentClassLogger().Error(jse);
+            }
+            catch (Exception ex)
+            {
+                LogManager.GetCurrentClassLogger().Error(ex);               
             }
-         
             LogManager.GetCurrentClassLogger().Info("Log Directory {0}", logfileDir);
-            LogManager.GetCurrentClassLogger().Info("Logging Level: {0}", LogManager.GlobalThreshold);        
-            
-            // Read the Configuration file
-           
-            if (Config.RedisOutputs != null)
-            {
-                foreach (var ro in Config.RedisOutputs)
-                {
-                    var redis = new RedisOutput(this, ro, cancelToken);
-                    Outputs.Add(redis);
-                }
-              
-            }
-            if (Config.ElasticsearchOutputs != null)
-            {
-                foreach (var ro in Config.ElasticsearchOutputs)
-                {
-                    var els = new ElasticsearchOutput(this, ro, cancelToken);
-                    Outputs.Add(els);
-                }
-            }
-            if (Config.StdoutOutputs != null)
-            {
-                foreach (var ro in Config.StdoutOutputs)
-                {
-                    var stdout = new StdoutOutput(this, ro, cancelToken);
-                    Outputs.Add(stdout);
-                }
-            }
-
-            foreach (Parser.IISW3CLog iisw3cConfig in Config.IISW3C)
-            {
-                var elistner = new IISW3CInputListener(iisw3cConfig, cancelToken);
-                Listeners.Add(elistner);
-                foreach(var output in Outputs)
-                    output.Connect(elistner);
-            }
-
-            foreach (Parser.WindowsEvent eventConfig in Config.Events)
-            {
-                var elistner = new WindowsEvtInputListener(eventConfig, cancelToken);
-                Listeners.Add(elistner);
-                foreach (var output in Outputs)
-                    output.Connect(elistner);
-            }
-
-            foreach (var logConfig in Config.Logs)
-            {
-                var elistner = new LogsListener(logConfig, cancelToken);
-                Listeners.Add(elistner);
-                foreach (var output in Outputs)
-                    output.Connect(elistner);
-            }
-
-            foreach (var tcp in Config.Tcps)
-            {
-                var elistner = new TcpInputListener(cancelToken, tcp.Port);
-                Listeners.Add(elistner);                
-                foreach (var output in Outputs)
-                    output.Connect(elistner);
-            }
-
-
-            foreach (var tcp in Config.Stdins)
-            {
-                var elistner = new StdinListener(cancelToken);
-                Listeners.Add(elistner);
-                foreach (var output in Outputs)
-                    output.Connect(elistner);
-            }
+            LogManager.GetCurrentClassLogger().Info("Logging Level: {0}", LogManager.GlobalThreshold);
 
+            ProcessConfiguration(cancelToken, Config);
         }
 
+        public void ProcessConfiguration(CancellationToken cancelToken, Configuration config)
+        {
+// Read the Configuration file
+            if (config != null)
+            {
+                if (config.RedisOutputs != null)
+                {
+                    foreach (var ro in config.RedisOutputs)
+                    {
+                        var redis = new RedisOutput(this, ro, cancelToken);
+                        Outputs.Add(redis);
+                    }
+                }
+                if (config.ElasticsearchOutputs != null)
+                {
+                    foreach (var ro in config.ElasticsearchOutputs)
+                    {
+                        var els = new ElasticsearchOutput(this, ro, cancelToken);
+                        Outputs.Add(els);
+                    }
+                }
+                if (config.StdoutOutputs != null)
+                {
+                    foreach (var ro in config.StdoutOutputs)
+                    {
+                        var stdout = new StdoutOutput(this, ro, cancelToken);
+                        Outputs.Add(stdout);
+                    }
+                }
+
+                foreach (Parser.IISW3CLog iisw3cConfig in config.IISW3C)
+                {
+                    var elistner = new IISW3CInputListener(iisw3cConfig, cancelToken);
+                    Listeners.Add(elistner);
+                    foreach (var output in Outputs)
+                        output.Connect(elistner);
+                }
+
+                foreach (Parser.W3CLog iisw3cConfig in config.W3C)
+                {
+                    var elistner = new W3CInputListener(iisw3cConfig, cancelToken);
+                    Listeners.Add(elistner);
+                    foreach (var output in Outputs)
+                        output.Connect(elistner);
+                }
+
+                foreach (Parser.WindowsEvent eventConfig in config.Events)
+                {
+                    var elistner = new WindowsEvtInputListener(eventConfig, cancelToken);
+                    Listeners.Add(elistner);
+                    foreach (var output in Outputs)
+                        output.Connect(elistner);
+                }
+
+                foreach (var logConfig in config.Logs)
+                {
+                    var elistner = new LogsListener(logConfig, cancelToken);
+                    Listeners.Add(elistner);
+                    foreach (var output in Outputs)
+                        output.Connect(elistner);
+                }
+
+                foreach (var logConfig in config.TailFiles)
+                {
+                    var elistner = new TailFileListener(logConfig, cancelToken);
+                    Listeners.Add(elistner);
+                    foreach (var output in Outputs)
+                        output.Connect(elistner);
+                }
+
+                foreach (var tcp in config.Tcps)
+                {
+                    var elistner = new TcpInputListener(cancelToken, tcp.Port);
+                    Listeners.Add(elistner);
+                    foreach (var output in Outputs)
+                        output.Connect(elistner);
+                }
+
+                foreach (var udp in config.Udps)
+                {
+                    var elistner = new UdpInputListener(cancelToken, udp.Port);
+                    Listeners.Add(elistner);
+                    foreach (var output in Outputs)
+                        output.Connect(elistner);
+                }
+
+                foreach (var stdin in config.Stdins)
+                {
+                    var elistner = new StdinListener(stdin, cancelToken);
+                    Listeners.Add(elistner);
+                    foreach (var output in Outputs)
+                        output.Connect(elistner);
+                }
+
+                var computerName = System.Environment.MachineName + "." +
+                                   Microsoft.Win32.Registry.LocalMachine.OpenSubKey(
+                                       @"SYSTEM\CurrentControlSet\services\Tcpip\Parameters")
+                                       .GetValue("Domain", "")
+                                       .ToString();
+
+                foreach (var output in Outputs)
+                {
+                    var name = Assembly.GetExecutingAssembly().GetName();
+                    JObject json = new JObject(
+                        new JProperty("TimberWinR",
+                            new JObject(
+                                new JProperty("version",
+                                    GetAssemblyByName("TimberWinR.ServiceHost").GetName().Version.ToString()),
+                                new JProperty("host", computerName),
+                                new JProperty("output", output.Name),
+                                new JProperty("initialized", DateTime.UtcNow)
+                                )));
+                    json.Add(new JProperty("type", "Win32-TimberWinR"));
+                    json.Add(new JProperty("host", computerName));
+                    output.Startup(json);
+                }
+            }
+        }
+
+
+        private Assembly GetAssemblyByName(string name)
+        {
+            return AppDomain.CurrentDomain.GetAssemblies().
+                   SingleOrDefault(assembly => assembly.GetName().Name == name);
+        }
+
+
         /// <summary>
         /// Creates the default <see cref="FileTarget"/>.
         /// </summary>

TimberWinR/Outputs/Elasticsearch.cs

@@ -2,15 +2,17 @@
 using System.Collections.Generic;
 using System.Linq;
 using System.Net;
-using System.Text;
 using System.Threading;
 using System.Threading.Tasks;
 using Newtonsoft.Json.Linq;
 using NLog;
+using RapidRegex.Core;
 using RestSharp;
 
 namespace TimberWinR.Outputs
 {
+    using System.Text.RegularExpressions;
+
     public partial class ElasticsearchOutput : OutputSender
     {
         private TimberWinR.Manager _manager;
@@ -18,7 +20,6 @@ namespace TimberWinR.Outputs
         private readonly int _interval;
         private readonly string[] _host;
         private readonly string _protocol;
-        private readonly string _index;
         private int _hostIndex;
         private readonly int _timeout;
         private readonly object _locker = new object();
@@ -26,28 +27,29 @@ namespace TimberWinR.Outputs
         private readonly int _numThreads;
         private long _sentMessages;
         private long _errorCount;
+        private Parser.ElasticsearchOutput eo;
+        public bool Stop { get; set; }
 
         public ElasticsearchOutput(TimberWinR.Manager manager, Parser.ElasticsearchOutput eo, CancellationToken cancelToken)
-            : base(cancelToken)
+            : base(cancelToken, "Elasticsearch")
         {
             _sentMessages = 0;
             _errorCount = 0;
 
+            this.eo = eo;
             _protocol = eo.Protocol;
             _timeout = eo.Timeout;
             _manager = manager;
             _port = eo.Port;
             _interval = eo.Interval;
             _host = eo.Host;
-            _index = eo.Index;
             _hostIndex = 0;
             _jsonQueue = new List<JObject>();
             _numThreads = eo.NumThreads;
 
             for (int i = 0; i < eo.NumThreads; i++)
             {
-                var elsThread = new Task(ElasticsearchSender, cancelToken);
-                elsThread.Start();
+                Task.Factory.StartNew(ElasticsearchSender, cancelToken, TaskCreationOptions.LongRunning, TaskScheduler.Current);
             }
         }
 
@@ -57,12 +59,12 @@ namespace TimberWinR.Outputs
                 new JProperty("elasticsearch",
                     new JObject(
                         new JProperty("host", string.Join(",", _host)),
-                        new JProperty("errors", _errorCount),                       
+                        new JProperty("errors", _errorCount),
                         new JProperty("sent_messages", _sentMessages),
                         new JProperty("queued_messages", _jsonQueue.Count),
                         new JProperty("port", _port),
                         new JProperty("interval", _interval),
-                        new JProperty("threads", _numThreads),                                             
+                        new JProperty("threads", _numThreads),
                         new JProperty("hosts",
                             new JArray(
                                 from h in _host
@@ -75,94 +77,113 @@ namespace TimberWinR.Outputs
         // 
         private void ElasticsearchSender()
         {
-            while (!CancelToken.IsCancellationRequested)
+            using (var syncHandle = new ManualResetEventSlim())
             {
-                JObject[] messages;
-                lock (_locker)
+                // Execute the query
+                while (!Stop)
                 {
-                    messages = _jsonQueue.Take(1).ToArray();
-                    _jsonQueue.RemoveRange(0, messages.Length);
-                    if (messages.Length > 0)
-                        _manager.IncrementMessageCount(messages.Length);
-                }
-
-                if (messages.Length > 0)
-                {
-                    int numHosts = _host.Length;
-                    while (numHosts-- > 0)
+                    if (!CancelToken.IsCancellationRequested)
                     {
                         try
                         {
-                            // Get the next client
-                            RestClient client = getClient();                           
-                            if (client != null)
-                            {                              
-                                LogManager.GetCurrentClassLogger()
-                                    .Debug("Sending {0} Messages to {1}", messages.Length, client.BaseUrl);
+                            JObject[] messages;
+                            lock (_locker)
+                            {
+                                var count = _jsonQueue.Count;
+                                messages = _jsonQueue.Take(count).ToArray();
+                                _jsonQueue.RemoveRange(0, count);
+                                if (messages.Length > 0)
+                                    _manager.IncrementMessageCount(messages.Length);
+                            }
 
-                                foreach (JObject json in messages)
+                            if (messages.Length > 0)
+                            {
+                                int numHosts = _host.Length;
+                                while (numHosts-- > 0)
                                 {
-                                    string typeName = "Win32-Elasticsearch";
-                                    if (json["type"] != null)
-                                        typeName = json["type"].ToString();
-                                    string indexName = _index;
-                                    if (string.IsNullOrEmpty(indexName))
-                                    {
-                                        DateTime now = DateTime.UtcNow;
-                                        indexName = string.Format("logstash-{0}", DateTime.UtcNow.ToString("yyyy.MM.dd"));
-                                    }
-                                    var req = new RestRequest(string.Format("/{0}/{1}/", indexName, typeName), Method.POST);                                  
-
-                                    req.AddParameter("text/json", json.ToString(), ParameterType.RequestBody);
-
-                                    req.RequestFormat = DataFormat.Json;
-
                                     try
                                     {
-                                        client.ExecuteAsync(req, response =>
+                                        // Get the next client
+                                        RestClient client = getClient();
+                                        if (client != null)
                                         {
-                                            if (response.StatusCode != HttpStatusCode.Created)
+                                            LogManager.GetCurrentClassLogger()
+                                                .Debug("Sending {0} Messages to {1}", messages.Length, client.BaseUrl);
+
+                                            foreach (JObject json in messages)
                                             {
-                                                LogManager.GetCurrentClassLogger()
-                                                    .Error("Failed to send: {0}", response.ErrorMessage);
-                                                Interlocked.Increment(ref _errorCount);
+                                                var typeName = this.eo.GetTypeName(json);
+                                                var indexName = this.eo.GetIndexName(json);
+                                                var req =
+                                                    new RestRequest(string.Format("/{0}/{1}/", indexName, typeName),
+                                                        Method.POST);
+
+                                                req.AddParameter("text/json", json.ToString(), ParameterType.RequestBody);
+
+                                                req.RequestFormat = DataFormat.Json;
+
+                                                try
+                                                {
+                                                    client.ExecuteAsync(req, response =>
+                                                    {
+                                                        if (response.StatusCode != HttpStatusCode.Created)
+                                                        {
+                                                            LogManager.GetCurrentClassLogger()
+                                                                .Error("Failed to send: {0}", response.ErrorMessage);
+                                                            Interlocked.Increment(ref _errorCount);
+                                                        }
+                                                        else
+                                                        {
+                                                            _sentMessages++;
+                                                            GC.Collect();
+                                                        }
+                                                    });
+                                                }
+                                                catch (Exception error)
+                                                {
+                                                    LogManager.GetCurrentClassLogger().Error(error);
+                                                    Interlocked.Increment(ref _errorCount);
+                                                }
                                             }
-                                            else
-                                            {
-                                                _sentMessages++;
-                                            }
-                                        });
+                                            GC.Collect();
+                                        }
+                                        else
+                                        {
+                                            LogManager.GetCurrentClassLogger()
+                                                .Fatal("Unable to connect with any Elasticsearch hosts, {0}",
+                                                    String.Join(",", _host));
+                                            Interlocked.Increment(ref _errorCount);
+                                        }
+
                                     }
-                                    catch (Exception error)
+                                    catch (Exception ex)
                                     {
-                                        LogManager.GetCurrentClassLogger().Error(error);
+                                        LogManager.GetCurrentClassLogger().Error(ex);
                                         Interlocked.Increment(ref _errorCount);
                                     }
                                 }
                             }
-                            else
-                            {
-                                LogManager.GetCurrentClassLogger()
-                                    .Fatal("Unable to connect with any Elasticsearch hosts, {0}",
-                                        String.Join(",", _host));
-                                Interlocked.Increment(ref _errorCount);
+                            GC.Collect();
+                            if (!Stop)
+                            {                              
+                                syncHandle.Wait(TimeSpan.FromMilliseconds(_interval), CancelToken);  
                             }
-                          
-                        }
-                        catch (Exception ex)
+                          }
+                        catch (OperationCanceledException oce)
                         {
-                            LogManager.GetCurrentClassLogger().Error(ex);
-                            Interlocked.Increment(ref _errorCount);
+                            break;
+                        }
+                        catch (Exception)
+                        {
+                            throw;
                         }
                     }
                 }
-                System.Threading.Thread.Sleep(_interval);
             }
         }
-
         private RestClient getClient()
-         {
-             if (_hostIndex >= _host.Length)
+        {
+            if (_hostIndex >= _host.Length)
                 _hostIndex = 0;
 
             int numTries = 0;
@@ -170,9 +191,9 @@ namespace TimberWinR.Outputs
             {
                 try
                 {
-                    string url = string.Format("{0}://{1}:{2}", _protocol.Replace(":",""), _host[_hostIndex], _port);
+                    string url = string.Format("{0}://{1}:{2}", _protocol.Replace(":", ""), _host[_hostIndex], _port);
                     var client = new RestClient(url);
-                    client.Timeout = _timeout;                  
+                    client.Timeout = _timeout;
 
                     _hostIndex++;
                     if (_hostIndex >= _host.Length)

TimberWinR/Outputs/OutputSender.cs

@@ -12,10 +12,12 @@ namespace TimberWinR.Outputs
     {
         public CancellationToken CancelToken { get; private set; }        
         private List<InputListener> _inputs;
-
-        public OutputSender(CancellationToken cancelToken)
+        public string Name { get; set; }
+      
+        public OutputSender(CancellationToken cancelToken, string name)
         {
             CancelToken = cancelToken;
+            Name = name;
             _inputs = new List<InputListener>();
         }
 
@@ -24,6 +26,11 @@ namespace TimberWinR.Outputs
             listener.OnMessageRecieved += MessageReceivedHandler;            
         }
 
+        public void Startup(JObject json)
+        {
+            MessageReceivedHandler(json);
+        }
+
         public abstract JObject ToJson();
         protected abstract void MessageReceivedHandler(JObject jsonMessage);
     }

TimberWinR/Outputs/Redis.cs

@@ -1,5 +1,6 @@
 using System;
 using System.Collections.Generic;
+using System.Diagnostics.Eventing.Reader;
 using System.Linq;
 using System.Linq.Expressions;
 using System.Net.Sockets;
@@ -13,6 +14,7 @@ using System.Threading.Tasks;
 using RapidRegex.Core;
 using System.Text.RegularExpressions;
 using System.Globalization;
+using TimberWinR.Parser;
 
 namespace TimberWinR.Outputs
 {
@@ -22,21 +24,29 @@ namespace TimberWinR.Outputs
         private readonly int _port;
         private readonly int _timeout;
         private readonly object _locker = new object();
-        private readonly List<string> _jsonQueue;
-        // readonly Task _consumerTask;
+        private readonly List<string> _jsonQueue;       
         private readonly string[] _redisHosts;
         private int _redisHostIndex;
         private TimberWinR.Manager _manager;
         private readonly int _batchCount;
+        private int _currentBatchCount;
+        private readonly int _maxBatchCount;       
         private readonly int _interval;
         private readonly int _numThreads;
-
+        private readonly long[] _sampleQueueDepths;
+        private int _sampleCountIndex;
         private long _sentMessages;
         private long _errorCount;
         private long _redisDepth;
+        private DateTime? _lastErrorTime;
+        private const int QUEUE_SAMPLE_SIZE = 30; // 30 samples over 2.5 minutes (default)
+        private readonly int _maxQueueSize;
+        private readonly bool _queueOverflowDiscardOldest;
+
+        public bool Stop { get; set; }
 
         /// <summary>
-        /// Get the next client
+        /// Get the next client from the list of hosts.
         /// </summary>
         /// <returns></returns>
         private RedisClient getClient()
@@ -50,16 +60,17 @@ namespace TimberWinR.Outputs
                 try
                 {
                     RedisClient client = new RedisClient(_redisHosts[_redisHostIndex], _port, _timeout);
-
-                    _redisHostIndex++;
-                    if (_redisHostIndex >= _redisHosts.Length)
-                        _redisHostIndex = 0;
-
                     return client;
                 }
                 catch (Exception)
                 {
                 }
+                finally
+                {
+                    _redisHostIndex++;
+                    if (_redisHostIndex >= _redisHosts.Length)
+                        _redisHostIndex = 0;
+                }
                 numTries++;
             }
 
@@ -73,13 +84,20 @@ namespace TimberWinR.Outputs
                     new JObject(
                         new JProperty("host", string.Join(",", _redisHosts)),
                         new JProperty("errors", _errorCount),
+                        new JProperty("lastErrorTime", _lastErrorTime),
                         new JProperty("redis_depth", _redisDepth),
                         new JProperty("sent_messages", _sentMessages),
                         new JProperty("queued_messages", _jsonQueue.Count),
                         new JProperty("port", _port),
+                        new JProperty("maxQueueSize", _maxQueueSize),
+                        new JProperty("overflowDiscardOldest", _queueOverflowDiscardOldest),
                         new JProperty("interval", _interval),
                         new JProperty("threads", _numThreads),
                         new JProperty("batchcount", _batchCount),
+                        new JProperty("currentBatchCount", _currentBatchCount),
+                        new JProperty("maxBatchCount", _maxBatchCount),
+                        new JProperty("averageQueueDepth", AverageQueueDepth()),   
+                        new JProperty("queueSamples", new JArray(_sampleQueueDepths)),
                         new JProperty("index", _logstashIndexName),
                         new JProperty("hosts",
                             new JArray(
@@ -90,13 +108,21 @@ namespace TimberWinR.Outputs
         }
 
         public RedisOutput(TimberWinR.Manager manager, Parser.RedisOutput ro, CancellationToken cancelToken)
-            : base(cancelToken)
+            : base(cancelToken, "Redis")
         {
+            // Last QUEUE_SAMPLE_SIZE queue samples (spans timestamp * 10)
+            _sampleQueueDepths = new long[QUEUE_SAMPLE_SIZE];
+            _sampleCountIndex = 0;
             _redisDepth = 0;
             _batchCount = ro.BatchCount;
+            _maxBatchCount = ro.MaxBatchCount;
+            // Make sure maxBatchCount is larger than batchCount
+            if (_maxBatchCount < _batchCount)
+                _maxBatchCount = _batchCount*10;
+            _currentBatchCount = _batchCount;
             _manager = manager;
             _redisHostIndex = 0;
-            _redisHosts = ro.Host;
+            _redisHosts = ro.Host;          
             _jsonQueue = new List<string>();
             _port = ro.Port;
             _timeout = ro.Timeout;
@@ -104,6 +130,9 @@ namespace TimberWinR.Outputs
             _interval = ro.Interval;
             _numThreads = ro.NumThreads;
             _errorCount = 0;
+            _lastErrorTime = null;
+            _maxQueueSize = ro.MaxQueueSize;
+            _queueOverflowDiscardOldest = ro.QueueOverflowDiscardOldest;
 
             for (int i = 0; i < ro.NumThreads; i++)
             {
@@ -114,7 +143,7 @@ namespace TimberWinR.Outputs
 
         public override string ToString()
         {
-            return string.Format("Redis Host: {0} Port: {1}, Threads: {2}, Interval: {3}, BatchCount: {4}", string.Join(",", _redisHosts) , _port, _numThreads, _interval, _batchCount);
+            return string.Format("Redis Host: {0} Port: {1}, Threads: {2}, Interval: {3}, BatchCount: {4}", string.Join(",", _redisHosts), _port, _numThreads, _interval, _batchCount);
         }
 
         /// <summary>
@@ -124,22 +153,64 @@ namespace TimberWinR.Outputs
         protected override void MessageReceivedHandler(JObject jsonMessage)
         {
             if (_manager.Config.Filters != null)
-                ApplyFilters(jsonMessage);
+            {
+                if (ApplyFilters(jsonMessage))
+                    return;
+            }
 
             var message = jsonMessage.ToString();
             LogManager.GetCurrentClassLogger().Debug(message);
 
             lock (_locker)
             {
+                if (_jsonQueue.Count >= _maxQueueSize)
+                {
+                    // If we've exceeded our queue size, and we're supposed to throw out the oldest objects first,
+                    // then remove as many as necessary to get us under our limit
+                    if (_queueOverflowDiscardOldest)
+                    {
+                        LogManager.GetCurrentClassLogger()
+                            .Warn("Overflow discarding oldest {0} messages", _jsonQueue.Count - _maxQueueSize + 1);
+
+                        _jsonQueue.RemoveRange(0, (_jsonQueue.Count - _maxQueueSize) + 1);
+                    }
+                    // Otherwise we're in a "discard newest" mode, and this is the newest message, so just ignore it
+                    else
+                    {
+                        LogManager.GetCurrentClassLogger()
+                            .Warn("Overflow discarding newest message: {0}", message);
+
+                        return;
+                    }
+                }
+
                 _jsonQueue.Add(message);
             }
         }
 
-        private void ApplyFilters(JObject json)
+        private bool ApplyFilters(JObject json)
         {
+            bool drop = false;
             foreach (var filter in _manager.Config.Filters)
             {
-                filter.Apply(json);
+                if (!filter.Apply(json))
+                {  
+                    LogManager.GetCurrentClassLogger().Debug("Dropping: {0}", json.ToString());
+                    drop = true;
+                }              
+            }
+            return drop;
+        }
+
+
+        // 
+        // What is average queue depth?
+        //
+        private int AverageQueueDepth()
+        {
+            lock(_locker)
+            {
+                return (int)_sampleQueueDepths.Average();
             }
         }
 
@@ -148,66 +219,126 @@ namespace TimberWinR.Outputs
         // 
         private void RedisSender()
         {
-            while (!CancelToken.IsCancellationRequested)
+            using (var syncHandle = new ManualResetEventSlim())
             {
-                string[] messages;
-                lock (_locker)
+                // Execute the query
+                while (!Stop)
                 {
-                    messages = _jsonQueue.Take(_batchCount).ToArray();
-                    _jsonQueue.RemoveRange(0, messages.Length);
-                    if (messages.Length > 0)
-                        _manager.IncrementMessageCount(messages.Length);
-                }
-
-                if (messages.Length > 0)
-                {
-                    int numHosts = _redisHosts.Length;
-                    while (numHosts-- > 0)
+                    if (!CancelToken.IsCancellationRequested)
                     {
                         try
                         {
-                            // Get the next client
-                            using (RedisClient client = getClient())
-                            {
-                                if (client != null)
-                                {
-                                    client.StartPipe();
-                                    LogManager.GetCurrentClassLogger()
-                                               .Debug("Sending {0} Messages to {1}", messages.Length, client.Host);
+                            string[] messages;
+                            // Exclusively
+                            lock (_locker)
+                            {           
+                                // Take a sample of the queue depth
+                                if (_sampleCountIndex >= QUEUE_SAMPLE_SIZE)
+                                    _sampleCountIndex = 0;     
+                                _sampleQueueDepths[_sampleCountIndex++] = _jsonQueue.Count;                                                                                    
+                                messages = _jsonQueue.Take(_currentBatchCount).ToArray();
+                                _jsonQueue.RemoveRange(0, messages.Length);
+                                var remainingCount = _jsonQueue.Count;
+                                if (messages.Length > 0)
+                                    _manager.IncrementMessageCount(messages.Length);
+                            }
 
-                                    foreach (string jsonMessage in messages)
+                            if (messages.Length > 0)
+                            {
+                                int numHosts = _redisHosts.Length;
+                                bool sentSuccessfully = false;
+                                while (numHosts-- > 0)
+                                {
+                                    try
                                     {
-                                        try
+                                        // Get the next client
+                                        using (RedisClient client = getClient())
                                         {
-                                           _redisDepth = client.RPush(_logstashIndexName, jsonMessage);
-                                           _sentMessages++;
-                                        }
-                                        catch (SocketException ex)
-                                        {
-                                            LogManager.GetCurrentClassLogger().Warn(ex);
-                                            Interlocked.Increment(ref _errorCount);
+                                            if (client != null)
+                                            {
+                                                client.StartPipe();
+                                                LogManager.GetCurrentClassLogger()
+                                                    .Debug("Sending {0} Messages to {1}", messages.Length, client.Host);
+
+                                                try
+                                                {
+                                                    _redisDepth = client.RPush(_logstashIndexName, messages);
+                                                    _sentMessages += messages.Length;                                                  
+                                                    client.EndPipe();
+                                                    sentSuccessfully = true;
+                                                }
+                                                catch (SocketException ex)
+                                                {
+                                                    LogManager.GetCurrentClassLogger().Warn(ex);
+                                                    Interlocked.Increment(ref _errorCount);
+                                                    _lastErrorTime = DateTime.UtcNow;
+                                                }
+                                                catch (Exception ex)
+                                                {
+                                                    LogManager.GetCurrentClassLogger().Error(ex);
+                                                    Interlocked.Increment(ref _errorCount);
+                                                    _lastErrorTime = DateTime.UtcNow;
+                                                }                                               
+                                                break;
+                                            }
+                                            else
+                                            {
+                                                Interlocked.Increment(ref _errorCount);
+                                                LogManager.GetCurrentClassLogger()
+                                                    .Fatal("Unable to connect with any Redis hosts, {0}",
+                                                        String.Join(",", _redisHosts));
+                                                _lastErrorTime = DateTime.UtcNow;
+                                            }
                                         }
                                     }
-                                    client.EndPipe();
-                                    break;
-                                }
-                                else
+                                    catch (Exception ex)
+                                    {
+                                        LogManager.GetCurrentClassLogger().Error(ex);
+                                        Interlocked.Increment(ref _errorCount);
+                                        _lastErrorTime = DateTime.UtcNow;                                
+                                    }
+                                } // No more hosts to try.
+
+                                // Re-compute current batch size
+                                updateCurrentBatchCount();
+
+                                if (!sentSuccessfully)                                
                                 {
-                                    Interlocked.Increment(ref _errorCount);
-                                    LogManager.GetCurrentClassLogger()
-                                        .Fatal("Unable to connect with any Redis hosts, {0}",
-                                            String.Join(",", _redisHosts));
+                                    lock (_locker)
+                                    {
+                                        _jsonQueue.InsertRange(0, messages);
+                                    }
                                 }
                             }
+                            GC.Collect();
+                            if (!Stop)
+                                syncHandle.Wait(TimeSpan.FromMilliseconds(_interval), CancelToken);
+                        }
+                        catch (OperationCanceledException oce)
+                        {
+                            break;
                         }
                         catch (Exception ex)
                         {
-                            LogManager.GetCurrentClassLogger().Error(ex);
+                            _lastErrorTime = DateTime.UtcNow;
                             Interlocked.Increment(ref _errorCount);
+                            LogManager.GetCurrentClassLogger().Error(ex);                         
                         }
                     }
                 }
-                System.Threading.Thread.Sleep(_interval);
+            }
+        }
+
+        // Sample the queue and adjust the batch count if needed (ramp up slowly)
+        private void updateCurrentBatchCount()
+        {
+            if (_currentBatchCount < _maxBatchCount && AverageQueueDepth() > _currentBatchCount)
+            {
+                _currentBatchCount += _maxBatchCount/QUEUE_SAMPLE_SIZE;
+            }
+            else // Reset to default
+            {
+                _currentBatchCount = _batchCount;
             }
         }
     }

TimberWinR/Outputs/Stdout.cs

@@ -15,9 +15,10 @@ namespace TimberWinR.Outputs
         private readonly object _locker = new object();
         private readonly List<JObject> _jsonQueue;
         private long _sentMessages;
+        public bool Stop { get; set; }
 
         public StdoutOutput(TimberWinR.Manager manager, Parser.StdoutOutput eo, CancellationToken cancelToken)
-            : base(cancelToken)
+            : base(cancelToken, "Stdout")
         {
             _sentMessages = 0;
             _manager = manager;
@@ -34,7 +35,7 @@ namespace TimberWinR.Outputs
                 new JProperty("stdout",
                     new JObject(
                         new JProperty("sent_messages", _sentMessages))));
-                      
+
             return json;
         }
 
@@ -43,38 +44,59 @@ namespace TimberWinR.Outputs
         // 
         private void StdoutSender()
         {
-            while (!CancelToken.IsCancellationRequested)
+            using (var syncHandle = new ManualResetEventSlim())
             {
-                JObject[] messages;
-                lock (_locker)
+                // Execute the query
+                while (!Stop)
                 {
-                    messages = _jsonQueue.Take(1).ToArray();
-                    _jsonQueue.RemoveRange(0, messages.Length);                  
-                }
-
-                if (messages.Length > 0)
-                {
-                    try
+                    if (!CancelToken.IsCancellationRequested)
                     {
-                        foreach (JObject obj in messages)
+                        try
+                        {
+                            JObject[] messages;
+                            lock (_locker)
+                            {
+                                messages = _jsonQueue.Take(_jsonQueue.Count).ToArray();
+                                _jsonQueue.RemoveRange(0, messages.Length);
+                            }
+
+                            if (messages.Length > 0)
+                            {
+                                try
+                                {
+                                    foreach (JObject obj in messages)
+                                    {
+                                        Console.WriteLine(obj.ToString());
+                                        _sentMessages++;
+                                    }
+                                }
+                                catch (Exception ex)
+                                {
+                                    LogManager.GetCurrentClassLogger().Error(ex);
+                                }
+                            }
+                            if (!Stop)
+                                syncHandle.Wait(TimeSpan.FromMilliseconds(_interval), CancelToken);
+                        }
+                        catch (OperationCanceledException oce)
+                        {
+                            break;
+                        }
+                        catch (Exception)
                         {
-                            Console.WriteLine(obj.ToString());
-                            _sentMessages++;
                         }
                     }
-                    catch (Exception ex)
-                    {
-                        LogManager.GetCurrentClassLogger().Error(ex);
-                    }
                 }
-                System.Threading.Thread.Sleep(_interval);
             }
         }
 
         protected override void MessageReceivedHandler(Newtonsoft.Json.Linq.JObject jsonMessage)
         {
             if (_manager.Config.Filters != null)
-                ApplyFilters(jsonMessage);
+            {
+                if (ApplyFilters(jsonMessage))
+                    return;
+            }
 
             var message = jsonMessage.ToString();
             LogManager.GetCurrentClassLogger().Debug(message);
@@ -85,12 +107,17 @@ namespace TimberWinR.Outputs
             }
         }
 
-        private void ApplyFilters(JObject json)
+        private bool ApplyFilters(JObject json)
         {
+            bool drop = false;
+
             foreach (var filter in _manager.Config.Filters)
             {
-                filter.Apply(json);
+                if (!filter.Apply(json))
+                    drop = true;
             }
+
+            return drop;
         }
 
     }

TimberWinR/Parser.cs

@@ -2,6 +2,8 @@
 using System.Collections.Generic;
 using System.Configuration;
 using System.Linq;
+using System.Linq.Expressions;
+using System.Linq.Dynamic;
 using System.Reflection;
 using System.Runtime.Remoting.Channels;
 using System.Text;
@@ -11,89 +13,59 @@ using Newtonsoft.Json;
 using Newtonsoft.Json.Linq;
 using NLog;
 using TimberWinR.Outputs;
-
+using System.CodeDom.Compiler;
 
 namespace TimberWinR.Parser
 {
+    using System.Text.RegularExpressions;
+
     interface IValidateSchema
     {
         void Validate();
     }
 
+
     public abstract class LogstashFilter : IValidateSchema
     {
         public abstract bool Apply(JObject json);
-       
+
+        protected void RemoveProperty(JObject json, string name)
+        {
+            JToken token = json[name];
+            if (token != null)
+            {
+                json.Remove(name);
+            }
+        }
+
         protected void RenameProperty(JObject json, string oldName, string newName)
         {
             JToken token = json[oldName];
             if (token != null)
             {
                 json.Remove(oldName);
-                json.Add(newName, token);
+                JToken newToken = json[newName];
+                if (newToken == null)
+                    json.Add(newName, token);
             }
         }
 
+        public abstract JObject ToJson();
+
         protected bool EvaluateCondition(JObject json, string condition)
         {
-            // Create a new instance of the C# compiler
             var cond = condition;
 
-            IList<string> keys = json.Properties().Select(p => p.Name).ToList();
+            IList<string> keys = json.Properties().Select(pn => pn.Name).ToList();
             foreach (string key in keys)
-                cond = cond.Replace(string.Format("[{0}]", key), string.Format("\"{0}\"", json[key].ToString()));
+                cond = cond.Replace(string.Format("[{0}]", key), string.Format("{0}", json[key].ToString()));
 
-            var compiler = new CSharpCodeProvider();
+            var p = Expression.Parameter(typeof(JObject), "");
+            var e = System.Linq.Dynamic.DynamicExpression.ParseLambda(new[] { p }, null, cond);
 
-            // Create some parameters for the compiler
-            var parms = new System.CodeDom.Compiler.CompilerParameters
-            {
-                GenerateExecutable = false,
-                GenerateInMemory = true
-            };
-            parms.ReferencedAssemblies.Add("System.dll");
-            parms.ReferencedAssemblies.Add("System.Core.dll");
-            parms.ReferencedAssemblies.Add("Newtonsoft.Json.dll");
+            var result = e.Compile().DynamicInvoke(json);
 
-            var code = string.Format(@" using System;
-                                        using System.Linq;
-                                        using Newtonsoft.Json;
-                                        using Newtonsoft.Json.Linq;
-
-                                        class EvaluatorClass
-                                        {{
-                                            public JObject json {{ get; set; }}
-                                            public bool Evaluate()
-                                            {{
-                                                return {0};
-                                            }}               
-                                        }}", cond);
-
-            // Try to compile the string into an assembly
-            var results = compiler.CompileAssemblyFromSource(parms, new string[] { code });
-
-            // If there weren't any errors get an instance of "MyClass" and invoke
-            // the "Message" method on it
-            if (results.Errors.Count == 0)
-            {
-                var evClass = results.CompiledAssembly.CreateInstance("EvaluatorClass");
-                evClass.GetType().GetProperty("json").SetValue(evClass, json, null);
-
-                var result = evClass.GetType().
-                    GetMethod("Evaluate").
-                    Invoke(evClass, null);
-                return bool.Parse(result.ToString());
-            }
-            else
-            {
-                foreach (var e in results.Errors)
-                {
-                    LogManager.GetCurrentClassLogger().Error(e);
-                    LogManager.GetCurrentClassLogger().Error("Bad Code: {0}", code);
-                }
-            }
-
-            return false;
+            return (bool)result;
         }
         protected void RemoveProperties(JToken token, string[] fields)
         {
@@ -101,6 +73,7 @@ namespace TimberWinR.Parser
             if (container == null) return;
 
             List<JToken> removeList = new List<JToken>();
+
             foreach (JToken el in container.Children())
             {
                 JProperty p = el as JProperty;
@@ -115,6 +88,7 @@ namespace TimberWinR.Parser
             {
                 el.Remove();
             }
+
         }
 
         protected void ReplaceProperty(JObject json, string propertyName, string propertyValue)
@@ -151,7 +125,7 @@ namespace TimberWinR.Parser
         }
 
         public abstract void Validate();
-   
+
     }
 
     [JsonObject(MemberSerialization.OptIn)]
@@ -200,7 +174,7 @@ namespace TimberWinR.Parser
             To = to;
         }
     }
-    
+
     public class WindowsEvent : IValidateSchema
     {
         public enum FormatKinds
@@ -241,16 +215,19 @@ namespace TimberWinR.Parser
         public List<Field> Fields { get; set; }
         [JsonProperty(PropertyName = "formatMsg")]
         public bool FormatMsg { get; set; }
+        [JsonProperty(PropertyName = "interval")]
+        public int Interval { get; set; }
 
         public WindowsEvent()
         {
+            Interval = 60; // Every minute
             Source = "System";
             StringsSep = "|";
             FormatMsg = true;
             FullText = true;
             BinaryFormat = FormatKinds.ASC;
             FullEventCode = false;
-          
+
             Fields = new List<Field>();
             Fields.Add(new Field("EventLog", "string"));
             Fields.Add(new Field("RecordNumber", "int"));
@@ -266,7 +243,7 @@ namespace TimberWinR.Parser
             Fields.Add(new Field("ComputerName", "string"));
             Fields.Add(new Field("SID", "string"));
             Fields.Add(new Field("Message", "string"));
-            Fields.Add(new Field("Data", "string"));              
+            Fields.Add(new Field("Data", "string"));
         }
 
         public void Validate()
@@ -274,12 +251,79 @@ namespace TimberWinR.Parser
 
         }
     }
-   
+
     public class Stdin : IValidateSchema
     {
+        [JsonProperty(PropertyName = "codec")]
+        public Codec Codec { get; set; }
+
         public void Validate()
         {
-           
+
+        }
+    }
+
+    public class Codec
+    {
+        public enum CodecType
+        {
+            singleline,
+            multiline
+        };
+
+        public enum WhatType
+        {
+            previous,
+            next
+        };
+
+        [JsonProperty(PropertyName = "type")]
+        public CodecType Type { get; set; }
+        [JsonProperty(PropertyName = "pattern")]
+        public string Pattern { get; set; }
+        [JsonProperty(PropertyName = "what")]
+        public WhatType What { get; set; }
+        [JsonProperty(PropertyName = "negate")]
+        public bool Negate { get; set; }
+        [JsonProperty(PropertyName = "multiline_tag")]
+        public string MultilineTag { get; set; }
+
+        public Regex Re { get; set; }
+
+        public Codec()
+        {
+            Negate = false;
+            MultilineTag = "multiline";
+        }
+    }
+
+    public class TailFile : IValidateSchema
+    {
+        [JsonProperty(PropertyName = "location")]
+        public string Location { get; set; }       
+        [JsonProperty(PropertyName = "recurse")]
+        public int Recurse { get; set; }        
+        [JsonProperty(PropertyName = "fields")]
+        public List<Field> Fields { get; set; }
+        [JsonProperty(PropertyName = "interval")]
+        public int Interval { get; set; }
+        [JsonProperty(PropertyName = "logSource")]
+        public string LogSource { get; set; }
+        [JsonProperty(PropertyName = "codec")]
+        public Codec Codec { get; set; }
+
+        public TailFile()
+        {
+            Fields = new List<Field>();
+            Fields.Add(new Field("LogFilename", "string"));
+            Fields.Add(new Field("Index", "integer"));
+            Fields.Add(new Field("Text", "string"));
+            Interval = 30;
+        }
+
+        public void Validate()
+        {
+
         }
     }
 
@@ -295,6 +339,12 @@ namespace TimberWinR.Parser
         public bool SplitLongLines { get; set; }
         [JsonProperty(PropertyName = "fields")]
         public List<Field> Fields { get; set; }
+        [JsonProperty(PropertyName = "interval")]
+        public int Interval { get; set; }
+        [JsonProperty(PropertyName = "logSource")]
+        public string LogSource { get; set; }
+        [JsonProperty(PropertyName = "codec")]
+        public Codec Codec { get; set; }
 
         public Log()
         {
@@ -302,11 +352,12 @@ namespace TimberWinR.Parser
             Fields.Add(new Field("LogFilename", "string"));
             Fields.Add(new Field("Index", "integer"));
             Fields.Add(new Field("Text", "string"));
+            Interval = 30;
         }
 
         public void Validate()
         {
-            
+
         }
     }
 
@@ -322,12 +373,63 @@ namespace TimberWinR.Parser
 
         public void Validate()
         {
-            
+
         }
     }
-    
+
+
+    public class Udp : IValidateSchema
+    {
+        [JsonProperty(PropertyName = "port")]
+        public int Port { get; set; }
+
+        public Udp()
+        {
+            Port = 5142;
+        }
+
+        public void Validate()
+        {
+
+        }
+    }
+    public class W3CLog : IValidateSchema
+    {
+        [JsonProperty(PropertyName = "location")]
+        public string Location { get; set; }
+        [JsonProperty(PropertyName = "separator")]
+        public string Separator { get; set; }
+        [JsonProperty(PropertyName = "iCodepage")]
+        public int CodePage { get; set; }
+        [JsonProperty(PropertyName = "dtLines")]
+        public int DtLines { get; set; }
+        [JsonProperty(PropertyName = "dQuotes")]
+        public bool DoubleQuotes { get; set; }
+
+
+        [JsonProperty(PropertyName = "fields")]
+        public List<Field> Fields { get; set; }
+
+        public W3CLog()
+        {
+            CodePage = 0;
+            DtLines = 10;
+            Fields = new List<Field>();
+            Separator = "auto";
+
+            Fields.Add(new Field("LogFilename", "string"));
+            Fields.Add(new Field("RowNumber", "integer"));
+        }
+
+        public void Validate()
+        {
+
+        }
+    }
+
+
     public class IISW3CLog : IValidateSchema
-    {       
+    {
         [JsonProperty(PropertyName = "location")]
         public string Location { get; set; }
         [JsonProperty(PropertyName = "iCodepage")]
@@ -353,47 +455,49 @@ namespace TimberWinR.Parser
             Fields = new List<Field>();
 
             Fields.Add(new Field("LogFilename", "string"));
-            Fields.Add(new Field("LogRow", "integer" ));
-            Fields.Add(new Field("date", "DateTime" ));
-            Fields.Add(new Field("time", "DateTime" ));
-            Fields.Add(new Field("c-ip", "string" ));
-            Fields.Add(new Field("cs-username", "string" ));
-            Fields.Add(new Field("s-sitename", "string" ));
-            Fields.Add(new Field("s-computername", "integer" ));
-            Fields.Add(new Field("s-ip", "string" ));
-            Fields.Add(new Field("s-port", "integer" ));
-            Fields.Add(new Field("cs-method", "string" ));
-            Fields.Add(new Field("cs-uri-stem", "string" ));
-            Fields.Add(new Field("cs-uri-query", "string" ));
-            Fields.Add(new Field("sc-status", "integer" ));
-            Fields.Add(new Field("sc-substatus", "integer" ));
-            Fields.Add(new Field("sc-win32-status", "integer" ));
-            Fields.Add(new Field("sc-bytes", "integer" ));
-            Fields.Add(new Field("cs-bytes", "integer" ));
-            Fields.Add(new Field("time-taken", "integer" ));
-            Fields.Add(new Field("cs-version", "string" ));
-            Fields.Add(new Field("cs-host", "string" ));
-            Fields.Add(new Field("cs(User-Agent)", "string" ));
-            Fields.Add(new Field("cs(Cookie)", "string" ));
-            Fields.Add(new Field("cs(Referer)", "string" ));
-            Fields.Add(new Field("s-event", "string" ));
-            Fields.Add(new Field("s-process-type", "string" ));
-            Fields.Add(new Field("s-user-time", "double" ));
-            Fields.Add(new Field("s-kernel-time", "double" ));
-            Fields.Add(new Field("s-page-faults", "integer" ));
-            Fields.Add(new Field("s-total-procs", "integer" ));
-            Fields.Add(new Field("s-active-procs", "integer" ));
+            Fields.Add(new Field("LogRow", "integer"));
+            Fields.Add(new Field("date", "DateTime"));
+            Fields.Add(new Field("time", "DateTime"));
+            Fields.Add(new Field("c-ip", "string"));
+            Fields.Add(new Field("cs-username", "string"));
+            Fields.Add(new Field("s-sitename", "string"));
+            Fields.Add(new Field("s-computername", "integer"));
+            Fields.Add(new Field("s-ip", "string"));
+            Fields.Add(new Field("s-port", "integer"));
+            Fields.Add(new Field("cs-method", "string"));
+            Fields.Add(new Field("cs-uri-stem", "string"));
+            Fields.Add(new Field("cs-uri-query", "string"));
+            Fields.Add(new Field("sc-status", "integer"));
+            Fields.Add(new Field("sc-substatus", "integer"));
+            Fields.Add(new Field("sc-win32-status", "integer"));
+            Fields.Add(new Field("sc-bytes", "integer"));
+            Fields.Add(new Field("cs-bytes", "integer"));
+            Fields.Add(new Field("time-taken", "integer"));
+            Fields.Add(new Field("cs-version", "string"));
+            Fields.Add(new Field("cs-host", "string"));
+            Fields.Add(new Field("cs(User-Agent)", "string"));
+            Fields.Add(new Field("cs(Cookie)", "string"));
+            Fields.Add(new Field("cs(Referer)", "string"));
+            Fields.Add(new Field("s-event", "string"));
+            Fields.Add(new Field("s-process-type", "string"));
+            Fields.Add(new Field("s-user-time", "double"));
+            Fields.Add(new Field("s-kernel-time", "double"));
+            Fields.Add(new Field("s-page-faults", "integer"));
+            Fields.Add(new Field("s-total-procs", "integer"));
+            Fields.Add(new Field("s-active-procs", "integer"));
             Fields.Add(new Field("s-stopped-procs", "integer"));
         }
 
         public void Validate()
         {
-          
+
         }
     }
 
     public class ElasticsearchOutput
     {
+        const string IndexDatePattern = "(%\\{(?<format>[^\\}]+)\\})";
+
         [JsonProperty(PropertyName = "host")]
         public string[] Host { get; set; }
         [JsonProperty(PropertyName = "index")]
@@ -408,21 +512,60 @@ namespace TimberWinR.Parser
         public string Protocol { get; set; }
         [JsonProperty(PropertyName = "interval")]
         public int Interval { get; set; }
-        
+
         public ElasticsearchOutput()
         {
             Protocol = "http";
             Port = 9200;
             Index = "";
             Host = new string[] { "localhost" };
-            Timeout = 10000;          
+            Timeout = 10000;
             NumThreads = 1;
             Interval = 1000;
         }
+
+        public string GetIndexName(JObject json)
+        {
+            ////check if the submitted JSON object provides a custom index. If yes, use this one
+            var token = json["_index"];
+            var indexName = token == null ? this.Index : token.Value<string>();
+
+            if (string.IsNullOrEmpty(indexName))
+            {
+                indexName = string.Format("logstash-{0}", DateTime.UtcNow.ToString("yyyy.MM.dd"));
+            }
+            else
+            {
+                var date = DateTime.UtcNow;
+                if (json["@timestamp"] != null)
+                {
+                    date = DateTime.Parse(json["@timestamp"].ToString());
+                }
+
+                var match = Regex.Match(indexName, IndexDatePattern);
+                if (match.Success)
+                {
+                    indexName = Regex.Replace(indexName, IndexDatePattern, date.ToString(match.Groups["format"].Value));
+                }
+            }
+
+            return indexName;
+        }
+
+        public string GetTypeName(JObject json)
+        {
+            string typeName = "Win32-Elasticsearch";
+            if (json["type"] != null)
+            {
+                typeName = json["type"].ToString();
+            }
+            return typeName;
+        }
+
     }
 
     public class RedisOutput
-    {      
+    {
         [JsonProperty(PropertyName = "host")]
         public string[] Host { get; set; }
         [JsonProperty(PropertyName = "index")]
@@ -433,20 +576,29 @@ namespace TimberWinR.Parser
         public int Timeout { get; set; }
         [JsonProperty(PropertyName = "batch_count")]
         public int BatchCount { get; set; }
+        [JsonProperty(PropertyName = "max_batch_count")]
+        public int MaxBatchCount { get; set; }
         [JsonProperty(PropertyName = "threads")]
         public int NumThreads { get; set; }
         [JsonProperty(PropertyName = "interval")]
         public int Interval { get; set; }
+        [JsonProperty(PropertyName = "max_queue_size")]
+        public int MaxQueueSize { get; set; }
+        [JsonProperty(PropertyName = "queue_overflow_discard_oldest")]
+        public bool QueueOverflowDiscardOldest { get; set; }
 
         public RedisOutput()
         {
             Port = 6379;
             Index = "logstash";
-            Host = new string[] {"localhost"};
+            Host = new string[] { "localhost" };
             Timeout = 10000;
             BatchCount = 10;
+            MaxBatchCount = BatchCount*10;
             NumThreads = 1;
             Interval = 5000;
+            QueueOverflowDiscardOldest = true;
+            MaxQueueSize = 50000;
         }
     }
 
@@ -481,22 +633,31 @@ namespace TimberWinR.Parser
         [JsonProperty("Logs")]
         public Log[] Logs { get; set; }
 
+        [JsonProperty("TailFiles")]
+        public TailFile[] TailFiles { get; set; }
+
         [JsonProperty("Tcp")]
         public Tcp[] Tcps { get; set; }
 
+        [JsonProperty("Udp")]
+        public Udp[] Udps { get; set; }
+
         [JsonProperty("IISW3CLogs")]
         public IISW3CLog[] IISW3CLogs { get; set; }
 
+        [JsonProperty("W3CLogs")]
+        public W3CLog[] W3CLogs { get; set; }
+
         [JsonProperty("Stdin")]
         public Stdin[] Stdins { get; set; }
     }
-         
+
     public partial class Grok : LogstashFilter, IValidateSchema
     {
         public class GrokFilterException : Exception
         {
             public GrokFilterException()
-                : base("Grok filter missing required match, must be 2 array entries.")                   
+                : base("Grok filter missing required match, must be 2 array entries.")
             {
             }
         }
@@ -508,10 +669,14 @@ namespace TimberWinR.Parser
             {
             }
         }
+
+        [JsonProperty("type")]
+        public string Type { get; set; }
+
         [JsonProperty("condition")]
         public string Condition { get; set; }
 
-        [JsonProperty("drop_if_match")]
+        [JsonProperty("drop")]
         public bool DropIfMatch { get; set; }
 
         [JsonProperty("match")]
@@ -521,10 +686,10 @@ namespace TimberWinR.Parser
         public string[] AddTag { get; set; }
 
         [JsonProperty("add_field")]
-        public string[] AddField { get; set; }        
-         
+        public string[] AddField { get; set; }
+
         [JsonProperty("remove_field")]
-        public string[] RemoveField { get; set; }    
+        public string[] RemoveField { get; set; }
 
         [JsonProperty("remove_tag")]
         public string[] RemoveTag { get; set; }
@@ -534,7 +699,7 @@ namespace TimberWinR.Parser
             if (Match == null || Match.Length != 2)
                 throw new GrokFilterException();
 
-            if (AddTag != null && AddTag.Length%2 != 0)
+            if (AddTag != null && AddTag.Length % 2 != 0)
                 throw new GrokAddTagException();
         }
     }
@@ -557,6 +722,9 @@ namespace TimberWinR.Parser
             }
         }
 
+        [JsonProperty("type")]
+        public string Type { get; set; }
+
         [JsonProperty("condition")]
         public string Condition { get; set; }
 
@@ -573,7 +741,7 @@ namespace TimberWinR.Parser
         public bool ConvertToUTC { get; set; }
 
         [JsonProperty("add_field")]
-        public string[] AddField { get; set; }           
+        public string[] AddField { get; set; }
 
         public override void Validate()
         {
@@ -593,9 +761,15 @@ namespace TimberWinR.Parser
 
     public partial class Mutate : LogstashFilter
     {
+        [JsonProperty("type")]
+        public string Type { get; set; }
+
         [JsonProperty("condition")]
         public string Condition { get; set; }
 
+        [JsonProperty("remove")]
+        public string[] Remove { get; set; }
+
         [JsonProperty("rename")]
         public string[] Rename { get; set; }
 
@@ -607,10 +781,127 @@ namespace TimberWinR.Parser
 
         public override void Validate()
         {
-           
+
         }
     }
-    
+
+    public partial class GeoIP : LogstashFilter
+    {
+        public class GeoIPMissingSourceException : Exception
+        {
+            public GeoIPMissingSourceException()
+                : base("GeoIP filter source is required")
+            {
+            }
+        }
+
+        public class GeoIPAddFieldException : Exception
+        {
+            public GeoIPAddFieldException()
+                : base("GeoIP filter add_field requires tuples")
+            {
+            }
+        }
+
+        [JsonProperty("type")]
+        public string Type { get; set; }
+
+        [JsonProperty("condition")]
+        public string Condition { get; set; }
+
+        [JsonProperty("source")]
+        public string Source { get; set; }
+
+        [JsonProperty("target")]
+        public string Target { get; set; }
+
+        [JsonProperty("add_tag")]
+        public string[] AddTag { get; set; }
+
+        [JsonProperty("add_field")]
+        public string[] AddField { get; set; }
+
+        [JsonProperty("remove_field")]
+        public string[] RemoveField { get; set; }
+
+        [JsonProperty("remove_tag")]
+        public string[] RemoveTag { get; set; }
+
+        public override void Validate()
+        {
+            if (string.IsNullOrEmpty(Source))
+                throw new GeoIPMissingSourceException();
+
+            if (AddField != null && AddField.Length % 2 != 0)
+                throw new GeoIPAddFieldException();
+        }
+    }
+
+   
+    public partial class Json : LogstashFilter
+    {
+        public class JsonMissingSourceException : Exception
+        {
+            public JsonMissingSourceException()
+                : base("JSON filter source is required")
+            {
+            }
+        }
+
+        public class JsonAddFieldException : Exception
+        {
+            public JsonAddFieldException()
+                : base("JSON filter add_field requires tuples")
+            {
+            }
+        }
+
+
+        [JsonProperty("type")]
+        public string Type { get; set; }
+
+        [JsonProperty("condition")]
+        public string Condition { get; set; }
+
+        [JsonProperty("source")]
+        public string Source { get; set; }
+
+        [JsonProperty("remove_source")]
+        public bool RemoveSource { get; set; }
+
+        [JsonProperty("target")]
+        public string Target { get; set; }
+
+        [JsonProperty("add_tag")]
+        public string[] AddTag { get; set; }
+
+        [JsonProperty("add_field")]
+        public string[] AddField { get; set; }
+
+        [JsonProperty("remove_field")]
+        public string[] RemoveField { get; set; }
+
+        [JsonProperty("remove_tag")]
+        public string[] RemoveTag { get; set; }
+
+        [JsonProperty("rename")]
+        public string[] Rename { get; set; }
+
+        [JsonProperty("promote")]
+        public string Promote { get; set; }
+
+
+        public override void Validate()
+        {
+            if (string.IsNullOrEmpty(Source))
+                throw new JsonMissingSourceException();
+
+            if (AddField != null && AddField.Length % 2 != 0)
+                throw new JsonAddFieldException();
+        }
+    }
+
+
     public class Filter
     {
         [JsonProperty("grok")]
@@ -621,8 +912,29 @@ namespace TimberWinR.Parser
 
         [JsonProperty("date")]
         public DateFilter Date { get; set; }
+
+        [JsonProperty("json")]
+        public Json Json { get; set; }
+
+        [JsonProperty("geoip")]
+        public GeoIP GeoIP { get; set; }
+
+        [JsonProperty("grokFilters")]
+        public Grok[] Groks { get; set; }
+      
+        [JsonProperty("mutateFilters")]
+        public Mutate[] Mutates { get; set; }
+
+        [JsonProperty("dateFilters")]
+        public DateFilter[] Dates { get; set; }
+
+        [JsonProperty("jsonFilters")]
+        public Json[] Jsons { get; set; }
+
+        [JsonProperty("geoipFilters")]
+        public GeoIP[] GeoIPs { get; set; }    
     }
-   
+
     public class TimberWinR
     {
         [JsonProperty("Inputs")]
@@ -630,13 +942,14 @@ namespace TimberWinR.Parser
         [JsonProperty("Filters")]
         public List<Filter> Filters { get; set; }
         [JsonProperty("Outputs")]
-        public OutputTargets Outputs { get; set;  }
+        public OutputTargets Outputs { get; set; }
 
         public LogstashFilter[] AllFilters
         {
             get
             {
                 var list = new List<LogstashFilter>();
+
                 foreach (var filter in Filters)
                 {
                     foreach (var prop in filter.GetType().GetProperties())
@@ -646,13 +959,18 @@ namespace TimberWinR.Parser
                         {
                             list.Add(typedFilter as LogstashFilter);
                         }
+                        else if (typedFilter != null && typedFilter.GetType().IsArray && typeof(LogstashFilter).IsAssignableFrom(typedFilter.GetType().GetElementType()))
+                        {
+                            IEnumerable<LogstashFilter> lf = typedFilter as IEnumerable<LogstashFilter>;
+                            list.AddRange(lf);
+                        }
                     }
                 }
                 return list.ToArray();
             }
         }
     }
-   
+
     public class RootObject
     {
         public TimberWinR TimberWinR { get; set; }

TimberWinR/Properties/AssemblyInfo.cs

@@ -32,5 +32,5 @@ using System.Runtime.InteropServices;
 // You can specify all the values or you can default the Build and Revision Numbers 
 // by using the '*' as shown below:
 // [assembly: AssemblyVersion("1.0.*")]
-[assembly: AssemblyVersion("1.0.0.0")]
-[assembly: AssemblyFileVersion("1.0.0.0")]
+[assembly: AssemblyVersion("1.2.0.0")]
+[assembly: AssemblyFileVersion("1.2.0.0")]

TimberWinR/ReleaseNotes.md

@@ -0,0 +1,76 @@
+TimberWinR Release Notes
+==================================
+A Native Windows to Redis/Elasticsearch Logstash Agent which runs as a service.
+
+Version History
+### 1.3.20.0 - 03/03/2015
+1. Added new Redis parameter _max\_batch\_count_ which increases the _batch\_count_ dynamically over time 
+   to handle input flooding.   Default is _batch\_count_ * 10 
+
+### 1.3.19.0 - 02/26/2015
+
+1. Added support for Multiline codecs for Stdin and Logs listeners, closes issue [#23](https://github.com/Cimpress-MCP/TimberWinR/issues/23)
+2. Added new TailFiles input type which uses a native implementation (more-efficient) than using LogParser's Log
+3. Updated Udp input listner to use UTF8 Encoding rather than ASCII
+4. Reduced noisy complaint about missing log files for Logs listener
+5. Fixed bug when tailing non-existent log files which resulted in high cpu-usage.
+6. Added feature to watch the configuration directory
+
+### 1.3.18.0 - 12/22/2014
+
+1. Fixed bug introduced in 1.3.17.0 which changed the meaning of the delay for Elasticsearch, Redis and Stdout 
+intervals to be interpreted as seconds instead of milliseconds.   1.3.17.0 should not be used.
+2. Removed ability for installer to downgrade which was leading to leaving previous versions laying around (i.e. reverts 1.3.13.0 change)
+
+### 1.3.17.0 - 12/19/2014
+
+1. Continued work improving shutdown time by using syncHandle.Wait instead of Thread.Sleep
+
+### 1.3.16.0 - 12/19/2014
+
+1. Added logSource property to the Log input to facility the steering of log messages to different indices.
+
+### 1.3.15.0 - 12/12/2014
+
+1. Fixed bug whereby if the Udp or Tcp inputs receive an impropery formatted Json it caused the thread to terminate, and ignore
+future messages.
+
+### 1.3.14.0 - 12/11/2014
+
+1. Fixed bug with the Grok filter to match properly the value of the Text field against non-blank entries.
+
+### 1.3.13.0 - 12/02/2014
+
+1. Fixed MSI installer to allow downgrades.
+
+### 1.3.12.0 - 11/25/2014
+
+1. Fixed all remaining memory leaks due to the COM Weak Surrogate which requires an explicit GC.Collect
+
+### 1.3.11.0 - 11/21/2014
+
+1. Re-worked WindowsEvent listener to enable shutting down in a quicker fashion.
+
+### 1.3.10.0 - 11/18/2014
+
+1. Refactored Conditions handler to use non-leaking evaluator.
+
+### 1.3.9.0 - 11/11/2014
+
+1. Merged in pull request #9
+2. Updated chocolately uninstall to preserve GUID
+
+### 1.3.8.0 - 11/06/2014
+1. Added interval parameter to WindowsEvent input listener
+2. Increased default value for interval to 60 seconds for polling WindowsEvents
+
+### 1.3.7.0 - 10/21/2014
+1. Added additional information for diagnostics port
+2. Completed minor handling of Log rolling detection
+
+### 1.3.6.0 - 10/16/2014
+1. Handle rolling of logs whereby the logfile remains the same, but the content resets back to 0 bytes.
+
+
+
+

TimberWinR/TimberWinR.csproj

@@ -39,6 +39,14 @@
       <EmbedInteropTypes>False</EmbedInteropTypes>
       <HintPath>lib\com-logparser\Interop.MSUtil.dll</HintPath>
     </Reference>
+    <Reference Include="MaxMind.Db">
+      <HintPath>..\packages\MaxMind.Db.0.2.3.0\lib\net40\MaxMind.Db.dll</HintPath>
+      <Private>True</Private>
+    </Reference>
+    <Reference Include="MaxMind.GeoIP2">
+      <HintPath>..\packages\MaxMind.GeoIP2.0.4.0.0\lib\net40\MaxMind.GeoIP2.dll</HintPath>
+      <Private>True</Private>
+    </Reference>
     <Reference Include="Newtonsoft.Json, Version=6.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed, processorArchitecture=MSIL">
       <SpecificVersion>False</SpecificVersion>
       <HintPath>..\packages\Newtonsoft.Json.6.0.4\lib\net40\Newtonsoft.Json.dll</HintPath>
@@ -54,6 +62,9 @@
     </Reference>
     <Reference Include="System" />
     <Reference Include="System.Core" />
+    <Reference Include="System.Linq.Dynamic">
+      <HintPath>..\packages\System.Linq.Dynamic.1.0.3\lib\net40\System.Linq.Dynamic.dll</HintPath>
+    </Reference>
     <Reference Include="System.Xml.Linq" />
     <Reference Include="System.Data.DataSetExtensions" />
     <Reference Include="Microsoft.CSharp" />
@@ -71,8 +82,15 @@
     <Compile Include="Filters\DateFilter.cs" />
     <Compile Include="Filters\FilterBase.cs" />
     <Compile Include="Filters\GrokFilter.cs" />
+    <Compile Include="Filters\GeoIPFilter.cs" />
+    <Compile Include="Filters\JsonFilter.cs" />
     <Compile Include="Filters\MutateFilter.cs" />
     <Compile Include="Inputs\FieldDefinitions.cs" />
+    <Compile Include="Inputs\IISW3CRowReader.cs" />
+    <Compile Include="Inputs\LogsFileDatabase.cs" />
+    <Compile Include="Inputs\TailFileListener.cs" />
+    <Compile Include="Inputs\UdpInputListener.cs" />
+    <Compile Include="Inputs\W3CInputListener.cs" />
     <Compile Include="Inputs\IISW3CInputListener.cs" />
     <Compile Include="Inputs\InputBase.cs" />
     <Compile Include="Inputs\InputListener.cs" />
@@ -103,8 +121,17 @@
     <None Include="configSchema.xsd">
       <SubType>Designer</SubType>
     </None>
+    <Content Include="GeoLite2City.mmdb">
+      <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
+    </Content>
+    <None Include="mdocs\Codec.md" />
     <None Include="mdocs\DateFilter.md" />
     <None Include="mdocs\Filters.md" />
+    <None Include="mdocs\GeoIPFilter.md" />
+    <None Include="mdocs\TailFiles.md" />
+    <None Include="mdocs\UdpInput.md" />
+    <None Include="mdocs\W3CInput.md" />
+    <None Include="mdocs\JsonFilter.md" />
     <None Include="mdocs\GrokFilter.md" />
     <None Include="mdocs\ElasticsearchOutput.md" />
     <None Include="mdocs\RedisOutput.md" />
@@ -115,6 +142,7 @@
     <None Include="mdocs\IISW3CInput.md" />
     <None Include="mdocs\WindowsEvents.md" />
     <None Include="packages.config" />
+    <None Include="ReleaseNotes.md" />
   </ItemGroup>
   <ItemGroup>
     <EmbeddedResource Include="Properties\Resources.resx">

TimberWinR/mdocs/Codec.md

@@ -0,0 +1,36 @@
+# Codec
+
+## Parameters
+The following parameters are allowed when configuring the Codec.
+
+| Parameter         |     Type       |  Description                                                             | Details                       |  Default |
+| :---------------- |:---------------| :----------------------------------------------------------------------- | :---------------------------  | :-- |
+| *type*            | enum           |Codec type 'multiline'                                                    | Must be 'multiline'           |     |
+| *pattern*         | regex          |Regular expression to be matched                                          | Must be legal .NET Regex      |     |
+| *what*            | enum           |Value can be previous or next                                             | If the pattern matched, does event belong to the next or previous event? |     |
+| *negate*          | bool           |Inverts the pattern sense                                                 | If true, a message not matching the pattern will constitute a match of the multiline filter and the what will be applied. (vice-versa is also true) |  false   |
+| *multiline_tag*   | string         |Tag to be added when multiline conversion is applied                      |                               |  multiline | 
+
+This codec applies to [Logs](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/Logs.md) and [Stdin](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/StdinInput.md) only.
+
+Example Input: Mutliline input log file
+
+```json
+{
+    "TimberWinR": {
+        "Inputs": {
+            "Logs": [
+            {
+                "location": "C:\\Logs1\\multiline.log",
+                "recurse": -1,
+                "codec": {
+                  "negate": false,
+                  "type": "multiline",
+                  "pattern": "(^.+Exception: .+)|(^\\s+at .+)|(^\\s+... \\d+ more)|(^\\s*Caused by:.+)",
+                  "what": "previous"
+                }
+            }
+		 }
+	}
+}
+```

TimberWinR/mdocs/DateFilter.md

@@ -34,7 +34,7 @@ Given this configuration
   "Filters": [     
     {
 	   "date":  {
-          "condition": "[type] == \"Win32-FileLog\"",
+          "condition": "\"[type]\" == \"Win32-FileLog\"",
           "match": [
               "timestamp",
               "MMM  d HH:mm:sss",
@@ -56,7 +56,7 @@ then the operation(s) will be executed in order.
   "Filters": [     
     {
 		"grok": {      			
-		    "condition": "[type] == \"Win32-EventLog\""
+		    "condition": "\"[type]\" == \"Win32-EventLog\""
 			"add_field": [
 				"ComputerName", "%{Host}"				              
 			]
@@ -72,7 +72,7 @@ The fields must be in pairs with fieldName first and value second.
   "Filters": [     
     {
 	   "date":  {
-          "condition": "[type] == \"Win32-FileLog\"",
+          "condition": "\"[type]\" == \"Win32-FileLog\"",
           "match": [
               "timestamp",
               "MMM  d HH:mm:sss",
@@ -92,7 +92,7 @@ If true and the filter matches, the time parsed will be converted to UTC
   "Filters": [     
     {
 	   "date":  {
-          "condition": "[type] == \"Win32-FileLog\"",
+          "condition": "\"[type]\" == \"Win32-FileLog\"",
           "match": [
               "timestamp",
               "MMM  d HH:mm:sss",

TimberWinR/mdocs/ElasticsearchOutput.md

@@ -13,6 +13,9 @@ The following parameters are allowed when configuring the Redis output.
 | *host*        | [string] | The hostname(s) of your Elasticsearch server(s) | IP or DNS name |  |
 | *port*        | integer  | Redis port number                                           | This port must be open  | 9200  |
 
+### Index parameter
+If you want to output your data everyday to a new index, use following index format: "index-%{yyyy.MM.dd}". Here date format could be any forwat which you need.
+
 Example Input:
 ```json
 {
@@ -23,7 +26,7 @@ Example Input:
                     "threads":  1,   
                     "interval": 5000,                             
                     "host": [
-                        "tstlexiceapp006.vistaprint.svc"
+                        "tstlexiceapp006.mycompany.svc"
                     ]
                 }
             ]

TimberWinR/mdocs/Filters.md

@@ -12,7 +12,7 @@ Example Input:
  "Filters": [          
     {
         "grok": {
-            "condition": "[type] == \"Win32-Eventlog\"",
+            "condition": "\"[type]\" == \"Win32-Eventlog\"",
             "match": [
                 "Message",
                 ""
@@ -35,7 +35,7 @@ Example Input:
     },
     {
         "date":  {
-            "condition": "[type] == \"Win32-FileLog\"",
+            "condition": "\"[type]\" == \"Win32-FileLog\"",
             "match": [
                 "timestamp",
                 "MMM  d HH:mm:sss",
@@ -64,6 +64,6 @@ Example Input:
   [1]: http://logstash.net/docs/1.4.2/filters/grok
   [2]: http://logstash.net/docs/1.4.2/filters/date
   [3]: http://logstash.net/docs/1.4.2/filters/mutate
-  [4]: https://github.com/efontana/TimberWinR/blob/master/mdocs/GrokFilter.md
-  [5]: https://github.com/efontana/TimberWinR/blob/master/mdocs/DateFilter.md
-  [6]: https://github.com/efontana/TimberWinR/blob/master/mdocs/MutateFilter.md
+  [4]: https://github.com/Cimpress-MCP/TimberWinR/blob/master/mdocs/GrokFilter.md
+  [5]: https://github.com/Cimpress-MCP/TimberWinR/blob/master/mdocs/DateFilter.md
+  [6]: https://github.com/Cimpress-MCP/TimberWinR/blob/master/mdocs/MutateFilter.md

TimberWinR/mdocs/GeoIPFilter.md

@@ -0,0 +1,138 @@
+# GeoIP Filter
+The GeoIP filter adds information about the geographical location of IP addresses, based on data from the Maxmind database.
+TimberWinR releases ship with the GeoLiteCity database made available from Maxmind with a CCA-ShareAlike 3.0 license.
+For more details on GeoLite, see http://www.maxmind.com/en/geolite.
+
+## GeoIP Operations
+The following operations are allowed when mutating a field.
+
+| Operation       |     Type        | Description                                                            
+| :---------------|:----------------|:-----------------------------------------------------------------------|
+| *type*          | property:string |Type to which this filter applies, if empty, applies to all types.
+| *condition*     | property:string |C# expression, if the expression is true, continue, otherwise, ignore
+| *source*        | property:string |Required field indicates which field contains the IP address to be parsed
+| *target*        | property:string |If suppled, the parsed json will be contained underneath a propery named *target*, default=geoip
+| *add_field*     | property:array  |If the filter is successful, add an arbitrary field to this event.  Field names can be dynamic and include parts of the event using the %{field} syntax.  This property must be specified in pairs.                                    
+| *remove_field*  | property:array  |If the filter is successful, remove arbitrary fields from this event.  Field names can be dynamic and include parts of the event using the %{field} syntax.                                
+| *add_tag*       | property:array  |If the filter is successful, add an arbitrary tag to this event.  Tag names can be dynamic and include parts of the event using the %{field} syntax.                                  
+| *remove_tag*    | property:array  |If the filter is successful, remove arbitrary tags from this event.  Field names can be dynamic and include parts of the event using the %{field} syntax.                          
+
+## Operation Details
+### source 
+The match field is required, the first argument is the field to inspect, and compare to the expression specified by the second
+argument.  In the below example, the message is spected to be something like this from a fictional sample log:
+
+Given this input configuration:
+
+Lets assume that a newline such as the following is appended to foo.jlog:
+```
+   {"type": "Win32-FileLog", "IP": "8.8.8.8" }
+```
+
+```json
+   "Inputs": {
+            "Logs": [
+                {
+                    "location": "C:\\Logs1\\foo.jlog",
+                    "recurse": -1
+                }
+            ]
+        },
+        "Filters":[  
+            {  
+                "geoip":{  
+                    "type":  "Win32-FileLog",                                      
+                    "source": "IP"              
+                }
+            }]
+        }       
+```
+
+In the above example, the file foo.jlog is being tailed, and when a newline is appended, it is assumed
+to be Json and is parsed from the Text field, the parsed Json is then inserted underneath a property *stuff*
+
+The resulting output would be:
+```
+  {
+      "type": "Win32-FileLog",
+      "IP": "8.8.8.8",
+      "mygeoip": {
+        "ip": "8.8.8.8",
+        "country_code2": "US",
+        "country_name": "United States",
+        "continent_code": "NA",
+        "region_name": "CA",
+        "city_name": "Mountain View",
+        "postal_code": null,
+        "latitude": 37.386,
+        "longitude": -122.0838,
+        "dma_code": 807,
+        "timezone": "America/Los_Angeles",
+        "real_region_name": "California",
+        "location": [
+          -122.0838,
+          37.386
+        ]
+  }
+```
+
+### add_field ["fieldName", "fieldValue", ...]
+The fields must be in pairs with fieldName first and value second.
+```json
+  "Filters": [     
+    {
+		"json": {      			
+			"add_field": [
+              "ComputerName", "Host",
+              "Username", "%{SID}"				         
+			]
+		}                
+    }     
+  ]
+```
+
+### remove_field ["tag1", "tag2", ...]
+Remove the fields.  More than one field can be specified at a time.
+```json
+  "Filters": [     
+    {
+		"json": {      			
+			"remove_tag": [             
+             "static_tag1",
+             "Computer_%{Host}"
+			]
+		}                
+    }     
+  ]
+```
+
+
+### add_tag ["tag1", "tag2", ...]
+Adds the tag(s) to the tag array.
+```json
+  "Filters": [     
+    {
+		"json": {      			
+			"add_tag": [
+               "foo_%{Host}",
+			   "static_tag1"      
+			]
+		}                
+    }     
+  ]
+```
+
+### remove_tag ["tag1", "tag2", ...]
+Remove the tag(s) to the tag array.  More than one tag can be specified at a time.
+```json
+  "Filters": [     
+    {
+		"json": {      			
+			"remove_tag": [             
+             "static_tag1",
+             "Username"
+			]
+		}                
+    }     
+  ]
+```

TimberWinR/mdocs/GrokFilter.md

@@ -26,7 +26,9 @@ The following operations are allowed when mutating a field.
 
 | Operation       |     Type        | Description                                                            
 | :---------------|:----------------|:-----------------------------------------------------------------------|
+| *type*          | property:string |Type to which this filter applyes, if empty, applies to all types.
 | *condition*     | property:string |C# expression
+| *rename*        | property:array  |Rename one or more fields                                       
 | *match*         | property:string |Required field must match before any subsequent grok operations are executed.
 | *add_field*     | property:array  |If the filter is successful, add an arbitrary field to this event.  Field names can be dynamic and include parts of the event using the %{field} syntax.  This property must be specified in pairs.                                    
 | *remove_field*  | property:array  |If the filter is successful, remove arbitrary fields from this event.  Field names can be dynamic and include parts of the event using the %{field} syntax.                                
@@ -82,8 +84,8 @@ then the operation(s) will be executed in order.
 ```json
   "Filters": [     
     {
-		"grok": {      			
-		    "condition": "[type] == \"Win32-EventLog\""
+		"grok": {      
+          "type": "Win32-EventLog",		  
 			"add_field": [
 				"ComputerName", "%{Host}"				              
 			]

TimberWinR/mdocs/JsonFilter.md

@@ -0,0 +1,153 @@
+# Json Filter
+The Json filter allows you to parse a single line of Json into its corresponding fields.  This is
+particularly useful when parsing log files.
+
+## Json Operations
+The following operations are allowed when mutating a field.
+
+| Operation       |     Type        | Description                                                            
+| :---------------|:----------------|:-----------------------------------------------------------------------|
+| *type*          | property:string |Type to which this filter applies, if empty, applies to all types.
+| *condition*     | property:string |C# expression, if the expression is true, continue, otherwise, ignore
+| *remove_source* | property:bool   |If true, the source property is removed, default: true
+| *source*        | property:string |Required field indicates which field contains the Json to be parsed
+| *promote*       | property:string |If supplied any properties named *promote* will be promoted to top-level
+| *target*        | property:string |If suppled, the parsed json will be contained underneath a propery named *target*
+| *add_field*     | property:array  |If the filter is successful, add an arbitrary field to this event.  Field names can be dynamic and include parts of the event using the %{field} syntax.  This property must be specified in pairs.                                    
+| *remove_field*  | property:array  |If the filter is successful, remove arbitrary fields from this event.  Field names can be dynamic and include parts of the event using the %{field} syntax.                                
+| *add_tag*       | property:array  |If the filter is successful, add an arbitrary tag to this event.  Tag names can be dynamic and include parts of the event using the %{field} syntax.                                  
+| *remove_tag*    | property:array  |If the filter is successful, remove arbitrary tags from this event.  Field names can be dynamic and include parts of the event using the %{field} syntax.                          
+
+## Operation Details
+### source 
+The match field is required, the first argument is the field to inspect, and compare to the expression specified by the second
+argument.  In the below example, the message is spected to be something like this from a fictional sample log:
+
+Given this input configuration:
+
+Lets assume that a newline such as the following is appended to foo.jlog:
+```
+   {"Email":"james@example.com","Active":true,"CreatedDate":"2013-01-20T00:00:00Z","Roles":["User","Admin"]}
+```
+
+```json
+   "Inputs": {
+            "Logs": [
+                {
+                    "location": "C:\\Logs1\\foo.jlog",
+                    "recurse": -1
+                }
+            ]
+        },
+        "Filters":[  
+            {  
+                "json":{  
+                    "type":  "Win32-FileLog",                   
+                    "target": "stuff",
+                    "source": "Text",
+                    "rename": [
+                        "Text",
+                        "Data"
+                    ]                   
+                }
+            }]
+        }       
+```
+
+In the above example, the file foo.jlog is being tailed, and when a newline is appended, it is assumed
+to be Json and is parsed from the Text field, the parsed Json is then inserted underneath a property *stuff*
+
+The resulting output would be:
+```
+  {
+    "type": "Win32-FileLog",
+    "ComputerName": "dev.mycompany.net",
+    "Text": "{\"Email\":\"james@example.com\",\"Active\":true,\"CreatedDate\":\"2013-01-20T00:00:00Z\",\"Roles\":[\"User\",\"Admin\"]}",
+    "stuff": {
+      "Email": "james@example.com",
+      "Active": true,
+      "CreatedDate": "2013-01-20T00:00:00Z",
+      "Roles": [
+        "User",
+        "Admin"
+      ]
+    }
+  }
+```
+### rename ["oldname", "newname", ...]
+The fields must be in pairs with oldname first and newname second.
+```json
+  "Filters": [     
+   {
+	  "json":{  
+        "type":  "Win32-FileLog",                   
+        "target": "stuff",
+        "source": "Text",
+        "rename": [
+            "Text",
+            "Data"
+        ]                   
+     } 
+   }     
+  ]
+```
+### add_field ["fieldName", "fieldValue", ...]
+The fields must be in pairs with fieldName first and value second.
+```json
+  "Filters": [     
+    {
+		"json": {      			
+			"add_field": [
+              "ComputerName", "Host",
+              "Username", "%{SID}"				         
+			]
+		}                
+    }     
+  ]
+```
+
+### remove_field ["tag1", "tag2", ...]
+Remove the fields.  More than one field can be specified at a time.
+```json
+  "Filters": [     
+    {
+		"json": {      			
+			"remove_tag": [             
+             "static_tag1",
+             "Computer_%{Host}"
+			]
+		}                
+    }     
+  ]
+```
+
+
+### add_tag ["tag1", "tag2", ...]
+Adds the tag(s) to the tag array.
+```json
+  "Filters": [     
+    {
+		"json": {      			
+			"add_tag": [
+               "foo_%{Host}",
+			   "static_tag1"      
+			]
+		}                
+    }     
+  ]
+```
+
+### remove_tag ["tag1", "tag2", ...]
+Remove the tag(s) to the tag array.  More than one tag can be specified at a time.
+```json
+  "Filters": [     
+    {
+		"json": {      			
+			"remove_tag": [             
+             "static_tag1",
+             "Username"
+			]
+		}                
+    }     
+  ]
+```

TimberWinR/mdocs/Logs.md

@@ -8,9 +8,11 @@ The following parameters are allowed when configuring WindowsEvents.
 | Parameter         |     Type       |  Description                                                             | Details               |  Default |
 | :---------------- |:---------------| :----------------------------------------------------------------------- | :---------------------------  | :-- |
 | *location*        | string  |Location of file(s) to monitor                                           | Path to text file(s) including wildcards. |     |
+| *logSource*       | string  |Source name                                  | Used for conditions |     |
 | *recurse*         | integer |Max subdirectory recursion level.                                       | 0 disables subdirectory recursion; -1 enables unlimited recursion. | 0 |
 | *splitLongLines*  | boolean |Behavior when event messages or event category names cannot be resolved. |When a text line is longer than 128K characters, the format truncates the line and either discards the remaining of the line (when this parameter is set to "false"), or processes the remainder of the line as a new line (when this parameter is set to "true").| false |
 | *iCodepage*       | integer |Codepage of the text file.                                              | 0 is the system codepage, -1 is UNICODE.                         | 0  |
+| [codec](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/Codec.md)  | object | Codec to use  |
 
 Example Input: Monitors all files (recursively) located at C:\Logs1\ matching *.log as a pattern.  I.e. C:\Logs1\foo.log, C:\Logs1\Subdir\Log2.log, etc.
 
@@ -20,6 +22,7 @@ Example Input: Monitors all files (recursively) located at C:\Logs1\ matching *.
         "Inputs": {
             "Logs": [
                 {
+                    "logSource": "log files",
                     "location": "C:\\Logs1\\*.log",
                     "recurse": -1
                 }

TimberWinR/mdocs/MutateFilter.md

@@ -8,6 +8,7 @@ The following operations are allowed when mutating a field.
 | Operation   |     Type        | Description                                                            
 | :-----------|:----------------|:-----------------------------------------------------------------------|
 | *condition* | property:string |C# Expression
+| *remove*    | property:array  |Remove one or more fields                                       
 | *rename*    | property:array  |Rename one or more fields                                       
 | *replace*   | property:array  |Replace a field with a new value.  The new value can include %{foo} strings to help you build a new value from other parts of the event.                                   
 | *split*     | property:array  |Separator between values of the "Strings" field.   
@@ -18,18 +19,31 @@ If present, the condition must evaluate to true in order for the remaining opera
 then the operation(s) will be executed in order.
 ```json
   "Filters": [     
-    {
+	{
 		"mutate": {      			
-		    "condition": "[type] == \"Win32-EventLog\""
+			"condition": "\"[type]\" == \"Win32-EventLog\"",
 			"rename": [
 				"ComputerName", "Host"				              
 			]
 		}                
+	}     
+  ]
+```
+The above example will rename ComputerName to Host only for Win32-EventLog types.
+
+### remove ["name", ...]
+Removes field.
+```json
+  "Filters": [     
+    {
+		"mutate": {      			
+			"remove": [
+				"ComputerName", "Username"
+			]
+		}                
     }     
   ]
 ```
-The above example will rename ComputerName to Host only for Win32-EventLog types.
-
 ### rename ["oldname", "newname", ...]
 The fields must be in pairs with oldname first and newname second.
 ```json

TimberWinR/mdocs/RedisOutput.md

@@ -8,11 +8,14 @@ The following parameters are allowed when configuring the Redis output.
 | Parameter     |   Type   |  Description                                                | Details               |  Default |
 | :-------------|:---------|:------------------------------------------------------------| :---------------------------  | :-- |
 | *threads*     | string   | Location of log files(s) to monitor                         | Number of worker theads to send messages | 1 |
+| *batch_count* | integer  | Sent as a single message                         | Number of messages to aggregate | 10 |
+| *max_batch_count* | integer  | Dynamically adjusted count maximum                      | Increases over time | batch_count*10 |
 | *interval*    | integer  | Interval in milliseconds to sleep during batch sends        | Interval      | 5000 |
-| *batch_count* | integer  | The number of events to send in a single transaction        |  | 10 |
 | *index*       | string   | The name of the redis list                                  | logstash index name | logstash |
 | *host*        | [string] | The hostname(s) of your Redis server(s) | IP or DNS name |  |
 | *port*        | integer  | Redis port number                                           | This port must be open  | 6379  |
+| *max_queue_size* | integer  | Maximum redis queue depth       |  | 50000 |
+| *queue_overflow_discard_oldest* | bool  | If true, discard oldest messages when max_queue_size reached otherwise discard newest |  | true |
 
 Example Input:
 ```json
@@ -25,7 +28,7 @@ Example Input:
                     "interval": 5000, 
                     "batch_count":  500,              
                     "host": [
-                        "tstlexiceapp006.vistaprint.svc"
+                        "tstlexiceapp006.mycompany.svc"
                     ]
                 }
             ]

TimberWinR/mdocs/StdinInput.md

@@ -3,7 +3,12 @@
 The Stdin Input will read from the console (Console.ReadLine) and build a simple message for testing.
 
 ## Parameters
-There are no Parameters at this time.
+The following parameters are allowed when configuring WindowsEvents.
+
+| Parameter         |     Type       |  Description                                                             | Details               |  Default |
+| :---------------- |:---------------| :----------------------------------------------------------------------- | :---------------------------  | :-- |
+| [codec](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/Codec.md)  | object | Codec to use  |
+
 
 ```json
 {
@@ -26,4 +31,3 @@ A field: "type": "Win32-Stdin" is automatically appended, and the entire JSON is
 | ---- |:-----| :-----------------------------------------------------------------------|
 | type | STRING |Win32-Stdin |
 | message | STRING | The message typed in  |
-

TimberWinR/mdocs/TailFiles.md

@@ -0,0 +1,41 @@
+# Input: TailFiles
+
+The TailFiles input will monitor a log (text) file similar to how a Linux "tail -f" command works.  This uses
+a native implementation rather than uses LogParser
+
+## Parameters
+The following parameters are allowed when configuring WindowsEvents.
+
+| Parameter         |     Type       |  Description                                                             | Details               |  Default |
+| :---------------- |:---------------| :----------------------------------------------------------------------- | :---------------------------  | :-- |
+| *location*        | string  |Location of file(s) to monitor                                           | Path to text file(s) including wildcards. |     |
+| *logSource*       | string  |Source name                                  | Used for conditions |     |
+| *recurse*         | integer |Max subdirectory recursion level.                                       | 0 disables subdirectory recursion; -1 enables unlimited recursion. | 0 |
+| *interval*        | integer |Polling interval in seconds                                     | Defaults every 60 seconds | 60 |
+| [codec](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/Codec.md)  | object | Codec to use  |
+
+Example Input: Monitors all files (recursively) located at C:\Logs1\ matching *.log as a pattern.  I.e. C:\Logs1\foo.log, C:\Logs1\Subdir\Log2.log, etc.
+
+```json
+{
+    "TimberWinR": {
+        "Inputs": {
+            "TailFiles": [
+                {
+                    "logSource": "log files",
+                    "location": "C:\\Logs1\\*.log",
+                    "recurse": -1
+                }
+            ]
+		}
+	}
+}
+```
+## Fields
+After a successful parse of an event, the following fields are added:
+
+| Name | Type | Description |
+| ---- |:-----| :-----------|
+| LogFilename | STRING |Full path of the file containing this line | 
+| Index | INTEGER | Line number |
+| Text | STRING | Text line content  |

TimberWinR/mdocs/TcpInput.md

@@ -9,7 +9,7 @@ The following parameters are allowed when configuring the Tcp input.
 | :---------------- |:---------------| :----------------------------------------------------------------------- | :---------------------------  | :-- |
 | *port*        | integer  |Port number to open        | Must be an available port |     |
 
-Example Input: Monitors all files (recursively) located at C:\Logs1\ matching *.log as a pattern.  I.e. C:\Logs1\foo.log, C:\Logs1\Subdir\Log2.log, etc.
+Example Input: Listen on Port 5140
 
 ```json
 {

TimberWinR/mdocs/UdpInput.md

@@ -0,0 +1,28 @@
+# Input: Udp
+
+The Udp input will open a port and listen for properly formatted UDP datagrams to be broadcast.
+
+## Parameters
+The following parameters are allowed when configuring the Udp input.
+
+| Parameter         |     Type       |  Description                                                             | Details               |  Default |
+| :---------------- |:---------------| :----------------------------------------------------------------------- | :---------------------------  | :-- |
+| *port*        | integer  |Port number to open        | Must be an available port |     |
+
+Example Input: Listen on Port 5142
+
+```json
+{
+    "TimberWinR": {
+        "Inputs": {
+            "Udp": [
+                {
+                    "port": 5142                
+                }
+            ]
+		}
+	}
+}
+```
+## Fields
+A field: "type": "Win32-Udp" is automatically appended, and the entire JSON is passed on vertabim.

TimberWinR/mdocs/W3CInput.md

@@ -0,0 +1,51 @@
+# Input: W3CLogs
+
+The W3C input format parses IIS log files in the W3C Extended Log File Format, and handles custom fields unlike the IISW3C input.
+
+IIS web sites logging in the W3C Extended format can be configured to log only a specific subset of the available fields.
+Log files in this format begin with some informative headers ("directives"), the most important of which is the "#Fields" directive, 
+describing which fields are logged at which position in a log row. After the directives, the log entries follow. 
+Each log entry is a space-separated list of field values.
+
+If the logging configuration of an IIS virtual site is updated, the structure of the fields in the file that is 
+currently logged to might change according to the new configuration. In this case, a new "#Fields" directive is 
+logged describing the new fields structure, and the W3C input format keeps track of the structure change and 
+parses the new log entries accordingly. 
+
+
+## Parameters
+The following parameters are allowed when configuring W3CLogs input.
+
+| Parameter         |     Type       |  Description                                                             | Details               |  Default |
+| :---------------- |:---------------| :----------------------------------------------------------------------- | :---------------------------  | :-- |
+| *location*        | string  |Location of log files(s) to monitor                                     | Path to text file(s) including wildcards, may be separated by commas |     |
+| *iCodepage*       | integer |Codepage of the text file.                                              | 0 is the system codepage, -1 is UNICODE.                         | 0  |
+| *dtLines*         | integer |Number of lines examined to determine field types at run time. | This parameter specifies the number of initial log lines that the W3C input format examines to determine the data type of the input record fields. If the value is zero, all fields will be assumed to be of the STRING data type. | false |
+| *dQuotes*         | boolean |Specifies that string values in the log are double-quoted.                | Log processors might generate W3C logs whose string values are enclosed in double-quotes.                      | false |
+| *separator*       | enum    |Separator character between fields.    | Different W3C log files can use different separator characters between the fields; for example, Exchange Tracking log files use tab characters, while Personal Firewall log files use space characters. The "auto" value instructs the W3C input format to detect automatically the separator character used in the input log(s).   | auto/space/tab/character |
+
+Example Input:
+```json
+{
+    "TimberWinR": {
+        "Inputs": {
+            "W3CLogs": [
+                {                   
+                   "location": "C:\\inetpub\\logs\\LogFiles\\W3SVC1\\*"
+                }
+            ]
+		}
+	}
+}
+```
+
+
+## Fields
+After a successful parse of an event, the following fields are added [(if configured to be logged)](http://www.iis.net/learn/extensions/advanced-logging-module/advanced-logging-readme)
+
+| Name | Type | Description |
+| ---- |:-----| :-----------------------------------------------------------------------|
+|LogFilename| STRING | Full path of the log file containing this entry |
+|LogRow | INTEGER |  Line in the log file containing this entry  |
+
+After the above fields, all other fields selected to be logged will be appended.

TimberWinR/mdocs/WindowsEvents.md

@@ -2,7 +2,7 @@
 
 The WindowsEvents input will collect events from the Windows Event Viewer.   The source parameter indicates which event 
 logs to collect data from.  You can specify more than one log by using the comma, i.e.  "Application,System" will collect
-logs from the Application and System event logs.
+logs from the Application and System event logs.  The default interval for scanning for new Events is 60 seconds.
 
 ## Parameters
 The following parameters are allowed when configuring WindowsEvents.
@@ -18,6 +18,7 @@ The following parameters are allowed when configuring WindowsEvents.
 | *fullText*        | bool   |Retrieve the full text message                                           | true,false                   | **true** |
 | *resolveSIDS*     | bool   |Resolve SID values into full account names                               | true,false                   | **true** |
 | *formatMsg*       | bool   |Format the text message as a single line.                                | true,false                   | **true** |
+| *interval*        | integer  | Interval in seconds to sleep during checks                            | Interval                     | 60 |
 
 ### source format
 The source indicates where to collect the event(s) from, it can be of these form(s): 

TimberWinR/packages.config

@@ -1,8 +1,11 @@
 <?xml version="1.0" encoding="utf-8"?>
 <packages>
   <package id="csredis" version="1.4.7.1" targetFramework="net40" />
+  <package id="MaxMind.Db" version="0.2.3.0" targetFramework="net40" />
+  <package id="MaxMind.GeoIP2" version="0.4.0.0" targetFramework="net40" />
   <package id="Newtonsoft.Json" version="6.0.4" targetFramework="net40" />
   <package id="NLog" version="3.1.0.0" targetFramework="net40" />
   <package id="RapidRegex.Core" version="1.0.0.2" targetFramework="net40" />
   <package id="RestSharp" version="104.4.0" targetFramework="net40" />
+  <package id="System.Linq.Dynamic" version="1.0.3" targetFramework="net40" />
 </packages>

TimberWix/Product.wxs

@@ -6,6 +6,14 @@
     <Property Id="LOGDIR">c:\logs</Property>
     <Property Id="LOGLEVEL">Info</Property>
     <Property Id="DIAGPORT">5141</Property>
+
+    <PropertyRef Id="NETFRAMEWORK40FULL"/>
+    <Condition Message="This application requires Microsoft .NET Framework 4.0 Runtime in order to run. Please install the .NET Framework and then run this installer again.">
+      <![CDATA[Installed OR NETFRAMEWORK40FULL]]>
+    </Condition>
+    
+    <UIRef Id="WixUI_InstallDir" />
+
     <!--
 				We need to be able to uninstall a newer version from an older version.
 				The default reinstallmode is "omus", of which the 'o' means "reinstall if missing or older"
@@ -14,8 +22,7 @@
 			-->
     <Property Id="REINSTALLMODE" Value="dmus" />
 
-    
-    <MajorUpgrade DowngradeErrorMessage="A newer version of [ProductName] is already installed." />
+    <MajorUpgrade DowngradeErrorMessage="A newer version of [ProductName] is already installed."  />
     <MediaTemplate EmbedCab="yes" />
 
     
@@ -30,8 +37,11 @@
         <Directory Id="INSTALLFOLDER" Name="TimberWinR" />
       </Directory>
     </Directory>
+    <Property Id="WIXUI_INSTALLDIR" Value="INSTALLFOLDER" />
+
   </Fragment>
 
+   
   <Fragment>
     <ComponentGroup Id="ProductComponents" Directory="INSTALLFOLDER">
       <Component Id="ProductComponent" Guid="1BDEC5F3-5E9F-4E2E-8B1B-30E7968A99E1">
@@ -39,16 +49,22 @@
         <File Id="TimberWinR.ServiceHost.exe.config" Source="$(var.TimberWinR.ServiceHost.TargetDir)\TimberWinR.ServiceHost.exe.config" />
         <File Id="Interop.MSUtil.dll" Source="$(var.TimberWinR.ServiceHost.TargetDir)\Interop.MSUtil.dll"  />
         <File Id="csredis.dll" Source="$(var.TimberWinR.ServiceHost.TargetDir)\csredis.dll"  />
+        <File Id="default.json" Source="$(var.TimberWinR.ServiceHost.TargetDir)\default.json"  />
         <File Id="Newtonsoft.Json.dll" Source="$(var.TimberWinR.ServiceHost.TargetDir)\Newtonsoft.Json.dll"  />
         <File Id="Nlog.dll" Source="$(var.TimberWinR.ServiceHost.TargetDir)\Nlog.dll"  />
         <File Id="RestSharp.dll" Source="$(var.TimberWinR.ServiceHost.TargetDir)\RestSharp.dll"  />
         <File Id="RapidRegex.Core.dll" Source="$(var.TimberWinR.ServiceHost.TargetDir)\RapidRegex.Core.dll"  />
         <File Id="TimberWinR.dll" Source="$(var.TimberWinR.ServiceHost.TargetDir)\TimberWinR.dll"  />
+        <File Id="System.Linq.Dynamic.dll" Source="$(var.TimberWinR.ServiceHost.TargetDir)\System.Linq.Dynamic.dll"  />
         <File Id="Topshelf.dll" Source="$(var.TimberWinR.ServiceHost.TargetDir)\Topshelf.dll"  />
+        <File Id="MaxMind.Db.dll" Source="$(var.TimberWinR.TargetDir)\MaxMind.db.dll"  />
+        <File Id="MaxMind.GeoIP2.dll" Source="$(var.TimberWinR.TargetDir)\MaxMind.GeoIP2.dll"  />
+        <File Id="GeoLite2City.mmdb" Source="$(var.TimberWinR.TargetDir)\GeoLite2City.mmdb"  />
       </Component>
     </ComponentGroup>
 
     <InstallExecuteSequence>
+      <RemoveExistingProducts After="InstallFinalize" Overridable="yes" />
       <Custom Action='ManagedInstall' After="InstallFiles">NOT Installed</Custom>
       <Custom Action='ManagedInstall2' After="ManagedInstall">NOT Installed</Custom>
       <Custom Action='ManagedUnInstall' Before="RemoveFiles">Installed AND (NOT UPGRADINGPRODUCTCODE) AND (REMOVE="ALL")</Custom>

TimberWix/TimberWinR.Wix.wixproj

@@ -4,7 +4,7 @@
     <Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
     <Platform Condition=" '$(Platform)' == '' ">x86</Platform>
     <ProductVersion>3.8</ProductVersion>
-    <MSIVersionNumber>1.1.0</MSIVersionNumber>
+    <MSIVersionNumber>1.2.0</MSIVersionNumber>
     <ProjectGuid>82a39b31-61ec-468d-aa71-0d949ac6528f</ProjectGuid>
     <SchemaVersion>2.0</SchemaVersion>
     <OutputName>TimberWinR-$(MSIVersionNumber)</OutputName>
@@ -43,11 +43,26 @@
       <RefProjectOutputGroups>Binaries;Content;Satellites</RefProjectOutputGroups>
       <RefTargetDir>INSTALLFOLDER</RefTargetDir>
     </ProjectReference>
+    <ProjectReference Include="..\TimberWinR\TimberWinR.csproj">
+      <Name>TimberWinR</Name>
+      <Project>{4ef96a08-21db-4178-be44-70dae594632c}</Project>
+      <Private>True</Private>
+      <DoNotHarvest>True</DoNotHarvest>
+      <RefProjectOutputGroups>Binaries;Content;Satellites</RefProjectOutputGroups>
+      <RefTargetDir>INSTALLFOLDER</RefTargetDir>
+    </ProjectReference>
+  </ItemGroup>
+  <ItemGroup>
+    <WixExtension Include="WixNetFxExtension">
+      <HintPath>$(WixExtDir)\WixNetFxExtension.dll</HintPath>
+      <Name>WixNetFxExtension</Name>
+    </WixExtension>
+    <WixExtension Include="WixUIExtension">
+      <HintPath>$(WixExtDir)\WixUIExtension.dll</HintPath>
+      <Name>WixUIExtension</Name>
+    </WixExtension>
   </ItemGroup>
   <Import Project="$(WixTargetsPath)" />
-  <PropertyGroup>
-    <PostBuildEvent />
-  </PropertyGroup>
   <PropertyGroup>
     <PreBuildEvent />
   </PropertyGroup>
@@ -85,6 +100,9 @@
       <Output TaskParameter="Value" PropertyName="ProjectDefineConstants" />
     </CreateProperty>
   </Target>
+  <PropertyGroup>
+    <PostBuildEvent>$(SolutionDir)\TimberWinR.ExtractID\$(OutDir)\TimberWinR.ExtractID.exe $(TargetDir) $(SolutionDir)chocolateyUninstall.ps1.guid  $(SolutionDir)chocolateyUninstall.ps1.template</PostBuildEvent>
+  </PropertyGroup>
   <!--
 	To modify your build process, add your task inside one of the targets below and uncomment it.
 	Other similar extension points exist, see Wix.targets.

chocolateyInstall.ps1.template

@@ -0,0 +1,8 @@
+$packageName = 'TimberWinR-${version}' 
+$installerType = 'msi'
+$url = 'http://www.ericfontana.com/TimberWinR/TimberWinR-${version}.0.msi'
+$silentArgs = '/quiet' 
+$validExitCodes = @(0)
+Install-ChocolateyPackage "$packageName" "$installerType" "$silentArgs" "$url" "$url64"  -validExitCodes $validExitCodes
+
+

chocolateyUninstall.ps1.guid

@@ -0,0 +1,8 @@
+$packageName = 'TimberWinR-${version}' # arbitrary name for the package, used in messages
+$installerType = 'msi' #only one of these: exe, msi, msu
+$url = 'http://www.ericfontana.com/TimberWinR/TimberWinR-${version}.0.msi' # download url
+$silentArgs = '${PROJECTGUID} /quiet' 
+$validExitCodes = @(0) #please insert other valid exit codes here, exit codes for ms http://msdn.microsoft.com/en-us/library/aa368542(VS.85).aspx
+UnInstall-ChocolateyPackage "$packageName" "$installerType" "$silentArgs" "$url" -validExitCodes $validExitCodes
+
+

chocolateyUninstall.ps1.template

@@ -0,0 +1,8 @@
+$packageName = 'TimberWinR-${version}' # arbitrary name for the package, used in messages
+$installerType = 'msi' #only one of these: exe, msi, msu
+$url = 'http://www.ericfontana.com/TimberWinR/TimberWinR-${version}.0.msi' # download url
+$silentArgs = '{593EF0C4-54E0-40D5-A3E3-922CD1C25B9E} /quiet' 
+$validExitCodes = @(0) #please insert other valid exit codes here, exit codes for ms http://msdn.microsoft.com/en-us/library/aa368542(VS.85).aspx
+UnInstall-ChocolateyPackage "$packageName" "$installerType" "$silentArgs" "$url" -validExitCodes $validExitCodes
+
+

chocolateyUninstall.ps1.template.orig

@@ -0,0 +1,8 @@
+$packageName = 'TimberWinR-${version}' # arbitrary name for the package, used in messages
+$installerType = 'msi' #only one of these: exe, msi, msu
+$url = 'http://www.ericfontana.com/TimberWinR/TimberWinR-${version}.0.msi' # download url
+$silentArgs = '${PROJECTGUID} /quiet' 
+$validExitCodes = @(0) #please insert other valid exit codes here, exit codes for ms http://msdn.microsoft.com/en-us/library/aa368542(VS.85).aspx
+UnInstall-ChocolateyPackage "$packageName" "$installerType" "$silentArgs" "$url" -validExitCodes $validExitCodes
+
+

packages/MaxMind.DB.0.2.3.0/lib/net40/MaxMind.Db.xml

@@ -0,0 +1,279 @@
+<?xml version="1.0"?>
+<doc>
+    <assembly>
+        <name>MaxMind.Db</name>
+    </assembly>
+    <members>
+        <member name="T:MaxMind.Db.ObjectType">
+            <summary>
+            Enumeration representing the types of objects read from the database
+            </summary>
+        </member>
+        <member name="T:MaxMind.Db.Result">
+            <summary>
+            A data structure to store an object read from the database
+            </summary>
+        </member>
+        <member name="M:MaxMind.Db.Result.#ctor(Newtonsoft.Json.Linq.JToken,System.Int32)">
+            <summary>
+            Initializes a new instance of the <see cref="T:MaxMind.Db.Result"/> class.
+            </summary>
+            <param name="node">The node.</param>
+            <param name="offset">The offset.</param>
+        </member>
+        <member name="P:MaxMind.Db.Result.Node">
+            <summary>
+            The object read from the database
+            </summary>
+        </member>
+        <member name="P:MaxMind.Db.Result.Offset">
+            <summary>
+            The offset
+            </summary>
+        </member>
+        <member name="T:MaxMind.Db.Decoder">
+            <summary>
+            Given a stream, this class decodes the object graph at a particular location
+            </summary>
+        </member>
+        <member name="M:MaxMind.Db.Decoder.#ctor(System.Threading.ThreadLocal{System.IO.Stream},System.Int32)">
+            <summary>
+            Initializes a new instance of the <see cref="T:MaxMind.Db.Decoder"/> class.
+            </summary>
+            <param name="stream">The stream.</param>
+            <param name="pointerBase">The base address in the stream.</param>
+        </member>
+        <member name="M:MaxMind.Db.Decoder.Decode(System.Int32)">
+            <summary>
+            Decodes the object at the specified offset.
+            </summary>
+            <param name="offset">The offset.</param>
+            <returns>An object containing the data read from the stream</returns>
+        </member>
+        <member name="M:MaxMind.Db.Decoder.ReadOne(System.Int32)">
+            <summary>
+            Reads the one.
+            </summary>
+            <param name="position">The position.</param>
+            <returns></returns>
+        </member>
+        <member name="M:MaxMind.Db.Decoder.ReadMany(System.Int32,System.Int32)">
+            <summary>
+            Reads the many.
+            </summary>
+            <param name="position">The position.</param>
+            <param name="size">The size.</param>
+            <returns></returns>
+        </member>
+        <member name="M:MaxMind.Db.Decoder.DecodeByType(MaxMind.Db.ObjectType,System.Int32,System.Int32)">
+            <summary>
+            Decodes the type of the by.
+            </summary>
+            <param name="type">The type.</param>
+            <param name="offset">The offset.</param>
+            <param name="size">The size.</param>
+            <returns></returns>
+            <exception cref="T:System.Exception">Unable to handle type!</exception>
+        </member>
+        <member name="M:MaxMind.Db.Decoder.FromControlByte(System.Byte)">
+            <summary>
+            Froms the control byte.
+            </summary>
+            <param name="b">The attribute.</param>
+            <returns></returns>
+        </member>
+        <member name="M:MaxMind.Db.Decoder.SizeFromCtrlByte(System.Byte,System.Int32)">
+            <summary>
+            Sizes from control byte.
+            </summary>
+            <param name="ctrlByte">The control byte.</param>
+            <param name="offset">The offset.</param>
+            <returns></returns>
+        </member>
+        <member name="M:MaxMind.Db.Decoder.DecodeBoolean(System.Int32)">
+            <summary>
+            Decodes the boolean.
+            </summary>
+            <param name="size">The size of the structure.</param>
+            <returns></returns>
+        </member>
+        <member name="M:MaxMind.Db.Decoder.DecodeDouble(System.Byte[])">
+            <summary>
+            Decodes the double.
+            </summary>
+            <param name="buffer">The buffer.</param>
+            <returns></returns>
+        </member>
+        <member name="M:MaxMind.Db.Decoder.DecodeFloat(System.Byte[])">
+            <summary>
+            Decodes the float.
+            </summary>
+            <param name="buffer">The buffer.</param>
+            <returns></returns>
+        </member>
+        <member name="M:MaxMind.Db.Decoder.DecodeString(System.Byte[])">
+            <summary>
+            Decodes the string.
+            </summary>
+            <param name="buffer">The buffer.</param>
+            <returns></returns>
+        </member>
+        <member name="M:MaxMind.Db.Decoder.DecodeMap(System.Int32,System.Int32)">
+            <summary>
+            Decodes the map.
+            </summary>
+            <param name="size">The size.</param>
+            <param name="offset">The offset.</param>
+            <returns></returns>
+        </member>
+        <member name="M:MaxMind.Db.Decoder.DecodeLong(System.Byte[])">
+            <summary>
+            Decodes the long.
+            </summary>
+            <param name="buffer">The buffer.</param>
+            <returns></returns>
+        </member>
+        <member name="M:MaxMind.Db.Decoder.DecodeIntegerToJValue(System.Byte[])">
+            <summary>
+            Decodes the integer.
+            </summary>
+            <param name="buffer">The buffer.</param>
+            <returns></returns>
+        </member>
+        <member name="M:MaxMind.Db.Decoder.DecodeArray(System.Int32,System.Int32)">
+            <summary>
+            Decodes the array.
+            </summary>
+            <param name="size">The size.</param>
+            <param name="offset">The offset.</param>
+            <returns></returns>
+        </member>
+        <member name="M:MaxMind.Db.Decoder.DecodeUInt64(System.Byte[])">
+            <summary>
+            Decodes the uint64.
+            </summary>
+            <param name="buffer">The buffer.</param>
+            <returns></returns>
+        </member>
+        <member name="M:MaxMind.Db.Decoder.DecodeBigInteger(System.Byte[])">
+            <summary>
+            Decodes the big integer.
+            </summary>
+            <param name="buffer">The buffer.</param>
+            <returns></returns>
+        </member>
+        <member name="M:MaxMind.Db.Decoder.DecodePointer(System.Int32,System.Int32,System.Int32@)">
+            <summary>
+            Decodes the pointer.
+            </summary>
+            <param name="ctrlByte">The control byte.</param>
+            <param name="offset">The offset.</param>
+            <param name="outOffset">The resulting offset</param>
+            <returns></returns>
+        </member>
+        <member name="M:MaxMind.Db.Decoder.DecodeInteger(System.Byte[])">
+            <summary>
+            Decodes the integer.
+            </summary>
+            <param name="buffer">The buffer.</param>
+            <returns></returns>
+        </member>
+        <member name="M:MaxMind.Db.Decoder.DecodeInteger(System.Int32,System.Byte[])">
+            <summary>
+            Decodes the integer.
+            </summary>
+            <param name="baseValue">The base value.</param>
+            <param name="buffer">The buffer.</param>
+            <returns></returns>
+        </member>
+        <member name="T:MaxMind.Db.InvalidDatabaseException">
+            <summary>
+            Thrown when the MaxMind database file is incorrectly formatted
+            </summary>
+        </member>
+        <member name="M:MaxMind.Db.InvalidDatabaseException.#ctor(System.String)">
+            <summary>
+            Initializes a new instance of the <see cref="T:MaxMind.Db.InvalidDatabaseException"/> class.
+            </summary>
+            <param name="message">A message that describes the error.</param>
+        </member>
+        <member name="M:MaxMind.Db.InvalidDatabaseException.#ctor(System.String,System.Exception)">
+            <summary>
+            Initializes a new instance of the <see cref="T:MaxMind.Db.InvalidDatabaseException"/> class.
+            </summary>
+            <param name="message">The error message that explains the reason for the exception.</param>
+            <param name="innerException">The exception that is the cause of the current exception. If the <paramref name="innerException"/> parameter is not a null reference, the current exception is raised in a catch block that handles the inner exception.</param>
+        </member>
+        <member name="T:MaxMind.Db.FileAccessMode">
+            <summary>
+            An enumeration specifying the API to use to read the database
+            </summary>
+        </member>
+        <member name="F:MaxMind.Db.FileAccessMode.MemoryMapped">
+            <summary>
+            Open the file in memory mapped mode. Does not load into real memory.
+            </summary>
+        </member>
+        <member name="F:MaxMind.Db.FileAccessMode.Memory">
+            <summary>
+            Load the file into memory.
+            </summary>
+        </member>
+        <member name="T:MaxMind.Db.Reader">
+            <summary>
+            Given a MaxMind DB file, this class will retrieve information about an IP address
+            </summary>
+        </member>
+        <member name="M:MaxMind.Db.Reader.#ctor(System.String)">
+            <summary>
+            Initializes a new instance of the <see cref="T:MaxMind.Db.Reader"/> class.
+            </summary>
+            <param name="file">The file.</param>
+        </member>
+        <member name="M:MaxMind.Db.Reader.#ctor(System.String,MaxMind.Db.FileAccessMode)">
+            <summary>
+            Initializes a new instance of the <see cref="T:MaxMind.Db.Reader"/> class.
+            </summary>
+            <param name="file">The MaxMind DB file.</param>
+            <param name="mode">The mode by which to access the DB file.</param>
+        </member>
+        <member name="M:MaxMind.Db.Reader.#ctor(System.IO.Stream)">
+            <summary>
+            Initialize with Stream
+            </summary>
+            <param name="stream"></param>
+        </member>
+        <member name="M:MaxMind.Db.Reader.Find(System.String)">
+            <summary>
+            Finds the data related to the specified address.
+            </summary>
+            <param name="ipAddress">The IP address.</param>
+            <returns>An object containing the IP related data</returns>
+        </member>
+        <member name="M:MaxMind.Db.Reader.Find(System.Net.IPAddress)">
+            <summary>
+            Finds the data related to the specified address.
+            </summary>
+            <param name="ipAddress">The IP address.</param>
+            <returns>An object containing the IP related data</returns>
+        </member>
+        <member name="M:MaxMind.Db.Reader.Dispose">
+            <summary>
+            Release resources back to the system.
+            </summary>
+        </member>
+        <member name="P:MaxMind.Db.Reader.Metadata">
+            <summary>
+            Gets the metadata.
+            </summary>
+            <value>
+            The metadata.
+            </value>
+        </member>
+        <member name="T:MaxMind.Db.Metadata">
+            <summary>
+            Data about the database file itself
+            </summary>
+        </member>
+    </members>
+</doc>

timberwinr.nuspec.template

@@ -0,0 +1,27 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Do not remove this test for UTF-8: if ??? doesn?t appear as greek uppercase omega letter enclosed in quotation marks, you should use an editor that supports UTF-8, not this one. -->
+<package xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd">
+  <metadata>
+    <!-- Read this before publishing packages to chocolatey.org: https://github.com/chocolatey/chocolatey/wiki/CreatePackages -->
+    <id>TimberWinR</id>
+    <title>TimberWinR</title>
+    <version>${version}.0</version>
+    <authors>efontana</authors>
+    <owners>Eric Fontana</owners>
+    <summary>TimberWinR Shipper</summary>
+    <description>TimberWinR Native .NET Logstash Shipper.  Use https://groups.google.com/forum/#!forum/timberwinr for support.</description>
+    <projectUrl>https://github.com/Cimpress-MCP/TimberWinR</projectUrl>
+    <tags>TimberWinR admin</tags>
+    <copyright></copyright>
+    <iconUrl>http://www.ericfontana.com/timberwinr.jpg</iconUrl>
+    <licenseUrl>http://www.apache.org/licenses/LICENSE-2.0.html</licenseUrl>
+    <requireLicenseAcceptance>false</requireLicenseAcceptance>   
+    <dependencies>
+      <dependency id="LogParser" version="2.2.0.1" />
+    </dependencies>
+    <releaseNotes></releaseNotes>
+  </metadata>
+  <files>
+    <file src="tools\**" target="tools" />
+  </files>
+</package>