2 Commits

Author SHA1 Message Date
Tommy Parnell
5165224c65 change DEPRECATION notice 2016-01-20 08:06:25 -05:00
Ryan Breen
fd9306e24e Update README.md
Add deprecation notice.
2015-11-05 14:53:57 -05:00
13 changed files with 85 additions and 183 deletions

View File

@@ -2,6 +2,14 @@ TimberWinR
==========
A Native Windows to Redis/Elasticsearch Logstash Agent which runs as a service.
## Development and Roadmap
TimberWinR's development has been and in the short term will continue to be on a hiatus. The project's ownership has been recently transferred, and thus the project will be slow to pick up steam again. Currently a roadmap is being established, and a vision for the project is currently being formulated.
#### Getting Involved
If you wish to be involved with TimberWinR in an on-going basis file an issue.
## Why have TimberWinR?
TimberWinR is a native .NET implementation utilizing Microsoft's [LogParser](http://technet.microsoft.com/en-us/scriptcenter/dd919274.aspx). This means
no JVM/JRuby is required, and LogParser does all the heavy lifting. TimberWinR collects
@@ -9,7 +17,7 @@ the data from LogParser and ships it to Logstash via Redis (or can ship direcly
## Release Notes
[View Version History](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/ReleaseNotes.md)
[View Version History](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/ReleaseNotes.md)
## Basics
TimberWinR uses a configuration file to control how the logs are collected, filtered and shipped off.
@@ -30,27 +38,27 @@ Latest Build:
## Inputs
The current supported Input format sources are:
1. [Logs](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/Logs.md) (Files, a.k.a Tailing a file)
2. [Tcp](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/TcpInput.md) (listens on TCP port for JSON messages)
3. [IISW3C](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/IISW3CInput.md)(Internet Information Services W3C Format)
4. [WindowsEvents](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/WindowsEvents.md) (Windows Event Viewer)
5. [Stdin](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/StdinInput.md) (Standard Input for Debugging)
6. [W3C](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/W3CInput.md)(Internet Information Services W3C Advanced/Custom Format)
7. [Udp](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/UdpInput.md) (listens for UDP on port for JSON messages)
8. [TailFiles](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/TailFiles.md) (Tails log files efficiently)
8. [Generator](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/Generator.md) (Generate logs for testing *New*)
1. [Logs](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/Logs.md) (Files, a.k.a Tailing a file)
2. [Tcp](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/TcpInput.md) (listens on TCP port for JSON messages)
3. [IISW3C](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/IISW3CInput.md)(Internet Information Services W3C Format)
4. [WindowsEvents](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/WindowsEvents.md) (Windows Event Viewer)
5. [Stdin](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/StdinInput.md) (Standard Input for Debugging)
6. [W3C](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/W3CInput.md)(Internet Information Services W3C Advanced/Custom Format)
7. [Udp](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/UdpInput.md) (listens for UDP on port for JSON messages)
8. [TailFiles](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/TailFiles.md) (Tails log files efficiently)
8. [Generator](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/Generator.md) (Generate logs for testing *New*)
## Codecs
The current list of supported codecs are:
1. [Multiline](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/Codec.md)
1. [Multiline](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/Codec.md)
## Filters
The current list of supported filters are:
1. [Grok](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/GrokFilter.md)
2. [Mutate](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/MutateFilter.md)
3. [Date](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/DateFilter.md)
4. [Json](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/JsonFilter.md)
5. [GeoIP](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/GeoIPFilter.md)
1. [Grok](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/GrokFilter.md)
2. [Mutate](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/MutateFilter.md)
3. [Date](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/DateFilter.md)
4. [Json](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/JsonFilter.md)
5. [GeoIP](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/GeoIPFilter.md)
Note that there are now two syntaxes for filters, array and single, if you want more than one filter of the same
type then you must use the array tag instead of the singular tag. i.e:
@@ -104,11 +112,11 @@ Since TimberWinR only ships to Redis and Elasticsearch, the format generated by
represented as a JSON Property or Array.
## Outputs
1. [Redis](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/RedisOutput.md)
2. [Elasticsearch](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/ElasticsearchOutput.md)
3. [Stdout](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/StdoutOutput.md)
4. [File](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/FileOutput.md)
5. [StatsD](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/StatsD.md)
1. [Redis](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/RedisOutput.md)
2. [Elasticsearch](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/ElasticsearchOutput.md)
3. [Stdout](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/StdoutOutput.md)
4. [File](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/FileOutput.md)
5. [StatsD](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/StatsD.md)
## Sample Configuration
TimberWinR reads a JSON configuration file, an example file is shown here:
@@ -261,3 +269,5 @@ Use these commands to Stop/Start the service.
sc stop TimberWinR ; stop the service
sc start TimberWinR; start the service
```

View File

@@ -32,5 +32,5 @@ using System.Runtime.InteropServices;
// You can specify all the values or you can default the Build and Revision Numbers
// by using the '*' as shown below:
// [assembly: AssemblyVersion("1.0.*")]
[assembly: AssemblyVersion("1.3.27.0")]
[assembly: AssemblyFileVersion("1.3.27.0")]
[assembly: AssemblyVersion("1.3.26.0")]
[assembly: AssemblyFileVersion("1.3.26.0")]

View File

@@ -8,7 +8,7 @@ using System.Threading;
using System.Net;
using System.Net.Sockets;
using System.IO;
using System.Linq.Expressions;
using Newtonsoft.Json.Linq;
using NLog;
@@ -52,48 +52,29 @@ namespace TimberWinR.Diagnostics
public JObject DiagnosticsOutput()
{
JObject json = new JObject(
new JProperty("timberwinr",
new JObject(
new JProperty("version", Assembly.GetEntryAssembly().GetName().Version.ToString()),
new JProperty("messages", Manager.NumMessages),
new JProperty("startedon", Manager.StartedOn),
new JProperty("configfile", Manager.JsonConfig),
new JProperty("logdir", Manager.LogfileDir),
new JProperty("logginglevel", LogManager.GlobalThreshold.ToString())
)));
AddDiagnosis(json);
new JProperty("timberwinr",
new JObject(
new JProperty("version", Assembly.GetEntryAssembly().GetName().Version.ToString()),
new JProperty("messages", Manager.NumMessages),
new JProperty("startedon", Manager.StartedOn),
new JProperty("configfile", Manager.JsonConfig),
new JProperty("logdir", Manager.LogfileDir),
new JProperty("logginglevel", LogManager.GlobalThreshold.ToString()),
new JProperty("inputs",
new JArray(
from i in Manager.Listeners
select new JObject(i.ToJson()))),
new JProperty("filters",
new JArray(
from f in Manager.Config.Filters
select new JObject(f.ToJson()))),
new JProperty("outputs",
new JArray(
from o in Manager.Outputs
select new JObject(o.ToJson()))))));
return json;
}
protected void AddDiagnosis(JObject wrapper)
{
wrapper.Add("inputs", GetDiagnosisByType("inputs", Manager.Listeners.ToList<IDiagnosable>()));
wrapper.Add("filters", GetDiagnosisByType("filters", Manager.Config.Filters.ToList<IDiagnosable>()));
wrapper.Add("outputs", GetDiagnosisByType("inputs", Manager.Outputs.ToList<IDiagnosable>()));
}
protected JObject GetDiagnosisByType(String type, List<IDiagnosable> diags)
{
JObject category = new JObject();
foreach(IDiagnosable diag in diags)
{
JArray array = GetTypeArray(diag.GetType().Name.ToString(), category);
array.Add(diag.ToJson());
}
return category;
}
protected JArray GetTypeArray(String name, JObject category)
{
JArray ret = (JArray)category.GetValue(name);
if (ret == null)
{
ret = new JArray();
category.Add(new JProperty(name, ret));
}
return ret;
}
private void DiagnosticCallback(IAsyncResult result)
{
if (web == null)

View File

@@ -1,13 +0,0 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Newtonsoft.Json.Linq;
namespace TimberWinR.Diagnostics
{
public interface IDiagnosable
{
JObject ToJson();
}
}

View File

@@ -135,6 +135,8 @@ namespace TimberWinR.Inputs
rs.close();
GC.Collect();
}
if (!Stop)
syncHandle.Wait(TimeSpan.FromSeconds(_pollingIntervalInSeconds), CancelToken);
}
catch (OperationCanceledException)
{
@@ -143,18 +145,7 @@ namespace TimberWinR.Inputs
catch (Exception ex)
{
LogManager.GetCurrentClassLogger().Error(ex);
}
finally
{
try
{
if (!Stop)
syncHandle.Wait(TimeSpan.FromSeconds(_pollingIntervalInSeconds), CancelToken);
}
catch (Exception)
{
}
}
}
}
}
}

View File

@@ -8,11 +8,10 @@ using System.Linq;
using System.Text;
using System.Threading;
using NLog;
using TimberWinR.Diagnostics;
namespace TimberWinR.Inputs
{
public abstract class InputListener: IDiagnosable
public abstract class InputListener
{
public CancellationToken CancelToken { get; set; }
public event Action<JObject> OnMessageRecieved;

View File

@@ -8,7 +8,6 @@ using TimberWinR.Parser;
using LogQuery = Interop.MSUtil.LogQueryClassClass;
using EventLogInputFormat = Interop.MSUtil.COMEventLogInputContextClassClass;
using LogRecordSet = Interop.MSUtil.ILogRecordset;
using System.IO;
namespace TimberWinR.Inputs
{
@@ -98,13 +97,12 @@ namespace TimberWinR.Inputs
// Execute the query
if (!CancelToken.IsCancellationRequested)
{
var oLogQuery = new LogQuery();
try
{
{
var oLogQuery = new LogQuery();
var qfiles = string.Format("SELECT Distinct [EventLog] FROM {0}", location);
var rsfiles = oLogQuery.Execute(qfiles, iFmt);
for (; !rsfiles.atEnd(); rsfiles.moveNext())
{
var record = rsfiles.getRecord();
@@ -115,7 +113,7 @@ namespace TimberWinR.Inputs
logName);
var rcount = oLogQuery.Execute(qcount, iFmt);
var qr = rcount.getRecord();
var lrn = (Int64) qr.getValueEx("MaxRecordNumber");
var lrn = (Int64)qr.getValueEx("MaxRecordNumber");
logFileMaxRecords[logName] = lrn;
}
}
@@ -139,13 +137,12 @@ namespace TimberWinR.Inputs
object v = record.getValue(field.Name);
if (field.Name == "Data")
v = ToPrintable(v.ToString());
if ((field.Name == "TimeGenerated" || field.Name == "TimeWritten") &&
field.DataType == typeof (DateTime))
if ((field.Name == "TimeGenerated" || field.Name == "TimeWritten") && field.DataType == typeof (DateTime))
v = ((DateTime) v).ToUniversalTime();
json.Add(new JProperty(field.Name, v));
}
var lrn = (Int64) record.getValueEx("RecordNumber");
var lrn = (Int64)record.getValueEx("RecordNumber");
logFileMaxRecords[fileName] = lrn;
ProcessJson(json);
@@ -166,24 +163,6 @@ namespace TimberWinR.Inputs
{
LogManager.GetCurrentClassLogger().Error(ex);
}
finally
{
try
{
oLogQuery = null;
// Sleep
if (!Stop)
syncHandle.Wait(TimeSpan.FromSeconds(_pollingIntervalInSeconds), CancelToken);
}
catch (OperationCanceledException)
{
}
catch (Exception ex1)
{
LogManager.GetCurrentClassLogger().Warn(ex1);
}
}
}
}
Finished();

View File

@@ -4,12 +4,11 @@ using System.Linq;
using System.Text;
using System.Threading;
using Newtonsoft.Json.Linq;
using TimberWinR.Diagnostics;
using TimberWinR.Inputs;
namespace TimberWinR.Outputs
{
public abstract class OutputSender : IDiagnosable
public abstract class OutputSender
{
public CancellationToken CancelToken { get; private set; }
private List<InputListener> _inputs;

View File

@@ -132,7 +132,6 @@ namespace TimberWinR.Outputs
private long _errorCount;
private long _redisDepth;
private DateTime? _lastErrorTimeUTC;
private DateTime? _lastSentTimeUTC;
private readonly int _maxQueueSize;
private readonly bool _queueOverflowDiscardOldest;
private BatchCounter _batchCounter;
@@ -181,7 +180,6 @@ namespace TimberWinR.Outputs
new JProperty("host", string.Join(",", _redisHosts)),
new JProperty("errors", _errorCount),
new JProperty("lastErrorTimeUTC", _lastErrorTimeUTC),
new JProperty("lastSentTimeUTC", _lastSentTimeUTC),
new JProperty("redisQueueDepth", _redisDepth),
new JProperty("sentMessageCount", _sentMessages),
new JProperty("queuedMessageCount", _jsonQueue.Count),
@@ -319,13 +317,9 @@ namespace TimberWinR.Outputs
_batchCounter.SampleQueueDepth(_jsonQueue.Count);
// Re-compute current batch size
LogManager.GetCurrentClassLogger()
.Trace("{0}: Average Queue Depth: {1}, Current Length: {2}",
Thread.CurrentThread.ManagedThreadId, _batchCounter.AverageQueueDepth(),
_jsonQueue.Count);
_currentBatchCount = _batchCounter.UpdateCurrentBatchCount(_jsonQueue.Count,
_currentBatchCount);
LogManager.GetCurrentClassLogger().Trace("{0}: Average Queue Depth: {1}, Current Length: {2}", Thread.CurrentThread.ManagedThreadId, _batchCounter.AverageQueueDepth(), _jsonQueue.Count);
_currentBatchCount = _batchCounter.UpdateCurrentBatchCount(_jsonQueue.Count, _currentBatchCount);
messages = _jsonQueue.Take(_currentBatchCount).ToArray();
_jsonQueue.RemoveRange(0, messages.Length);
@@ -346,9 +340,7 @@ namespace TimberWinR.Outputs
{
client.StartPipe();
LogManager.GetCurrentClassLogger()
.Debug("{0}: Sending {1} Messages to {2}",
Thread.CurrentThread.ManagedThreadId, messages.Length,
client.Host);
.Debug("{0}: Sending {1} Messages to {2}", Thread.CurrentThread.ManagedThreadId, messages.Length, client.Host);
try
{
@@ -356,7 +348,6 @@ namespace TimberWinR.Outputs
Interlocked.Add(ref _sentMessages, messages.Length);
client.EndPipe();
sentSuccessfully = true;
_lastSentTimeUTC = DateTime.UtcNow;
if (messages.Length > 0)
_manager.IncrementMessageCount(messages.Length);
}
@@ -400,7 +391,9 @@ namespace TimberWinR.Outputs
_jsonQueue.InsertRange(0, messages);
}
}
}
}
if (!Stop)
syncHandle.Wait(TimeSpan.FromMilliseconds(_interval), CancelToken);
}
catch (OperationCanceledException)
{
@@ -416,17 +409,6 @@ namespace TimberWinR.Outputs
Interlocked.Increment(ref _errorCount);
LogManager.GetCurrentClassLogger().Error(ex);
}
finally
{
try
{
if (!Stop)
syncHandle.Wait(TimeSpan.FromMilliseconds(_interval), CancelToken);
}
catch (Exception)
{
}
}
}
}
}

View File

@@ -15,7 +15,6 @@ using NLog;
using NLog.Config;
using TimberWinR.Outputs;
using System.CodeDom.Compiler;
using TimberWinR.Diagnostics;
namespace TimberWinR.Parser
{
@@ -27,7 +26,7 @@ namespace TimberWinR.Parser
}
public abstract class LogstashFilter : IValidateSchema, IDiagnosable
public abstract class LogstashFilter : IValidateSchema
{
public abstract bool Apply(JObject json);

View File

@@ -96,7 +96,6 @@
<Compile Include="Configuration.cs" />
<Compile Include="ConfigurationErrors.cs" />
<Compile Include="Diagnostics\Diagnostics.cs" />
<Compile Include="Diagnostics\IDiagnosable.cs" />
<Compile Include="Filters\DateFilter.cs" />
<Compile Include="Filters\FilterBase.cs" />
<Compile Include="Filters\GrokFilter.cs" />

View File

@@ -1,38 +1,14 @@
version: build_number_{build}
configuration: Release
skip_tags: true
init:
- ps: "$v = [regex]::match($env:APPVEYOR_REPO_BRANCH,'release/(.*)').Groups[1].Value\nWrite-Host \"On branch $($env:APPVEYOR_REPO_BRANCH)\"\nIF($v) { \n $env:VERSION_FROM_BRANCH = \"$($v).$($env:APPVEYOR_BUILD_NUMBER)\"\n} else {\n $env:VERSION_FROM_BRANCH = \"0.0.0.$($env:APPVEYOR_BUILD_NUMBER)\"\n}\nWrite-Host \"Set version to $($env:VERSION_FROM_BRANCH)\""
version: 1.2.{build}
build:
verbosity: minimal
assembly_info:
patch: true
file: '**\AssemblyInfo.*'
assembly_version: $(VERSION_FROM_BRANCH)
assembly_file_version: $(VERSION_FROM_BRANCH)
assembly_informational_version: $(VERSION_FROM_BRANCH)
before_build:
- ps: >-
mkdir tools
file: AssemblyInfo.*
assembly_version: "1.2.{build}"
assembly_file_version: "{version}"
assembly_informational_version: "{version}"
NuGet.exe restore TimberWinR.sln
build:
verbosity: normal
after_build:
- ps: >-
cat chocolateyInstall.ps1.template|%{$_-replace "\$\{version\}",$env:VERSION_FROM_BRANCH} > tools\chocolateyInstall.ps1
cat chocolateyUninstall.ps1.template|%{$_-replace "\$\{version\}",$env:VERSION_FROM_BRANCH} > tools\chocolateyUninstall.ps1
cat tools\chocolateyUninstall.ps1
cat timberwinr.nuspec.template|%{$_-replace "\$\{version\}",$env:VERSION_FROM_BRANCH} > timberwinr.nuspec
choco pack timberwinr.nuspec
artifacts:
- path: .\*.nupkg
name: NuGet
- path: TimberWinR.ServiceHost\bin\*\*.dll
name: Dlls
- path: TimberWinR.ServiceHost\bin\*\*.exe
name: Exes
- path: TimberWix\bin\*\*.msi
name: MSI
- path: '**\*.msi'

View File

@@ -6,14 +6,14 @@
<id>TimberWinR</id>
<title>TimberWinR</title>
<version>${version}.0</version>
<authors>ethergreg</authors>
<owners>Cimpress-MCP Infrastructure</owners>
<authors>efontana</authors>
<owners>Eric Fontana</owners>
<summary>TimberWinR Shipper</summary>
<description>TimberWinR Native .NET Logstash Shipper. Use https://groups.google.com/forum/#!forum/timberwinr for support.</description>
<projectUrl>https://github.com/Cimpress-MCP/TimberWinR</projectUrl>
<tags>TimberWinR admin</tags>
<copyright></copyright>
<iconUrl>https://raw.githubusercontent.com/Cimpress-MCP/TimberWinR/master/timberwinr.jpg</iconUrl>
<iconUrl>http://www.ericfontana.com/timberwinr.jpg</iconUrl>
<licenseUrl>http://www.apache.org/licenses/LICENSE-2.0.html</licenseUrl>
<requireLicenseAcceptance>false</requireLicenseAcceptance>
<dependencies>