Compare commits
2 Commits
release/1.
...
master
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5165224c65 | ||
|
|
fd9306e24e |
52
README.md
52
README.md
@@ -2,6 +2,14 @@ TimberWinR
|
|||||||
==========
|
==========
|
||||||
A Native Windows to Redis/Elasticsearch Logstash Agent which runs as a service.
|
A Native Windows to Redis/Elasticsearch Logstash Agent which runs as a service.
|
||||||
|
|
||||||
|
## Development and Roadmap
|
||||||
|
|
||||||
|
TimberWinR's development has been and in the short term will continue to be on a hiatus. The project's ownership has been recently transferred, and thus the project will be slow to pick up steam again. Currently a roadmap is being established, and a vision for the project is currently being formulated.
|
||||||
|
|
||||||
|
#### Getting Involved
|
||||||
|
|
||||||
|
If you wish to be involved with TimberWinR in an on-going basis file an issue.
|
||||||
|
|
||||||
## Why have TimberWinR?
|
## Why have TimberWinR?
|
||||||
TimberWinR is a native .NET implementation utilizing Microsoft's [LogParser](http://technet.microsoft.com/en-us/scriptcenter/dd919274.aspx). This means
|
TimberWinR is a native .NET implementation utilizing Microsoft's [LogParser](http://technet.microsoft.com/en-us/scriptcenter/dd919274.aspx). This means
|
||||||
no JVM/JRuby is required, and LogParser does all the heavy lifting. TimberWinR collects
|
no JVM/JRuby is required, and LogParser does all the heavy lifting. TimberWinR collects
|
||||||
@@ -9,7 +17,7 @@ the data from LogParser and ships it to Logstash via Redis (or can ship direcly
|
|||||||
|
|
||||||
## Release Notes
|
## Release Notes
|
||||||
|
|
||||||
[View Version History](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/ReleaseNotes.md)
|
[View Version History](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/ReleaseNotes.md)
|
||||||
|
|
||||||
## Basics
|
## Basics
|
||||||
TimberWinR uses a configuration file to control how the logs are collected, filtered and shipped off.
|
TimberWinR uses a configuration file to control how the logs are collected, filtered and shipped off.
|
||||||
@@ -30,27 +38,27 @@ Latest Build:
|
|||||||
|
|
||||||
## Inputs
|
## Inputs
|
||||||
The current supported Input format sources are:
|
The current supported Input format sources are:
|
||||||
1. [Logs](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/Logs.md) (Files, a.k.a Tailing a file)
|
1. [Logs](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/Logs.md) (Files, a.k.a Tailing a file)
|
||||||
2. [Tcp](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/TcpInput.md) (listens on TCP port for JSON messages)
|
2. [Tcp](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/TcpInput.md) (listens on TCP port for JSON messages)
|
||||||
3. [IISW3C](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/IISW3CInput.md)(Internet Information Services W3C Format)
|
3. [IISW3C](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/IISW3CInput.md)(Internet Information Services W3C Format)
|
||||||
4. [WindowsEvents](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/WindowsEvents.md) (Windows Event Viewer)
|
4. [WindowsEvents](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/WindowsEvents.md) (Windows Event Viewer)
|
||||||
5. [Stdin](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/StdinInput.md) (Standard Input for Debugging)
|
5. [Stdin](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/StdinInput.md) (Standard Input for Debugging)
|
||||||
6. [W3C](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/W3CInput.md)(Internet Information Services W3C Advanced/Custom Format)
|
6. [W3C](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/W3CInput.md)(Internet Information Services W3C Advanced/Custom Format)
|
||||||
7. [Udp](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/UdpInput.md) (listens for UDP on port for JSON messages)
|
7. [Udp](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/UdpInput.md) (listens for UDP on port for JSON messages)
|
||||||
8. [TailFiles](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/TailFiles.md) (Tails log files efficiently)
|
8. [TailFiles](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/TailFiles.md) (Tails log files efficiently)
|
||||||
8. [Generator](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/Generator.md) (Generate logs for testing *New*)
|
8. [Generator](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/Generator.md) (Generate logs for testing *New*)
|
||||||
|
|
||||||
## Codecs
|
## Codecs
|
||||||
The current list of supported codecs are:
|
The current list of supported codecs are:
|
||||||
1. [Multiline](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/Codec.md)
|
1. [Multiline](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/Codec.md)
|
||||||
|
|
||||||
## Filters
|
## Filters
|
||||||
The current list of supported filters are:
|
The current list of supported filters are:
|
||||||
1. [Grok](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/GrokFilter.md)
|
1. [Grok](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/GrokFilter.md)
|
||||||
2. [Mutate](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/MutateFilter.md)
|
2. [Mutate](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/MutateFilter.md)
|
||||||
3. [Date](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/DateFilter.md)
|
3. [Date](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/DateFilter.md)
|
||||||
4. [Json](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/JsonFilter.md)
|
4. [Json](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/JsonFilter.md)
|
||||||
5. [GeoIP](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/GeoIPFilter.md)
|
5. [GeoIP](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/GeoIPFilter.md)
|
||||||
|
|
||||||
Note that there are now two syntaxes for filters, array and single, if you want more than one filter of the same
|
Note that there are now two syntaxes for filters, array and single, if you want more than one filter of the same
|
||||||
type then you must use the array tag instead of the singular tag. i.e:
|
type then you must use the array tag instead of the singular tag. i.e:
|
||||||
@@ -104,11 +112,11 @@ Since TimberWinR only ships to Redis and Elasticsearch, the format generated by
|
|||||||
represented as a JSON Property or Array.
|
represented as a JSON Property or Array.
|
||||||
|
|
||||||
## Outputs
|
## Outputs
|
||||||
1. [Redis](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/RedisOutput.md)
|
1. [Redis](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/RedisOutput.md)
|
||||||
2. [Elasticsearch](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/ElasticsearchOutput.md)
|
2. [Elasticsearch](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/ElasticsearchOutput.md)
|
||||||
3. [Stdout](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/StdoutOutput.md)
|
3. [Stdout](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/StdoutOutput.md)
|
||||||
4. [File](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/FileOutput.md)
|
4. [File](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/FileOutput.md)
|
||||||
5. [StatsD](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/StatsD.md)
|
5. [StatsD](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/StatsD.md)
|
||||||
|
|
||||||
## Sample Configuration
|
## Sample Configuration
|
||||||
TimberWinR reads a JSON configuration file, an example file is shown here:
|
TimberWinR reads a JSON configuration file, an example file is shown here:
|
||||||
@@ -261,3 +269,5 @@ Use these commands to Stop/Start the service.
|
|||||||
sc stop TimberWinR ; stop the service
|
sc stop TimberWinR ; stop the service
|
||||||
sc start TimberWinR; start the service
|
sc start TimberWinR; start the service
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -32,5 +32,5 @@ using System.Runtime.InteropServices;
|
|||||||
// You can specify all the values or you can default the Build and Revision Numbers
|
// You can specify all the values or you can default the Build and Revision Numbers
|
||||||
// by using the '*' as shown below:
|
// by using the '*' as shown below:
|
||||||
// [assembly: AssemblyVersion("1.0.*")]
|
// [assembly: AssemblyVersion("1.0.*")]
|
||||||
[assembly: AssemblyVersion("1.3.27.0")]
|
[assembly: AssemblyVersion("1.3.26.0")]
|
||||||
[assembly: AssemblyFileVersion("1.3.27.0")]
|
[assembly: AssemblyFileVersion("1.3.26.0")]
|
||||||
|
|||||||
@@ -8,7 +8,7 @@ using System.Threading;
|
|||||||
using System.Net;
|
using System.Net;
|
||||||
using System.Net.Sockets;
|
using System.Net.Sockets;
|
||||||
using System.IO;
|
using System.IO;
|
||||||
using System.Linq.Expressions;
|
|
||||||
using Newtonsoft.Json.Linq;
|
using Newtonsoft.Json.Linq;
|
||||||
|
|
||||||
using NLog;
|
using NLog;
|
||||||
@@ -52,48 +52,29 @@ namespace TimberWinR.Diagnostics
|
|||||||
public JObject DiagnosticsOutput()
|
public JObject DiagnosticsOutput()
|
||||||
{
|
{
|
||||||
JObject json = new JObject(
|
JObject json = new JObject(
|
||||||
new JProperty("timberwinr",
|
new JProperty("timberwinr",
|
||||||
new JObject(
|
new JObject(
|
||||||
new JProperty("version", Assembly.GetEntryAssembly().GetName().Version.ToString()),
|
new JProperty("version", Assembly.GetEntryAssembly().GetName().Version.ToString()),
|
||||||
new JProperty("messages", Manager.NumMessages),
|
new JProperty("messages", Manager.NumMessages),
|
||||||
new JProperty("startedon", Manager.StartedOn),
|
new JProperty("startedon", Manager.StartedOn),
|
||||||
new JProperty("configfile", Manager.JsonConfig),
|
new JProperty("configfile", Manager.JsonConfig),
|
||||||
new JProperty("logdir", Manager.LogfileDir),
|
new JProperty("logdir", Manager.LogfileDir),
|
||||||
new JProperty("logginglevel", LogManager.GlobalThreshold.ToString())
|
new JProperty("logginglevel", LogManager.GlobalThreshold.ToString()),
|
||||||
)));
|
new JProperty("inputs",
|
||||||
AddDiagnosis(json);
|
new JArray(
|
||||||
|
from i in Manager.Listeners
|
||||||
|
select new JObject(i.ToJson()))),
|
||||||
|
new JProperty("filters",
|
||||||
|
new JArray(
|
||||||
|
from f in Manager.Config.Filters
|
||||||
|
select new JObject(f.ToJson()))),
|
||||||
|
new JProperty("outputs",
|
||||||
|
new JArray(
|
||||||
|
from o in Manager.Outputs
|
||||||
|
select new JObject(o.ToJson()))))));
|
||||||
return json;
|
return json;
|
||||||
}
|
}
|
||||||
|
|
||||||
protected void AddDiagnosis(JObject wrapper)
|
|
||||||
{
|
|
||||||
wrapper.Add("inputs", GetDiagnosisByType("inputs", Manager.Listeners.ToList<IDiagnosable>()));
|
|
||||||
wrapper.Add("filters", GetDiagnosisByType("filters", Manager.Config.Filters.ToList<IDiagnosable>()));
|
|
||||||
wrapper.Add("outputs", GetDiagnosisByType("inputs", Manager.Outputs.ToList<IDiagnosable>()));
|
|
||||||
}
|
|
||||||
|
|
||||||
protected JObject GetDiagnosisByType(String type, List<IDiagnosable> diags)
|
|
||||||
{
|
|
||||||
JObject category = new JObject();
|
|
||||||
foreach(IDiagnosable diag in diags)
|
|
||||||
{
|
|
||||||
JArray array = GetTypeArray(diag.GetType().Name.ToString(), category);
|
|
||||||
array.Add(diag.ToJson());
|
|
||||||
}
|
|
||||||
return category;
|
|
||||||
}
|
|
||||||
|
|
||||||
protected JArray GetTypeArray(String name, JObject category)
|
|
||||||
{
|
|
||||||
JArray ret = (JArray)category.GetValue(name);
|
|
||||||
if (ret == null)
|
|
||||||
{
|
|
||||||
ret = new JArray();
|
|
||||||
category.Add(new JProperty(name, ret));
|
|
||||||
}
|
|
||||||
return ret;
|
|
||||||
}
|
|
||||||
|
|
||||||
private void DiagnosticCallback(IAsyncResult result)
|
private void DiagnosticCallback(IAsyncResult result)
|
||||||
{
|
{
|
||||||
if (web == null)
|
if (web == null)
|
||||||
|
|||||||
@@ -1,13 +0,0 @@
|
|||||||
using System;
|
|
||||||
using System.Collections.Generic;
|
|
||||||
using System.Linq;
|
|
||||||
using System.Text;
|
|
||||||
using Newtonsoft.Json.Linq;
|
|
||||||
|
|
||||||
namespace TimberWinR.Diagnostics
|
|
||||||
{
|
|
||||||
public interface IDiagnosable
|
|
||||||
{
|
|
||||||
JObject ToJson();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -135,6 +135,8 @@ namespace TimberWinR.Inputs
|
|||||||
rs.close();
|
rs.close();
|
||||||
GC.Collect();
|
GC.Collect();
|
||||||
}
|
}
|
||||||
|
if (!Stop)
|
||||||
|
syncHandle.Wait(TimeSpan.FromSeconds(_pollingIntervalInSeconds), CancelToken);
|
||||||
}
|
}
|
||||||
catch (OperationCanceledException)
|
catch (OperationCanceledException)
|
||||||
{
|
{
|
||||||
@@ -143,18 +145,7 @@ namespace TimberWinR.Inputs
|
|||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
LogManager.GetCurrentClassLogger().Error(ex);
|
LogManager.GetCurrentClassLogger().Error(ex);
|
||||||
}
|
}
|
||||||
finally
|
|
||||||
{
|
|
||||||
try
|
|
||||||
{
|
|
||||||
if (!Stop)
|
|
||||||
syncHandle.Wait(TimeSpan.FromSeconds(_pollingIntervalInSeconds), CancelToken);
|
|
||||||
}
|
|
||||||
catch (Exception)
|
|
||||||
{
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -8,11 +8,10 @@ using System.Linq;
|
|||||||
using System.Text;
|
using System.Text;
|
||||||
using System.Threading;
|
using System.Threading;
|
||||||
using NLog;
|
using NLog;
|
||||||
using TimberWinR.Diagnostics;
|
|
||||||
|
|
||||||
namespace TimberWinR.Inputs
|
namespace TimberWinR.Inputs
|
||||||
{
|
{
|
||||||
public abstract class InputListener: IDiagnosable
|
public abstract class InputListener
|
||||||
{
|
{
|
||||||
public CancellationToken CancelToken { get; set; }
|
public CancellationToken CancelToken { get; set; }
|
||||||
public event Action<JObject> OnMessageRecieved;
|
public event Action<JObject> OnMessageRecieved;
|
||||||
|
|||||||
@@ -8,7 +8,6 @@ using TimberWinR.Parser;
|
|||||||
using LogQuery = Interop.MSUtil.LogQueryClassClass;
|
using LogQuery = Interop.MSUtil.LogQueryClassClass;
|
||||||
using EventLogInputFormat = Interop.MSUtil.COMEventLogInputContextClassClass;
|
using EventLogInputFormat = Interop.MSUtil.COMEventLogInputContextClassClass;
|
||||||
using LogRecordSet = Interop.MSUtil.ILogRecordset;
|
using LogRecordSet = Interop.MSUtil.ILogRecordset;
|
||||||
using System.IO;
|
|
||||||
|
|
||||||
namespace TimberWinR.Inputs
|
namespace TimberWinR.Inputs
|
||||||
{
|
{
|
||||||
@@ -98,13 +97,12 @@ namespace TimberWinR.Inputs
|
|||||||
// Execute the query
|
// Execute the query
|
||||||
if (!CancelToken.IsCancellationRequested)
|
if (!CancelToken.IsCancellationRequested)
|
||||||
{
|
{
|
||||||
var oLogQuery = new LogQuery();
|
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
|
var oLogQuery = new LogQuery();
|
||||||
|
|
||||||
var qfiles = string.Format("SELECT Distinct [EventLog] FROM {0}", location);
|
var qfiles = string.Format("SELECT Distinct [EventLog] FROM {0}", location);
|
||||||
var rsfiles = oLogQuery.Execute(qfiles, iFmt);
|
var rsfiles = oLogQuery.Execute(qfiles, iFmt);
|
||||||
|
|
||||||
for (; !rsfiles.atEnd(); rsfiles.moveNext())
|
for (; !rsfiles.atEnd(); rsfiles.moveNext())
|
||||||
{
|
{
|
||||||
var record = rsfiles.getRecord();
|
var record = rsfiles.getRecord();
|
||||||
@@ -115,7 +113,7 @@ namespace TimberWinR.Inputs
|
|||||||
logName);
|
logName);
|
||||||
var rcount = oLogQuery.Execute(qcount, iFmt);
|
var rcount = oLogQuery.Execute(qcount, iFmt);
|
||||||
var qr = rcount.getRecord();
|
var qr = rcount.getRecord();
|
||||||
var lrn = (Int64) qr.getValueEx("MaxRecordNumber");
|
var lrn = (Int64)qr.getValueEx("MaxRecordNumber");
|
||||||
logFileMaxRecords[logName] = lrn;
|
logFileMaxRecords[logName] = lrn;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -139,13 +137,12 @@ namespace TimberWinR.Inputs
|
|||||||
object v = record.getValue(field.Name);
|
object v = record.getValue(field.Name);
|
||||||
if (field.Name == "Data")
|
if (field.Name == "Data")
|
||||||
v = ToPrintable(v.ToString());
|
v = ToPrintable(v.ToString());
|
||||||
if ((field.Name == "TimeGenerated" || field.Name == "TimeWritten") &&
|
if ((field.Name == "TimeGenerated" || field.Name == "TimeWritten") && field.DataType == typeof (DateTime))
|
||||||
field.DataType == typeof (DateTime))
|
|
||||||
v = ((DateTime) v).ToUniversalTime();
|
v = ((DateTime) v).ToUniversalTime();
|
||||||
json.Add(new JProperty(field.Name, v));
|
json.Add(new JProperty(field.Name, v));
|
||||||
}
|
}
|
||||||
|
|
||||||
var lrn = (Int64) record.getValueEx("RecordNumber");
|
var lrn = (Int64)record.getValueEx("RecordNumber");
|
||||||
logFileMaxRecords[fileName] = lrn;
|
logFileMaxRecords[fileName] = lrn;
|
||||||
|
|
||||||
ProcessJson(json);
|
ProcessJson(json);
|
||||||
@@ -166,24 +163,6 @@ namespace TimberWinR.Inputs
|
|||||||
{
|
{
|
||||||
LogManager.GetCurrentClassLogger().Error(ex);
|
LogManager.GetCurrentClassLogger().Error(ex);
|
||||||
}
|
}
|
||||||
finally
|
|
||||||
{
|
|
||||||
try
|
|
||||||
{
|
|
||||||
oLogQuery = null;
|
|
||||||
// Sleep
|
|
||||||
if (!Stop)
|
|
||||||
syncHandle.Wait(TimeSpan.FromSeconds(_pollingIntervalInSeconds), CancelToken);
|
|
||||||
}
|
|
||||||
catch (OperationCanceledException)
|
|
||||||
{
|
|
||||||
}
|
|
||||||
catch (Exception ex1)
|
|
||||||
{
|
|
||||||
LogManager.GetCurrentClassLogger().Warn(ex1);
|
|
||||||
}
|
|
||||||
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
Finished();
|
Finished();
|
||||||
|
|||||||
@@ -4,12 +4,11 @@ using System.Linq;
|
|||||||
using System.Text;
|
using System.Text;
|
||||||
using System.Threading;
|
using System.Threading;
|
||||||
using Newtonsoft.Json.Linq;
|
using Newtonsoft.Json.Linq;
|
||||||
using TimberWinR.Diagnostics;
|
|
||||||
using TimberWinR.Inputs;
|
using TimberWinR.Inputs;
|
||||||
|
|
||||||
namespace TimberWinR.Outputs
|
namespace TimberWinR.Outputs
|
||||||
{
|
{
|
||||||
public abstract class OutputSender : IDiagnosable
|
public abstract class OutputSender
|
||||||
{
|
{
|
||||||
public CancellationToken CancelToken { get; private set; }
|
public CancellationToken CancelToken { get; private set; }
|
||||||
private List<InputListener> _inputs;
|
private List<InputListener> _inputs;
|
||||||
|
|||||||
@@ -132,7 +132,6 @@ namespace TimberWinR.Outputs
|
|||||||
private long _errorCount;
|
private long _errorCount;
|
||||||
private long _redisDepth;
|
private long _redisDepth;
|
||||||
private DateTime? _lastErrorTimeUTC;
|
private DateTime? _lastErrorTimeUTC;
|
||||||
private DateTime? _lastSentTimeUTC;
|
|
||||||
private readonly int _maxQueueSize;
|
private readonly int _maxQueueSize;
|
||||||
private readonly bool _queueOverflowDiscardOldest;
|
private readonly bool _queueOverflowDiscardOldest;
|
||||||
private BatchCounter _batchCounter;
|
private BatchCounter _batchCounter;
|
||||||
@@ -181,7 +180,6 @@ namespace TimberWinR.Outputs
|
|||||||
new JProperty("host", string.Join(",", _redisHosts)),
|
new JProperty("host", string.Join(",", _redisHosts)),
|
||||||
new JProperty("errors", _errorCount),
|
new JProperty("errors", _errorCount),
|
||||||
new JProperty("lastErrorTimeUTC", _lastErrorTimeUTC),
|
new JProperty("lastErrorTimeUTC", _lastErrorTimeUTC),
|
||||||
new JProperty("lastSentTimeUTC", _lastSentTimeUTC),
|
|
||||||
new JProperty("redisQueueDepth", _redisDepth),
|
new JProperty("redisQueueDepth", _redisDepth),
|
||||||
new JProperty("sentMessageCount", _sentMessages),
|
new JProperty("sentMessageCount", _sentMessages),
|
||||||
new JProperty("queuedMessageCount", _jsonQueue.Count),
|
new JProperty("queuedMessageCount", _jsonQueue.Count),
|
||||||
@@ -319,13 +317,9 @@ namespace TimberWinR.Outputs
|
|||||||
_batchCounter.SampleQueueDepth(_jsonQueue.Count);
|
_batchCounter.SampleQueueDepth(_jsonQueue.Count);
|
||||||
// Re-compute current batch size
|
// Re-compute current batch size
|
||||||
|
|
||||||
LogManager.GetCurrentClassLogger()
|
LogManager.GetCurrentClassLogger().Trace("{0}: Average Queue Depth: {1}, Current Length: {2}", Thread.CurrentThread.ManagedThreadId, _batchCounter.AverageQueueDepth(), _jsonQueue.Count);
|
||||||
.Trace("{0}: Average Queue Depth: {1}, Current Length: {2}",
|
|
||||||
Thread.CurrentThread.ManagedThreadId, _batchCounter.AverageQueueDepth(),
|
_currentBatchCount = _batchCounter.UpdateCurrentBatchCount(_jsonQueue.Count, _currentBatchCount);
|
||||||
_jsonQueue.Count);
|
|
||||||
|
|
||||||
_currentBatchCount = _batchCounter.UpdateCurrentBatchCount(_jsonQueue.Count,
|
|
||||||
_currentBatchCount);
|
|
||||||
|
|
||||||
messages = _jsonQueue.Take(_currentBatchCount).ToArray();
|
messages = _jsonQueue.Take(_currentBatchCount).ToArray();
|
||||||
_jsonQueue.RemoveRange(0, messages.Length);
|
_jsonQueue.RemoveRange(0, messages.Length);
|
||||||
@@ -346,9 +340,7 @@ namespace TimberWinR.Outputs
|
|||||||
{
|
{
|
||||||
client.StartPipe();
|
client.StartPipe();
|
||||||
LogManager.GetCurrentClassLogger()
|
LogManager.GetCurrentClassLogger()
|
||||||
.Debug("{0}: Sending {1} Messages to {2}",
|
.Debug("{0}: Sending {1} Messages to {2}", Thread.CurrentThread.ManagedThreadId, messages.Length, client.Host);
|
||||||
Thread.CurrentThread.ManagedThreadId, messages.Length,
|
|
||||||
client.Host);
|
|
||||||
|
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
@@ -356,7 +348,6 @@ namespace TimberWinR.Outputs
|
|||||||
Interlocked.Add(ref _sentMessages, messages.Length);
|
Interlocked.Add(ref _sentMessages, messages.Length);
|
||||||
client.EndPipe();
|
client.EndPipe();
|
||||||
sentSuccessfully = true;
|
sentSuccessfully = true;
|
||||||
_lastSentTimeUTC = DateTime.UtcNow;
|
|
||||||
if (messages.Length > 0)
|
if (messages.Length > 0)
|
||||||
_manager.IncrementMessageCount(messages.Length);
|
_manager.IncrementMessageCount(messages.Length);
|
||||||
}
|
}
|
||||||
@@ -400,7 +391,9 @@ namespace TimberWinR.Outputs
|
|||||||
_jsonQueue.InsertRange(0, messages);
|
_jsonQueue.InsertRange(0, messages);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if (!Stop)
|
||||||
|
syncHandle.Wait(TimeSpan.FromMilliseconds(_interval), CancelToken);
|
||||||
}
|
}
|
||||||
catch (OperationCanceledException)
|
catch (OperationCanceledException)
|
||||||
{
|
{
|
||||||
@@ -416,17 +409,6 @@ namespace TimberWinR.Outputs
|
|||||||
Interlocked.Increment(ref _errorCount);
|
Interlocked.Increment(ref _errorCount);
|
||||||
LogManager.GetCurrentClassLogger().Error(ex);
|
LogManager.GetCurrentClassLogger().Error(ex);
|
||||||
}
|
}
|
||||||
finally
|
|
||||||
{
|
|
||||||
try
|
|
||||||
{
|
|
||||||
if (!Stop)
|
|
||||||
syncHandle.Wait(TimeSpan.FromMilliseconds(_interval), CancelToken);
|
|
||||||
}
|
|
||||||
catch (Exception)
|
|
||||||
{
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -15,7 +15,6 @@ using NLog;
|
|||||||
using NLog.Config;
|
using NLog.Config;
|
||||||
using TimberWinR.Outputs;
|
using TimberWinR.Outputs;
|
||||||
using System.CodeDom.Compiler;
|
using System.CodeDom.Compiler;
|
||||||
using TimberWinR.Diagnostics;
|
|
||||||
|
|
||||||
namespace TimberWinR.Parser
|
namespace TimberWinR.Parser
|
||||||
{
|
{
|
||||||
@@ -27,7 +26,7 @@ namespace TimberWinR.Parser
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
public abstract class LogstashFilter : IValidateSchema, IDiagnosable
|
public abstract class LogstashFilter : IValidateSchema
|
||||||
{
|
{
|
||||||
public abstract bool Apply(JObject json);
|
public abstract bool Apply(JObject json);
|
||||||
|
|
||||||
|
|||||||
@@ -96,7 +96,6 @@
|
|||||||
<Compile Include="Configuration.cs" />
|
<Compile Include="Configuration.cs" />
|
||||||
<Compile Include="ConfigurationErrors.cs" />
|
<Compile Include="ConfigurationErrors.cs" />
|
||||||
<Compile Include="Diagnostics\Diagnostics.cs" />
|
<Compile Include="Diagnostics\Diagnostics.cs" />
|
||||||
<Compile Include="Diagnostics\IDiagnosable.cs" />
|
|
||||||
<Compile Include="Filters\DateFilter.cs" />
|
<Compile Include="Filters\DateFilter.cs" />
|
||||||
<Compile Include="Filters\FilterBase.cs" />
|
<Compile Include="Filters\FilterBase.cs" />
|
||||||
<Compile Include="Filters\GrokFilter.cs" />
|
<Compile Include="Filters\GrokFilter.cs" />
|
||||||
|
|||||||
44
appveyor.yml
44
appveyor.yml
@@ -1,38 +1,14 @@
|
|||||||
version: build_number_{build}
|
version: 1.2.{build}
|
||||||
configuration: Release
|
|
||||||
skip_tags: true
|
build:
|
||||||
init:
|
verbosity: minimal
|
||||||
- ps: "$v = [regex]::match($env:APPVEYOR_REPO_BRANCH,'release/(.*)').Groups[1].Value\nWrite-Host \"On branch $($env:APPVEYOR_REPO_BRANCH)\"\nIF($v) { \n $env:VERSION_FROM_BRANCH = \"$($v).$($env:APPVEYOR_BUILD_NUMBER)\"\n} else {\n $env:VERSION_FROM_BRANCH = \"0.0.0.$($env:APPVEYOR_BUILD_NUMBER)\"\n}\nWrite-Host \"Set version to $($env:VERSION_FROM_BRANCH)\""
|
|
||||||
assembly_info:
|
assembly_info:
|
||||||
patch: true
|
patch: true
|
||||||
file: '**\AssemblyInfo.*'
|
file: AssemblyInfo.*
|
||||||
assembly_version: $(VERSION_FROM_BRANCH)
|
assembly_version: "1.2.{build}"
|
||||||
assembly_file_version: $(VERSION_FROM_BRANCH)
|
assembly_file_version: "{version}"
|
||||||
assembly_informational_version: $(VERSION_FROM_BRANCH)
|
assembly_informational_version: "{version}"
|
||||||
before_build:
|
|
||||||
- ps: >-
|
|
||||||
mkdir tools
|
|
||||||
|
|
||||||
NuGet.exe restore TimberWinR.sln
|
|
||||||
build:
|
|
||||||
verbosity: normal
|
|
||||||
after_build:
|
|
||||||
- ps: >-
|
|
||||||
cat chocolateyInstall.ps1.template|%{$_-replace "\$\{version\}",$env:VERSION_FROM_BRANCH} > tools\chocolateyInstall.ps1
|
|
||||||
|
|
||||||
cat chocolateyUninstall.ps1.template|%{$_-replace "\$\{version\}",$env:VERSION_FROM_BRANCH} > tools\chocolateyUninstall.ps1
|
|
||||||
|
|
||||||
cat tools\chocolateyUninstall.ps1
|
|
||||||
|
|
||||||
cat timberwinr.nuspec.template|%{$_-replace "\$\{version\}",$env:VERSION_FROM_BRANCH} > timberwinr.nuspec
|
|
||||||
|
|
||||||
choco pack timberwinr.nuspec
|
|
||||||
artifacts:
|
artifacts:
|
||||||
- path: .\*.nupkg
|
- path: '**\*.msi'
|
||||||
name: NuGet
|
|
||||||
- path: TimberWinR.ServiceHost\bin\*\*.dll
|
|
||||||
name: Dlls
|
|
||||||
- path: TimberWinR.ServiceHost\bin\*\*.exe
|
|
||||||
name: Exes
|
|
||||||
- path: TimberWix\bin\*\*.msi
|
|
||||||
name: MSI
|
|
||||||
|
|||||||
@@ -6,14 +6,14 @@
|
|||||||
<id>TimberWinR</id>
|
<id>TimberWinR</id>
|
||||||
<title>TimberWinR</title>
|
<title>TimberWinR</title>
|
||||||
<version>${version}.0</version>
|
<version>${version}.0</version>
|
||||||
<authors>ethergreg</authors>
|
<authors>efontana</authors>
|
||||||
<owners>Cimpress-MCP Infrastructure</owners>
|
<owners>Eric Fontana</owners>
|
||||||
<summary>TimberWinR Shipper</summary>
|
<summary>TimberWinR Shipper</summary>
|
||||||
<description>TimberWinR Native .NET Logstash Shipper. Use https://groups.google.com/forum/#!forum/timberwinr for support.</description>
|
<description>TimberWinR Native .NET Logstash Shipper. Use https://groups.google.com/forum/#!forum/timberwinr for support.</description>
|
||||||
<projectUrl>https://github.com/Cimpress-MCP/TimberWinR</projectUrl>
|
<projectUrl>https://github.com/Cimpress-MCP/TimberWinR</projectUrl>
|
||||||
<tags>TimberWinR admin</tags>
|
<tags>TimberWinR admin</tags>
|
||||||
<copyright></copyright>
|
<copyright></copyright>
|
||||||
<iconUrl>https://raw.githubusercontent.com/Cimpress-MCP/TimberWinR/master/timberwinr.jpg</iconUrl>
|
<iconUrl>http://www.ericfontana.com/timberwinr.jpg</iconUrl>
|
||||||
<licenseUrl>http://www.apache.org/licenses/LICENSE-2.0.html</licenseUrl>
|
<licenseUrl>http://www.apache.org/licenses/LICENSE-2.0.html</licenseUrl>
|
||||||
<requireLicenseAcceptance>false</requireLicenseAcceptance>
|
<requireLicenseAcceptance>false</requireLicenseAcceptance>
|
||||||
<dependencies>
|
<dependencies>
|
||||||
|
|||||||
Reference in New Issue
Block a user