11 Commits

Author SHA1 Message Date
Eric Fontana
b07d15c9aa Must use secure version of url 2015-06-04 08:33:18 -04:00
Eric Fontana
063471ff28 Publish to Chocolatey use correct server. 2015-06-04 08:21:54 -04:00
Eric Fontana
606d72b55f .nuspec template needed version number fixing 2015-06-04 08:04:06 -04:00
Eric Fontana
d89e949185 Fixed version number for nuget package scripts. 2015-06-04 08:02:13 -04:00
Eric Fontana
d93ea4a5a2 Yaml wants spaces not tabs. 2015-06-04 07:52:05 -04:00
Eric Fontana
062b076c61 Commands must have newlines. 2015-06-04 07:50:55 -04:00
Eric Fontana
1312e2ddd5 Fixed tab -> spaces 2015-06-04 07:42:43 -04:00
Eric Fontana
9fe7e1e15a Package for choco after build. 2015-06-04 07:39:59 -04:00
Eric Fontana
9e11c4c926 Use correct env var for version. 2015-06-04 07:37:19 -04:00
Eric Fontana
1f211f2713 Fixed powershell before_build 2015-06-04 07:34:02 -04:00
Eric Fontana
2aebd5689a First attempt at new yaml. 2015-06-04 07:21:08 -04:00
13 changed files with 80 additions and 143 deletions

View File

@@ -2,14 +2,6 @@ TimberWinR
==========
A Native Windows to Redis/Elasticsearch Logstash Agent which runs as a service.
## Development and Roadmap
TimberWinR's development has been and in the short term will continue to be on a hiatus. The project's ownership has been recently transferred, and thus the project will be slow to pick up steam again. Currently a roadmap is being established, and a vision for the project is currently being formulated.
#### Getting Involved
If you wish to be involved with TimberWinR in an on-going basis file an issue.
## Why have TimberWinR?
TimberWinR is a native .NET implementation utilizing Microsoft's [LogParser](http://technet.microsoft.com/en-us/scriptcenter/dd919274.aspx). This means
no JVM/JRuby is required, and LogParser does all the heavy lifting. TimberWinR collects
@@ -17,7 +9,7 @@ the data from LogParser and ships it to Logstash via Redis (or can ship direcly
## Release Notes
[View Version History](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/ReleaseNotes.md)
[View Version History](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/ReleaseNotes.md)
## Basics
TimberWinR uses a configuration file to control how the logs are collected, filtered and shipped off.
@@ -38,27 +30,27 @@ Latest Build:
## Inputs
The current supported Input format sources are:
1. [Logs](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/Logs.md) (Files, a.k.a Tailing a file)
2. [Tcp](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/TcpInput.md) (listens on TCP port for JSON messages)
3. [IISW3C](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/IISW3CInput.md)(Internet Information Services W3C Format)
4. [WindowsEvents](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/WindowsEvents.md) (Windows Event Viewer)
5. [Stdin](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/StdinInput.md) (Standard Input for Debugging)
6. [W3C](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/W3CInput.md)(Internet Information Services W3C Advanced/Custom Format)
7. [Udp](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/UdpInput.md) (listens for UDP on port for JSON messages)
8. [TailFiles](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/TailFiles.md) (Tails log files efficiently)
8. [Generator](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/Generator.md) (Generate logs for testing *New*)
1. [Logs](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/Logs.md) (Files, a.k.a Tailing a file)
2. [Tcp](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/TcpInput.md) (listens on TCP port for JSON messages)
3. [IISW3C](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/IISW3CInput.md)(Internet Information Services W3C Format)
4. [WindowsEvents](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/WindowsEvents.md) (Windows Event Viewer)
5. [Stdin](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/StdinInput.md) (Standard Input for Debugging)
6. [W3C](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/W3CInput.md)(Internet Information Services W3C Advanced/Custom Format)
7. [Udp](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/UdpInput.md) (listens for UDP on port for JSON messages)
8. [TailFiles](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/TailFiles.md) (Tails log files efficiently)
8. [Generator](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/Generator.md) (Generate logs for testing *New*)
## Codecs
The current list of supported codecs are:
1. [Multiline](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/Codec.md)
1. [Multiline](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/Codec.md)
## Filters
The current list of supported filters are:
1. [Grok](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/GrokFilter.md)
2. [Mutate](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/MutateFilter.md)
3. [Date](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/DateFilter.md)
4. [Json](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/JsonFilter.md)
5. [GeoIP](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/GeoIPFilter.md)
1. [Grok](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/GrokFilter.md)
2. [Mutate](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/MutateFilter.md)
3. [Date](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/DateFilter.md)
4. [Json](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/JsonFilter.md)
5. [GeoIP](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/GeoIPFilter.md)
Note that there are now two syntaxes for filters, array and single, if you want more than one filter of the same
type then you must use the array tag instead of the singular tag. i.e:
@@ -112,11 +104,11 @@ Since TimberWinR only ships to Redis and Elasticsearch, the format generated by
represented as a JSON Property or Array.
## Outputs
1. [Redis](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/RedisOutput.md)
2. [Elasticsearch](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/ElasticsearchOutput.md)
3. [Stdout](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/StdoutOutput.md)
4. [File](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/FileOutput.md)
5. [StatsD](https://github.com/tparnell8/TimberWinR/blob/master/TimberWinR/mdocs/StatsD.md)
1. [Redis](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/RedisOutput.md)
2. [Elasticsearch](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/ElasticsearchOutput.md)
3. [Stdout](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/StdoutOutput.md)
4. [File](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/FileOutput.md)
5. [StatsD](https://github.com/Cimpress-MCP/TimberWinR/blob/master/TimberWinR/mdocs/StatsD.md)
## Sample Configuration
TimberWinR reads a JSON configuration file, an example file is shown here:

View File

@@ -1,6 +1,4 @@
using TimberWinR.Outputs;
namespace TimberWinR.UnitTests.Parser
namespace TimberWinR.UnitTests.Parser
{
using System;
@@ -53,51 +51,5 @@ namespace TimberWinR.UnitTests.Parser
Assert.AreEqual("someindex-" + DateTime.UtcNow.ToString("yyyy.MM.dd"), result);
}
[Test]
public void Given_no_ssl_then_validate_does_not_throw()
{
parser.Ssl = false;
Assert.That(() => parser.Validate(), Throws.Nothing);
}
[Test]
public void Given_ssl_and_no_username_then_validate_throws()
{
parser.Ssl = true;
parser.Password = "pass";
Assert.That(() => parser.Validate(), Throws.Exception.InstanceOf<ElasticsearchOutputParameters.ElasticsearchBasicAuthException>());
}
[Test]
public void Given_ssl_and_no_password_then_validate_throws()
{
parser.Ssl = true;
parser.Username = "user";
Assert.That(() => parser.Validate(), Throws.Exception.InstanceOf<ElasticsearchOutputParameters.ElasticsearchBasicAuthException>());
}
[Test]
public void Given_ssl_and_username_and_password_then_validate_does_not_throw()
{
parser.Ssl = true;
parser.Username = "user";
parser.Password = "pass";
Assert.That(() => parser.Validate(), Throws.Nothing);
}
[Test]
[TestCase("host", 1234, false, null, null, "http://host:1234/")]
[TestCase("host", 1234, true, "user", "pass", "https://user:pass@host:1234/")]
[TestCase("host", 1234, true, "user:", "pass@", "https://user%3A:pass%40@host:1234/")]
public void ComposeUri_Matches_Expected(string host, int port, bool ssl, string username, string password, string expectedUri)
{
var uri = ElasticsearchOutput.ComposeUri(host, port, ssl, username, password);
Assert.That(uri.ToString(), Is.EqualTo(expectedUri));
}
}
}

View File

@@ -29,9 +29,6 @@ namespace TimberWinR.Outputs
{
private TimberWinR.Manager _manager;
private readonly int _port;
private readonly bool _ssl;
private readonly string _username;
private readonly string _password;
private readonly int _interval;
private readonly int _flushSize;
private readonly int _idleFlushTimeSeconds;
@@ -60,8 +57,8 @@ namespace TimberWinR.Outputs
var nodes = new List<Uri>();
foreach (var host in _hosts)
{
var uri = ComposeUri(host, _port, _ssl, _username, _password);
nodes.Add(uri);
var url = string.Format("http://{0}:{1}", host, _port);
nodes.Add(new Uri(url));
}
var pool = new StaticConnectionPool(nodes.ToArray());
var settings = new ConnectionSettings(pool)
@@ -76,13 +73,6 @@ namespace TimberWinR.Outputs
return client;
}
public static Uri ComposeUri(string host, int port, bool ssl, string username, string password)
{
return ssl
? new Uri(string.Format("https://{0}:{1}@{2}:{3}", Uri.EscapeDataString(username), Uri.EscapeDataString(password), host, port))
: new Uri(string.Format("http://{0}:{1}", host, port));
}
public ElasticsearchOutput(TimberWinR.Manager manager, Parser.ElasticsearchOutputParameters parameters, CancellationToken cancelToken)
: base(cancelToken, "Elasticsearch")
{
@@ -96,9 +86,6 @@ namespace TimberWinR.Outputs
_timeout = parameters.Timeout;
_manager = manager;
_port = parameters.Port;
_ssl = parameters.Ssl;
_username = parameters.Username;
_password = parameters.Password;
_interval = parameters.Interval;
_hosts = parameters.Host;
_jsonQueue = new List<JObject>();
@@ -124,9 +111,6 @@ namespace TimberWinR.Outputs
new JProperty("messages", _sentMessages),
new JProperty("queuedMessageCount", _jsonQueue.Count),
new JProperty("port", _port),
new JProperty("ssl", _ssl),
new JProperty("username", _username),
new JProperty("password", _password),
new JProperty("flushSize", _flushSize),
new JProperty("idleFlushTime", _idleFlushTimeSeconds),
new JProperty("interval", _interval),

View File

@@ -625,16 +625,8 @@ namespace TimberWinR.Parser
}
public class ElasticsearchOutputParameters : IValidateSchema
public class ElasticsearchOutputParameters
{
public class ElasticsearchBasicAuthException : Exception
{
public ElasticsearchBasicAuthException()
: base("Elasticsearch 'username' and 'password' properties must be set when SSL is enabled.")
{
}
}
const string IndexDatePattern = "(%\\{(?<format>[^\\}]+)\\})";
[JsonProperty(PropertyName = "host")]
@@ -643,12 +635,6 @@ namespace TimberWinR.Parser
public string Index { get; set; }
[JsonProperty(PropertyName = "port")]
public int Port { get; set; }
[JsonProperty(PropertyName = "ssl")]
public bool Ssl { get; set; }
[JsonProperty(PropertyName = "username")]
public string Username { get; set; }
[JsonProperty(PropertyName = "password")]
public string Password { get; set; }
[JsonProperty(PropertyName = "timeout")]
public int Timeout { get; set; }
[JsonProperty(PropertyName = "threads")]
@@ -676,9 +662,6 @@ namespace TimberWinR.Parser
IdleFlushTimeInSeconds = 10;
Protocol = "http";
Port = 9200;
Ssl = false;
Username = string.Empty;
Password = string.Empty;
Index = "";
Host = new string[] { "localhost" };
Timeout = 10000;
@@ -728,11 +711,6 @@ namespace TimberWinR.Parser
return typeName;
}
public void Validate()
{
if (Ssl && (string.IsNullOrWhiteSpace(Username) || string.IsNullOrWhiteSpace(Password)))
throw new ElasticsearchBasicAuthException();
}
}
public class RedisOutputParameters

View File

@@ -14,9 +14,6 @@ The following parameters are allowed when configuring the Elasticsearch output.
| *interval* | integer | Interval in milliseconds to sleep during batch sends | Interval | 5000 |
| *max_queue_size* | integer | Maximum Elasticsearch queue depth | | 50000 |
| *port* | integer | Elasticsearch port number | This port must be open | 9200 |
| *ssl* | bool | If true, use an HTTPS connection to Elasticsearch. See [this page] (https://www.elastic.co/guide/en/found/current/elk-and-found.html#_using_logstash) for a configuration example. | *username* and *password* are also required for HTTPS connections. | false |
| *username* | string | Username for Elasticsearch credentials. | Required for HTTPS connection. | |
| *password* | string | Password for Elasticsearch credentials. | Required for HTTPS connection. | |
| *queue_overflow_discard_oldest* | bool | If true, discard oldest messages when max_queue_size reached otherwise discard newest | | true |
| *threads* | [string] | Number of Threads | Number of worker threads processing messages | 1 |
| *enable_ping* | bool | If true, pings the server to test for keep alive | | false |

View File

@@ -65,5 +65,3 @@ After a successful parse of an event, the following fields are added:
| Message | STRING | The full event message |
| Data | STRING | The binary data associated with the event |

View File

@@ -1,14 +1,50 @@
version: 1.2.{build}
build:
verbosity: minimal
assembly_info:
patch: true
file: AssemblyInfo.*
assembly_version: "1.2.{build}"
assembly_file_version: "{version}"
assembly_informational_version: "{version}"
artifacts:
- path: '**\*.msi'
version: build_number_{build}
configuration: Release
skip_tags: true
init:
- ps: "$v = [regex]::match($env:APPVEYOR_REPO_BRANCH,'release/(.*)').Groups[1].Value\nWrite-Host \"On branch $($env:APPVEYOR_REPO_BRANCH)\"\nIF($v) { \n $env:VERSION_FROM_BRANCH = \"$($v).$($env:APPVEYOR_BUILD_NUMBER)\"\n} else {\n $env:VERSION_FROM_BRANCH = \"0.0.0.$($env:APPVEYOR_BUILD_NUMBER)\"\n}\nWrite-Host \"Set version to $($env:VERSION_FROM_BRANCH)\""
assembly_info:
patch: true
file: '**\AssemblyInfo.*'
assembly_version: $(VERSION_FROM_BRANCH)
assembly_file_version: $(VERSION_FROM_BRANCH)
assembly_informational_version: $(VERSION_FROM_BRANCH)
before_build:
- ps: >-
mkdir tools
NuGet.exe restore TimberWinR.sln
build:
verbosity: normal
after_build:
- ps: >-
cat chocolateyInstall.ps1.template|%{$_-replace "\$\{version\}",$env:VERSION_FROM_BRANCH} > tools\chocolateyInstall.ps1
cat chocolateyUninstall.ps1.template|%{$_-replace "\$\{version\}",$env:VERSION_FROM_BRANCH} > tools\chocolateyUninstall.ps1
cat tools\chocolateyUninstall.ps1
cat timberwinr.nuspec.template|%{$_-replace "\$\{version\}",$env:VERSION_FROM_BRANCH} > timberwinr.nuspec
choco pack timberwinr.nuspec
artifacts:
- path: .\*.nupkg
name: NuGet
- path: TimberWinR.ServiceHost\bin\*\*.dll
name: Dlls
- path: TimberWinR.ServiceHost\bin\*\*.exe
name: Exes
- path: TimberWix\bin\*\*.msi
name: MSI
deploy:
- provider: NuGet
server: https://chocolatey.org/
api_key:
secure: y8zhZTNm3PWLLxtKiMO79Shx3TyG3lnuGpJ6J7GT+2SihEspFxuBXz+WvmI6qzX+
skip_symbols: true
artifact: NuGet
on:
branch: /release/.*/
- provider: GitHub
release: $(VERSION_FROM_BRANCH)
auth_token:

View File

@@ -2,7 +2,7 @@
$fileType = 'msi'
$silentArgs = '/quiet'
$scriptPath = $(Split-Path $MyInvocation.MyCommand.Path)
$fileFullPath = Join-Path $scriptPath 'TimberWinR-${version}.0.msi'
$fileFullPath = Join-Path $scriptPath 'TimberWinR-${version}.msi'
try {
Install-ChocolateyInstallPackage $packageName $fileType $silentArgs $fileFullPath
} catch {

View File

@@ -1,7 +1,7 @@
$packageName = 'TimberWinR-${version}' # arbitrary name for the package, used in messages
$installerType = 'msi' #only one of these: exe, msi, msu
$scriptPath = $(Split-Path $MyInvocation.MyCommand.Path)
$fileFullPath = Join-Path $scriptPath 'TimberWinR-${version}.0.msi'
$fileFullPath = Join-Path $scriptPath 'TimberWinR-${version}.msi'
$silentArgs = '${PROJECTGUID} /quiet'
$validExitCodes = @(0) #please insert other valid exit codes here, exit codes for ms http://msdn.microsoft.com/en-us/library/aa368542(VS.85).aspx
UnInstall-ChocolateyPackage "$packageName" "$installerType" "$silentArgs" "fileFullPath" -validExitCodes $validExitCodes

View File

@@ -1,7 +1,7 @@
$packageName = 'TimberWinR-${version}' # arbitrary name for the package, used in messages
$installerType = 'msi' #only one of these: exe, msi, msu
$scriptPath = $(Split-Path $MyInvocation.MyCommand.Path)
$fileFullPath = Join-Path $scriptPath 'TimberWinR-${version}.0.msi'
$fileFullPath = Join-Path $scriptPath 'TimberWinR-${version}.msi'
$silentArgs = '{E001D138-669B-4604-88C5-02C756461C15} /quiet'
$validExitCodes = @(0) #please insert other valid exit codes here, exit codes for ms http://msdn.microsoft.com/en-us/library/aa368542(VS.85).aspx
UnInstall-ChocolateyPackage "$packageName" "$installerType" "$silentArgs" "fileFullPath" -validExitCodes $validExitCodes

View File

@@ -1,7 +1,7 @@
$packageName = 'TimberWinR-${version}' # arbitrary name for the package, used in messages
$installerType = 'msi' #only one of these: exe, msi, msu
$scriptPath = $(Split-Path $MyInvocation.MyCommand.Path)
$fileFullPath = Join-Path $scriptPath 'TimberWinR-${version}.0.msi'
$fileFullPath = Join-Path $scriptPath 'TimberWinR-${version}.msi'
$silentArgs = '${PROJECTGUID} /quiet'
$validExitCodes = @(0) #please insert other valid exit codes here, exit codes for ms http://msdn.microsoft.com/en-us/library/aa368542(VS.85).aspx
UnInstall-ChocolateyPackage "$packageName" "$installerType" "$silentArgs" "fileFullPath" -validExitCodes $validExitCodes

Binary file not shown.

Before

Width:  |  Height:  |  Size: 6.0 KiB

View File

@@ -5,7 +5,7 @@
<!-- Read this before publishing packages to chocolatey.org: https://github.com/chocolatey/chocolatey/wiki/CreatePackages -->
<id>TimberWinR</id>
<title>TimberWinR</title>
<version>${version}.0</version>
<version>${version}</version>
<authors>efontana</authors>
<owners>Eric Fontana</owners>
<summary>TimberWinR Shipper</summary>