Automatically handle TypeName parameter for different versions of Elasticsearch (<7, 7 or higher) (#462)

* fix file references in the Visual Studio Solution file.

* Do not set `TypeName` by default any more.
* last version of Elasticsearch that supported user-defined `_type` field - v6.8.x is running out of support on 2022-02-10 (https://www.elastic.co/support/eol)

* Automatically handle `ElasticsearchSinkOptions.TypeName` for different versions Elasticsearch (<7, 7+) when `ElasticsearchSinkOptions.DetectElasticsearchVersion` is enabled.

* Add unit test for automatic settings of `TypeName` when `DetectElasticsearchVersion` is set to `true`.
- Two methods used - instantiate `ElasticsearchSinkState` directly and via `LoggerConfiguration`

* Add Elasticsearch v8 template + parsing of Elasticsearch major version to `int` + decision branch for which version of index-template API to use + removal of obsolete `ElasticsearchTemplateProvider.GetTemplate(...)` method overload.

* Upgrade to .NET 6.0 and update test-frameworks related NuGet pacakges

* Upgrade to Elasticsearch.NET 7.17.5 + handle new "preflight" request

https://discuss.elastic.co/t/the-client-is-unable-to-verify-that-the-server-is-elasticsearch-due-to-an-unsuccessful-product-check-call-some-functionality-may-not-be-compatible-if-the-server-is-running-an-unsupported-product/310969/9

"The 7.16 client performs a pre-flight GET request to the root URL of the server before the first request.".

* Make `ConnectionStub` a bit more robust .

* Use `System.Version` to parse Elasticsearch server version number (similar to what `Elasticsearch.Net` does)

* Update NuGet packages

* Replace obsolete NuGet package `Serilog.Sinks.ColoredConsole` with `Serilog.Sinks.Console`

* Update `Serilog.Sinks.PeriodicBatching` package and reimplent `ElasticsearchSink` so that it does not use obsolete `PeriodicBatchingSink` constructor.

* Better handling of Elasticsearch server version number in mocked `ConnectionStub`

* Cleanup: refactor to use single `JsonEquals` method.

* Turn on `DetectElasticSearchVersion` option by default. Assume version 7 on fallback.

* Cleanup: remove unused namespaces

* Cleanup: move `ElasticsearchSinkTestsBase` into `Stubs` subfolder.

* Refactor: extract `ConnectionStub` into a separate file.

* Fix: json comparison in .NET Framework 4.6+

* Run unit-tests on multiple .NET frameworks.

* Cleanup: remove unused NUnit runner package.

* Use newer, built-in compilation constants.
https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/preprocessor-directives#conditional-compilation

* Use standard MSBuild property `IsPackable` for clarity.

* Cleanup: remove unused package refrence.

* Update GitHub actions

* docs: updated documentation to reflect  changes in behavior of the sink.

Co-authored-by: Nenad Vicentic <vicentic@immounited.com>

Author: nenadvicentic

CHANGES.md

@@ -6,6 +6,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [9.0.0] - 2023-01-23
+
+### Added
+ - PR #462
+
+### Major Changes
+- `DetectElasticsearchVersion` is set to `true` by default.
+- When `DetectElasticsearchVersion` is set to `false` Elasticsearch version 7 is assumed (as it has broadest wire-compatibility at the moment - v7 and v8)
+- When `DetectElasticsearchVersion` is set to `true`, `TypeName` is handled automatically across different versions of Elasticserach (6.x to 8.x). For example, user-defined name will NOT be used on v7 and v8. Also, correct templates endpoint will be picked up.
+- Elasticsearch 8.x endpoint for templates is supported (`_index_template`)
+- Internal class `ElasticsearchVersionManager` has been added, mainly to handle situations where detection of version fails or when it is disabled. In the case of fallback, sink will assume "default" version 7.
+- Elasticsearch.NET client version 7.15.2 (latest version 7, until new `Elastic.Clients.Elasticsearch` 8.x catches up functional parity with 7.x)
+
+### Other Changes
+- Nuget pacakges have been updated (except for the Elasticsearch integration-tests related packages)
+- Most of the `ElasticserachSink` functionality has been moved into internal `BatchedElasticsearchSink` class that inherits from `IBatchedLogEventSink`, so it complies with new recommended way of integration with `PeriodicBatchingSink` and we don't use obsolete constructors.
+- `ConnectionStub` was moved out of `ElasticsearchSinkTestsBase` and extended. Both are now in `/Stubs` subfolder. Newer versions of Elasticsearch.NET client are now using "pre-flight" request to determine if endpoint is Elasticsearch and if it is indeed between 6.x and 8.x. `ConnectionStub` had to accommodate for that.
+- Unit tests have been fixed/added accordingly, running on multiple target frameworks (`net6`, `net7` and `net48`).
+- Built-in .NET SDK conditional compilation symbols are now used (e.g NETFRAMEWORK).
+
 ## [9.0.0] - 2022-04-29
 
 ### Fixed

README.md

@@ -30,6 +30,7 @@ The Serilog Elasticsearch sink project is a sink (basically a writer) for the Se
 * Starting from version 3, compatible with Elasticsearch 2.
 * Version 6.x supports the new Elasticsearch.net version 6.x library.
 * From version 8.x there is support for Elasticsearch.net version 7.
+* From version 9.x there is support for Elasticsearch.net version 8. Version detection is enabled by default, in which case `TypeName` is handled automatically across major versions 6, 7 and 8.
 
 
 ## Quick start
@@ -40,16 +41,80 @@ The Serilog Elasticsearch sink project is a sink (basically a writer) for the Se
 Install-Package serilog.sinks.elasticsearch
 ```
 
-Register the sink in code or using the appSettings reader (from v2.0.42+) as shown below. Make sure to specify the version of ES you are targeting. Be aware that the AutoRegisterTemplate option will not overwrite an existing template.
+Simplest way to register this sink is to use default configuration:
+
+```csharp
+var loggerConfig = new LoggerConfiguration()
+    .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200"));
+```
+
+Or, if using .NET Core and `Serilog.Settings.Configuration` Nuget package and `appsettings.json`, default configuration would look like this:
+
+```json
+{
+  "Serilog": {
+    "Using": [ "Serilog.Sinks.Elasticsearch" ],
+    "MinimumLevel": "Warning",
+    "WriteTo": [
+      {
+        "Name": "Elasticsearch",
+        "Args": {
+          "nodeUris": "http://localhost:9200"
+        }
+      }
+    ],
+    "Enrich": [ "FromLogContext", "WithMachineName" ],
+    "Properties": {
+      "Application": "ImmoValuation.Swv - Web"
+    }
+  }
+}
+```
+
+More elaborate configuration, using additional Nuget packages (e.g. `Serilog.Enrichers.Environment`) would look like:
+
+```json
+{
+  "Serilog": {
+    "Using": [ "Serilog.Sinks.Elasticsearch" ],
+    "MinimumLevel": "Warning",
+    "WriteTo": [
+      {
+        "Name": "Elasticsearch",
+        "Args": {
+          "nodeUris": "http://localhost:9200"
+        }
+      }
+    ],
+    "Enrich": [ "FromLogContext", "WithMachineName" ],
+    "Properties": {
+      "Application": "My app"
+    }
+  }
+}
+```
+
+This way the sink will detect version of Elasticsearch server (`DetectElasticsearchVersion` is set to `true` by default) and handle `TypeName` behavior correctly, based on the server version (6.x, 7.x or 8.x).
+
+### Disable detection of Elasticsearch server version
+
+Alternatively, `DetectElasticsearchVersion` can be set to `false` and certain option can be configured manually. In that case, the sink will assume version 7 of Elasticsearch, but options will be ignored due to a potential version incompatibility.
+
+For example, you can configure the sink to force registeration of v6 index template. Be aware that the AutoRegisterTemplate option will not overwrite an existing template.
 
 ```csharp
 var loggerConfig = new LoggerConfiguration()
     .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200") ){
+             DetectElasticsearchVersion = false,
              AutoRegisterTemplate = true,
              AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv6
      });
 ```
 
+### Configurable properties
+
+Besides a registration of the sink in the code, it is possible to register it using appSettings reader (from v2.0.42+) reader (from v2.0.42+) as shown below.
+
 This example shows the options that are currently available when using the appSettings reader.
 
 ```xml
@@ -75,7 +140,8 @@ This example shows the options that are currently available when using the appSe
     <add key="serilog:write-to:Elasticsearch.emitEventFailure" value="WriteToSelfLog" />
     <add key="serilog:write-to:Elasticsearch.queueSizeLimit" value="100000" />
     <add key="serilog:write-to:Elasticsearch.autoRegisterTemplate" value="true" />
-    <add key="serilog:write-to:Elasticsearch.autoRegisterTemplateVersion" value="ESv2" />
+    <add key="serilog:write-to:Elasticsearch.autoRegisterTemplateVersion" value="ESv7" />
+    <add key="serilog:write-to:Elasticsearch.detectElasticsearchVersion" value="false" /><!-- `true` is default -->
     <add key="serilog:write-to:Elasticsearch.overwriteTemplate" value="false" />
     <add key="serilog:write-to:Elasticsearch.registerTemplateFailure" value="IndexAnyway" />
     <add key="serilog:write-to:Elasticsearch.deadLetterIndexName" value="deadletter-{0:yyyy.MM}" />

sample/Serilog.Sinks.Elasticsearch.Sample/Serilog.Sinks.Elasticsearch.Sample.csproj

@@ -2,7 +2,7 @@
 
   <PropertyGroup>
     <OutputType>Exe</OutputType>
-    <TargetFramework>netcoreapp3.1</TargetFramework>
+    <TargetFramework>net6.0</TargetFramework>
     <IsPackable>false</IsPackable>
   </PropertyGroup>
 
@@ -11,8 +11,8 @@
     <PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="6.0.1" />
     <PackageReference Include="Microsoft.Extensions.Configuration.FileExtensions" Version="6.0.0" />
     <PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="6.0.0" />
-    <PackageReference Include="Serilog" Version="2.11.0" />
-    <PackageReference Include="Serilog.Sinks.Console" Version="4.0.1" />
+    <PackageReference Include="Serilog" Version="2.12.0" />
+    <PackageReference Include="Serilog.Sinks.Console" Version="4.1.0" />
   </ItemGroup>
 
   <ItemGroup>

src/Serilog.Formatting.Elasticsearch/Serilog.Formatting.Elasticsearch.csproj

@@ -28,7 +28,7 @@
   </PropertyGroup>
 
   <ItemGroup>
-    <PackageReference Include="Serilog" Version="2.11.0" />
+    <PackageReference Include="Serilog" Version="2.12.0" />
   </ItemGroup>
 
   <ItemGroup>

src/Serilog.Sinks.Elasticsearch/LoggerConfigurationElasticSearchExtensions.cs

@@ -143,6 +143,7 @@ public static LoggerConfiguration Elasticsearch(
         /// <param name="failureSink">Sink to use when Elasticsearch is unable to accept the events. This is optionally and depends on the EmitEventFailure setting.</param>   
         /// <param name="singleEventSizePostingLimit"><see cref="ElasticsearchSinkOptions.SingleEventSizePostingLimit"/>The maximum length of an event allowed to be posted to Elasticsearch.default null</param>
         /// <param name="templateCustomSettings">Add custom elasticsearch settings to the template</param>
+        /// <param name="detectElasticsearchVersion">Turns on detection of elasticsearch version via background HTTP call. This will also set `TypeName` automatically, according to the version of Elasticsearch.</param>
         /// <param name="batchAction">Configures the OpType being used when inserting document in batch. Must be set to create for data streams.</param>
         /// <returns>LoggerConfiguration object</returns>
         /// <exception cref="ArgumentNullException"><paramref name="nodeUris"/> is <see langword="null" />.</exception>
@@ -151,7 +152,7 @@ public static LoggerConfiguration Elasticsearch(
             string nodeUris,
             string indexFormat = null,
             string templateName = null,
-            string typeName = "logevent",
+            string typeName = null,
             int batchPostingLimit = 50,
             int period = 2,
             bool inlineFields = false,
@@ -166,7 +167,7 @@ public static LoggerConfiguration Elasticsearch(
             int queueSizeLimit = 100000,
             string pipelineName = null,
             bool autoRegisterTemplate = false,
-            AutoRegisterTemplateVersion autoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv2,
+            AutoRegisterTemplateVersion? autoRegisterTemplateVersion = null,
             bool overwriteTemplate = false,
             RegisterTemplateRecovery registerTemplateFailure = RegisterTemplateRecovery.IndexAnyway,
             string deadLetterIndexName = null,
@@ -182,7 +183,8 @@ public static LoggerConfiguration Elasticsearch(
             long? singleEventSizePostingLimit = null,
             int? bufferFileCountLimit = null,
             Dictionary<string,string> templateCustomSettings = null,
-            ElasticOpType batchAction = ElasticOpType.Index)
+            ElasticOpType batchAction = ElasticOpType.Index,
+            bool detectElasticsearchVersion = true)
         {
             if (string.IsNullOrEmpty(nodeUris))
                 throw new ArgumentNullException(nameof(nodeUris), "No Elasticsearch node(s) specified.");
@@ -204,8 +206,6 @@ public static LoggerConfiguration Elasticsearch(
                 options.TemplateName = templateName;
             }
 
-            options.TypeName = !string.IsNullOrWhiteSpace(typeName) ? typeName : null;
-
             options.BatchPostingLimit = batchPostingLimit;
             options.BatchAction = batchAction;
             options.SingleEventSizePostingLimit = singleEventSizePostingLimit;
@@ -272,6 +272,8 @@ public static LoggerConfiguration Elasticsearch(
 
             options.TemplateCustomSettings = templateCustomSettings;
 
+            options.DetectElasticsearchVersion = detectElasticsearchVersion;
+
             return Elasticsearch(loggerSinkConfiguration, options);
         }
     }

src/Serilog.Sinks.Elasticsearch/Properties/AssemblyInfo.cs

@@ -5,7 +5,7 @@
 [assembly: AssemblyDescription("Serilog sink for Elasticsearch")]
 [assembly: AssemblyCopyright("Copyright © Serilog Contributors 2018")]
 
-[assembly: InternalsVisibleTo("Serilog.Tests, PublicKey=0024000004800000940000000602000000240000525341310004000001000100fb8d13fd344a1c" +
+[assembly: InternalsVisibleTo("Serilog.Sinks.Elasticsearch.Tests, PublicKey=0024000004800000940000000602000000240000525341310004000001000100fb8d13fd344a1c" +
                                                        "6fe0fe83ef33c1080bf30690765bc6eb0df26ebfdf8f21670c64265b30db09f73a0dea5b3db4c9" +
                                                        "d18dbf6d5a25af5ce9016f281014d79dc3b4201ac646c451830fc7e61a2dfd633d34c39f87b818" +
                                                        "94191652df5ac63cc40c77f3542f702bda692e6e8a9158353df189007a49da0f3cfd55eb250066" +

src/Serilog.Sinks.Elasticsearch/Serilog.Sinks.Elasticsearch.csproj

@@ -4,6 +4,7 @@
     <Authors>Michiel van Oudheusden, Martijn Laarman, Mogens Heller Grabe, Serilog Contributors</Authors>
     <TargetFrameworks>netstandard2.0</TargetFrameworks>
     <TreatWarningsAsErrors>true</TreatWarningsAsErrors>
+    <LangVersion>latest</LangVersion>
     <GenerateDocumentationFile>true</GenerateDocumentationFile>
     <AssemblyName>Serilog.Sinks.Elasticsearch</AssemblyName>
     <AssemblyOriginatorKeyFile>../../assets/Serilog.snk</AssemblyOriginatorKeyFile>
@@ -35,11 +36,11 @@
 
     <ProjectReference Include="..\Serilog.Formatting.Elasticsearch\Serilog.Formatting.Elasticsearch.csproj" />
 
-    <PackageReference Include="Serilog" Version="2.11.0" />
+    <PackageReference Include="Serilog" Version="2.12.0" />
     <PackageReference Include="Serilog.Formatting.Compact" Version="1.1.0" />
     <PackageReference Include="Serilog.Sinks.File" Version="5.0.0" />
-    <PackageReference Include="Serilog.Sinks.PeriodicBatching" Version="2.1.1" />
-    <PackageReference Include="Elasticsearch.Net" Version="7.8.1" />
+    <PackageReference Include="Serilog.Sinks.PeriodicBatching" Version="3.1.0" />
+    <PackageReference Include="Elasticsearch.Net" Version="7.17.5" />
     <PackageReference Include="System.Diagnostics.DiagnosticSource" Version="6.0.0" />
   </ItemGroup>
 

src/Serilog.Sinks.Elasticsearch/Sinks/ElasticSearch/Durable/Elasticsearch/ElasticsearchLogClient.cs

@@ -86,7 +86,7 @@ private InvalidResult GetInvalidPayloadAsync(DynamicResponse baseResult, List<st
             bool hasErrors = false;
             foreach (dynamic item in items)
             {
-                var itemIndex = item?[ElasticsearchSink.BulkAction(_elasticOpType)];
+                var itemIndex = item?[BatchedElasticsearchSink.BulkAction(_elasticOpType)];
                 long? status = itemIndex?["status"];
                 i++;
                 if (!status.HasValue || status < 300)

src/Serilog.Sinks.Elasticsearch/Sinks/ElasticSearch/Durable/Elasticsearch/ElasticsearchPayloadReader.cs

@@ -92,7 +92,7 @@ protected override List<string> FinishPayLoad()
         protected override void AddToPayLoad(string nextLine)
         {
             var indexName = _getIndexForEvent(nextLine, _date);
-            var action = ElasticsearchSink.CreateElasticAction(
+            var action = BatchedElasticsearchSink.CreateElasticAction(
                 opType: _elasticOpType, 
                 indexName: indexName, pipelineName: _pipelineName,
                 id: _count + "_" + Guid.NewGuid(),

src/Serilog.Sinks.Elasticsearch/Sinks/ElasticSearch/ElasticSearchSink.cs

@@ -25,26 +25,39 @@
 
 namespace Serilog.Sinks.Elasticsearch
 {
+    public sealed class ElasticsearchSink : PeriodicBatchingSink
+    {
+        public ElasticsearchSink(ElasticsearchSinkOptions options)
+            : this(
+                  new BatchedElasticsearchSink(options),
+                  new PeriodicBatchingSinkOptions
+                    {
+                        BatchSizeLimit = options.BatchPostingLimit,
+                        Period = options.Period,
+                        EagerlyEmitFirstEvent = true,
+                        QueueLimit = (options.QueueSizeLimit == -1) ? null : new int?(options.QueueSizeLimit)
+                    }
+                  )
+        {
+        }
+
+        private ElasticsearchSink(IBatchedLogEventSink batchedSink, PeriodicBatchingSinkOptions options)
+            : base(batchedSink, options)
+        {
+        }
+    }
     /// <summary>
     /// Writes log events as documents to ElasticSearch.
     /// </summary>
-    public class ElasticsearchSink : PeriodicBatchingSink
+    internal sealed class BatchedElasticsearchSink : IBatchedLogEventSink
     {
-
         private readonly ElasticsearchSinkState _state;
 
-        /// <summary>
-        /// Creates a new ElasticsearchSink instance with the provided options
-        /// </summary>
-        /// <param name="options">Options configuring how the sink behaves, may NOT be null</param>
-        public ElasticsearchSink(ElasticsearchSinkOptions options)
-            : base(options.BatchPostingLimit, options.Period, options.QueueSizeLimit)
+        public BatchedElasticsearchSink(ElasticsearchSinkOptions options)
         {
             _state = ElasticsearchSinkState.Create(options);
-            _state.DiscoverClusterVersion();
             _state.RegisterTemplateIfNeeded();
         }
-
         /// <summary>
         /// Emit a batch of log events, running to completion synchronously.
         /// </summary>
@@ -54,7 +67,7 @@ public ElasticsearchSink(ElasticsearchSinkOptions options)
         ///  or <see cref="M:Serilog.Sinks.PeriodicBatching.PeriodicBatchingSink.EmitBatchAsync(System.Collections.Generic.IEnumerable{Serilog.Events.LogEvent})" />,
         /// not both.
         /// </remarks>
-        protected override async Task EmitBatchAsync(IEnumerable<LogEvent> events)
+        public async Task EmitBatchAsync(IEnumerable<LogEvent> events)
         {
             DynamicResponse result;
 
@@ -71,42 +84,32 @@ protected override async Task EmitBatchAsync(IEnumerable<LogEvent> events)
             HandleResponse(events, result);
         }
 
-        /// <summary>
-        /// Emit a batch of log events, running to completion synchronously.
-        /// </summary>
-        /// <param name="events">The events to emit.</param>
-        /// <returns>Response from Elasticsearch</returns>
-        protected virtual Task<T> EmitBatchCheckedAsync<T>(IEnumerable<LogEvent> events) where T : class, IElasticsearchResponse, new()
+        public Task OnEmptyBatchAsync()
         {
-            // ReSharper disable PossibleMultipleEnumeration
-            if (events == null || !events.Any())
-                return Task.FromResult<T>(default(T));
-
-            var payload = CreatePayload(events);
-            return _state.Client.BulkAsync<T>(PostData.MultiJson(payload));
+            return Task.CompletedTask;
         }
 
         /// <summary>
         /// Emit a batch of log events, running to completion synchronously.
         /// </summary>
         /// <param name="events">The events to emit.</param>
         /// <returns>Response from Elasticsearch</returns>
-        protected virtual T EmitBatchChecked<T>(IEnumerable<LogEvent> events) where T : class, IElasticsearchResponse, new()
+        private Task<T> EmitBatchCheckedAsync<T>(IEnumerable<LogEvent> events) where T : class, IElasticsearchResponse, new()
         {
             // ReSharper disable PossibleMultipleEnumeration
             if (events == null || !events.Any())
-                return null;
+                return Task.FromResult<T>(default(T));
 
             var payload = CreatePayload(events);
-            return _state.Client.Bulk<T>(PostData.MultiJson(payload));
+            return _state.Client.BulkAsync<T>(PostData.MultiJson(payload));
         }
 
         /// <summary>
         /// Handles the exceptions.
         /// </summary>
         /// <param name="ex"></param>
         /// <param name="events"></param>
-        protected virtual void HandleException(Exception ex, IEnumerable<LogEvent> events)
+        private void HandleException(Exception ex, IEnumerable<LogEvent> events)
         {
             if (_state.Options.EmitEventFailure.HasFlag(EmitEventFailureHandling.WriteToSelfLog))
             {
@@ -153,18 +156,6 @@ protected virtual void HandleException(Exception ex, IEnumerable<LogEvent> event
                 throw ex;
         }
 
-        // Helper function: checks if a given dynamic member / dictionary key exists at runtime
-        private static bool HasProperty(dynamic settings, string name)
-        {
-            if (settings is IDictionary<string, object>)
-                return ((IDictionary<string, object>)settings).ContainsKey(name);
-
-            if (settings is System.Dynamic.DynamicObject)
-                return ((System.Dynamic.DynamicObject)settings).GetDynamicMemberNames().Contains(name);
-
-            return settings.GetType().GetProperty(name) != null;
-        }
-
         private IEnumerable<string> CreatePayload(IEnumerable<LogEvent> events)
         {
             if (!_state.TemplateRegistrationSuccess && _state.Options.RegisterTemplateFailure == RegisterTemplateRecovery.FailSink)
@@ -280,7 +271,7 @@ internal static object CreateElasticAction(ElasticOpType opType, string indexNam
                 : new ElasticIndexAction(actionPayload);
             return action;
         }
-        
+
         sealed class ElasticCreateAction
         {
             public ElasticCreateAction(ElasticActionPayload payload)