Any news on when could this be implemented?

You could also try to use the detect-exceptions plugin mentioned in #476

https://github.com/GoogleCloudPlatform/fluent-plugin-detect-exceptions

Edit: ahh... damn... the fluentd plugins don't seem compatible :-(

It seems this issue is not being addressed. This is a showstopper for me, so I guess it's back to plain old fluentd...

@gavrie Can we handle the same issue in fluentd?

fluentd already has an existing Loggly plugin that handles this correctly.

On Mon, 25 Jun 2018 at 8:39 abhishek notifications@github.com wrote:

@gavrie https://github.com/gavrie Can we handle the same issue in fluentd?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fluent/fluent-bit/issues/337#issuecomment-399837526, or mute the thread https://github.com/notifications/unsubscribe-auth/AARnjPQlVN21qTk_Sie8FWFIPv5gnBelks5uAHejgaJpZM4OjVIY .

@gavrie This is a paid plugin rgt? are there any open source plugins that does the same?

@abhishek the plugin is Open Source (Apache license):

https://github.com/patant/fluent-plugin-loggly On Wed, 27 Jun 2018 at 17:11 abhishek notifications@github.com wrote:

@gavrie https://github.com/gavrie This https://www.loggly.com/docs/fluentd-logs/ is a paid plugin rgt? are there any open source plugins that does the same?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fluent/fluent-bit/issues/337#issuecomment-400685785, or mute the thread https://github.com/notifications/unsubscribe-auth/AARnjK3E_g9QTUyhdjDYYc5znLJN_F-uks5uA5KPgaJpZM4OjVIY .

@gavrie thanks .

@edsiper just a small request for information - is this something which is being developed now (or soon)? if yes I could wait, if not I would have to look for alternatives. I think this use case is rather straightforward, so my (maybe naive ;) hope is that it is at least on the map.

This is an issue for us as well. I spun up a fluent-bit daemonset and was happy with the performance and footprint, but I have not been able to figure out a workaround for the issue with multiline logs. It's not difficult to create a parser for the additional lines which drops the docker cruft and captures the message content. The problem is that the named field gets added to the record for each line, which creates a json record with duplicate keys. One idea comes to mind which would solve this problem neatly: an option to allow the input plugin to append to an existing field when it encounters a new field of the same name. For now we're going back to fluentd because the detect_exceptions plugin helps with some of these cases.

@Markbnj I think, for the time being, Fluent Bit is just an experiment, a POC, as it lacks rudimentary features and cannot be used in the real world.

Same issue for me. The cause of this is default docker logging-driver of json-file.

@nikolay Just FYI we're using it in the real world, as the leaf node collector for over 300 GCP instances, logging over 150 million events per day. The issues on K8S are due to docker's pretty silly logging format, which annotates each line with cruft that ranges from useless to semi-useless. Dealing with that format in a multiline logging scenario is probably beyond fluent-bit's charter, but unfortunately there are not really any easier places to deal with it when you're running on a hosted cluster.

@markbnj Fluentd doesn't have this problem with Docker logs, so why does fluentbit? As "silly" or "useless" as its output may be, this appears to be a solved problem; but not with fluent-bit

@rawkode Actually fluentd has pretty much the same issue, except that it has the detect exceptions plugin, which does a pretty good job of detecting multiline exceptions. It doesn't handle all cases of multi-line container logs however.

So if I use a different CRI implementation, this problem goes away?

Has anyone ported the plugin to fluentbit?

@rawkode good question... I don't have experience actually running anything on cri0, but from looking at google's fluentd config for stackdriver logging it seems like you could also expect per line additions of at least timestamp and implied severity (based on stream) so probably the same issue in a slightly different format.

@Markbnj @nikolay same here, real world deployment. We are using fluent-bit as k8s node collector across various environments at a similar scale and event volume

@Markbnj Look, your seemingly big numbers are meaningless when even their example setup does not work on a single node. It kinda works for you with tons of hacks and compromises, but Fluent Bit, unlike Fluentd, is targeting Kubernetes, and, yet, it is totally defunct with it. So, if Fluentd needs a plugin - it's understandable and acceptable, but Fluent Bit needs this basic use case out of the box without the requirement for a plugin from the future... as there's no such plugin at this point. So, I repeat what I said - Fluent Bit is possibly the future, but definitely not the present! At this point, it's just a POC, which hopefully will be shaped to something workable around v1.0... but it's still just a v0.13. My point was if I wasn't clear that it needs a big warning sign so that people don't spin wheels!

@michiel As explained, the number of nodes is irrelevant when even the "hello world" equivalent fails with a single node! Provide versions of Fluent Bit and types of apps running in the cluster, which would be something substation other than just bragging!

@nikolay our numbers aren't really big. I was just giving you a data point to consider. It doesn't seem accurate to me to suggest that fluent-bit is "targeting kubernetes" and is thus insufficient for its primary use case, although the authors can address that better than I can. Kubernetes is mentioned on one line of the readme, in the filters section. In other words, it is one potential source of logs that fluent-bit can be used to collect.

@Markbnj That's what Eduardo said himself during KubeCon 2017, which I attended.

IMHO pretty much the whole discussion is pointless. I really don't care if fluentbit is production or not, 0.x or not, supercool or not - it's useful to me. And getting this fixed makes it even more useful to me. what more is there to say? why even bother "warning" people who are happy with their choice so far?!

so if @nikolay wants to jump in here and troll an opinion, I personally choose to ignore him because I don't see him contributing anything remotely useful, just some strongly worded opinion about which label to attach to fluentbit, which does not help me at all and I franky don't care about that.

EDIT: changed subject ;) - I only speak for myself.

Our focus is cloud native in general which includes Kubernetes, and yes, this is a missing feature. It's good to have different opinions.

The ticket continue being an enhancement request; if this missing feature is a blocker for your environment you should go with Fluentd instead.

+1 for this feature. my use case is to collect logs for Spinnaker running in Kubernetes. as majority of the Spinnaker services are written in java and I do have the needs to parse multiple lines of java exception and feed them into ES to trigger some follow-up actions.

and I like fluent-bit over fluentd as well :)

thanks.

@breeze7086 , does your comment mean, that the problem won't exist with any other logging driver from docker? I guess it definitely won't exist with the fluentd driver?

@rogerstorm funded this issue with $50. Visit this issue on Issuehunt

@loadbalance-sudachi-kun funded this issue with $256. Visit this issue on Issuehunt

@edsiper, any news on this?

@edsiper filebeat 6.x can deals with multiline from nested JSON strings，https://www.elastic.co/guide/en/beats/filebeat/6.x/filebeat-input-docker.html , is it possible to implement the operation refer to the above content. Thanks

note the are two different cases associated with multiline in Docker logs:

1. Docker split long lines in multiple lines.
2. Application generate "multiline logs" and Docker put each one as separate entities.

Case #1 is already solved by the new Docker mode of in_tail plugin. Case #2 is still a missing feature.

I would like to help on this when I have free time the next week. How is the input and the expected output? Is there some guide to configure a testing docker images in order to test the log outputs?

@ioprotium you can run any java application and throw an error in it it will generate multi line

If you want i can provide a docker image for simulation

{"log":"2019-01-18 07:46:22.634 [               ]  INFO 1 --- [vent-bus.prod-1] c.t.listener.CommonListener              : warehousing Dailywarehousing.daily\n","stream":"stdout","time":"2019-01-18T07:46:22.63525751Z"}
{"log":"2019-01-18 07:49:27.370 [ali            ] ERROR 1 --- [nio-8083-exec-7] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is java.lang.Exception: This is a test exception] with root cause\n","stream":"stdout","time":"2019-01-18T07:49:27.373656664Z"}
{"log":"java.lang.Exception: This is a test exception\n","stream":"stdout","time":"2019-01-18T07:49:27.373776864Z"}
{"log":"\u0009at com.xxxxxxxx.controller.ExceptionTestController.exceptionTest(ExceptionTestController.java:15)\n","stream":"stdout","time":"2019-01-18T07:49:27.373799011Z"}
{"log":"\u0009at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n","stream":"stdout","time":"2019-01-18T07:49:27.373815758Z"}
{"log":"\u0009at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n","stream":"stdout","time":"2019-01-18T07:49:27.373831075Z"}
{"log":"\u0009at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n","stream":"stdout","time":"2019-01-18T07:49:27.373847381Z"}
{"log":"\u0009at java.lang.reflect.Method.invoke(Method.java:498)\n","stream":"stdout","time":"2019-01-18T07:49:27.373862835Z"}
{"log":"\u0009at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)\n","stream":"stdout","time":"2019-01-18T07:49:27.373877918Z"}
{"log":"\u0009at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)\n","stream":"stdout","time":"2019-01-18T07:49:27.373893255Z"}
{"log":"\u0009at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)\n","stream":"stdout","time":"2019-01-18T07:49:27.373933561Z"}
{"log":"\u0009at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:891)\n","stream":"stdout","time":"2019-01-18T07:49:27.373950281Z"}
{"log":"\u0009at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797)\n","stream":"stdout","time":"2019-01-18T07:49:27.373966221Z"}
{"log":"\u0009at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)\n","stream":"stdout","time":"2019-01-18T07:49:27.373981291Z"}
{"log":"\u0009at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991)\n","stream":"stdout","time":"2019-01-18T07:49:27.373995698Z"}
{"log":"\u0009at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925)\n","stream":"stdout","time":"2019-01-18T07:49:27.374009749Z"}
{"log":"\u0009at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:981)\n","stream":"stdout","time":"2019-01-18T07:49:27.374023972Z"}
{"log":"\u0009at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:873)\n","stream":"stdout","time":"2019-01-18T07:49:27.374038132Z"}
{"log":"\u0009at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)\n","stream":"stdout","time":"2019-01-18T07:49:27.374052439Z"}
{"log":"\u0009at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:858)\n","stream":"stdout","time":"2019-01-18T07:49:27.374069639Z"}
{"log":"\u0009at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)\n","stream":"stdout","time":"2019-01-18T07:49:27.374084189Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)\n","stream":"stdout","time":"2019-01-18T07:49:27.374097982Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n","stream":"stdout","time":"2019-01-18T07:49:27.374111936Z"}
{"log":"\u0009at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)\n","stream":"stdout","time":"2019-01-18T07:49:27.374125656Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n","stream":"stdout","time":"2019-01-18T07:49:27.374139159Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n","stream":"stdout","time":"2019-01-18T07:49:27.374152839Z"}
{"log":"\u0009at com.xxxxxxxx.filter.LoggingFilter.doFilter(LoggingFilter.java:27)\n","stream":"stdout","time":"2019-01-18T07:49:27.374166452Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n","stream":"stdout","time":"2019-01-18T07:49:27.374180012Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n","stream":"stdout","time":"2019-01-18T07:49:27.374193836Z"}
{"log":"\u0009at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:90)\n","stream":"stdout","time":"2019-01-18T07:49:27.374207526Z"}
{"log":"\u0009at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n","stream":"stdout","time":"2019-01-18T07:49:27.374221337Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n","stream":"stdout","time":"2019-01-18T07:49:27.374234977Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n","stream":"stdout","time":"2019-01-18T07:49:27.37424904Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)\n","stream":"stdout","time":"2019-01-18T07:49:27.374295243Z"}
{"log":"\u0009at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)\n","stream":"stdout","time":"2019-01-18T07:49:27.374329427Z"}
{"log":"\u0009at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)\n","stream":"stdout","time":"2019-01-18T07:49:27.37434519Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n","stream":"stdout","time":"2019-01-18T07:49:27.37435971Z"}
{"log":"\u0009at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)\n","stream":"stdout","time":"2019-01-18T07:49:27.37437389Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n","stream":"stdout","time":"2019-01-18T07:49:27.374388197Z"}
{"log":"\u0009at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)\n","stream":"stdout","time":"2019-01-18T07:49:27.374402333Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n","stream":"stdout","time":"2019-01-18T07:49:27.374418393Z"}
{"log":"\u0009at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)\n","stream":"stdout","time":"2019-01-18T07:49:27.374432798Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n","stream":"stdout","time":"2019-01-18T07:49:27.374447341Z"}
{"log":"\u0009at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)\n","stream":"stdout","time":"2019-01-18T07:49:27.374461598Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n","stream":"stdout","time":"2019-01-18T07:49:27.374476194Z"}
{"log":"\u0009at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)\n","stream":"stdout","time":"2019-01-18T07:49:27.374490461Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n","stream":"stdout","time":"2019-01-18T07:49:27.374504388Z"}
{"log":"\u0009at org.springframework.security.oauth2.provider.authentication.OAuth2AuthenticationProcessingFilter.doFilter(OAuth2AuthenticationProcessingFilter.java:176)\n","stream":"stdout","time":"2019-01-18T07:49:27.374518434Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n","stream":"stdout","time":"2019-01-18T07:49:27.374533138Z"}
{"log":"\u0009at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)\n","stream":"stdout","time":"2019-01-18T07:49:27.374546824Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n","stream":"stdout","time":"2019-01-18T07:49:27.374560551Z"}
{"log":"\u0009at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:74)\n","stream":"stdout","time":"2019-01-18T07:49:27.374574684Z"}
{"log":"\u0009at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n","stream":"stdout","time":"2019-01-18T07:49:27.374588648Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n","stream":"stdout","time":"2019-01-18T07:49:27.374602644Z"}
{"log":"\u0009at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)\n","stream":"stdout","time":"2019-01-18T07:49:27.374616758Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n","stream":"stdout","time":"2019-01-18T07:49:27.374640288Z"}
{"log":"\u0009at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)\n","stream":"stdout","time":"2019-01-18T07:49:27.374655505Z"}
{"log":"\u0009at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n","stream":"stdout","time":"2019-01-18T07:49:27.374671955Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)\n","stream":"stdout","time":"2019-01-18T07:49:27.374690312Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215)\n","stream":"stdout","time":"2019-01-18T07:49:27.374704522Z"}
{"log":"\u0009at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)\n","stream":"stdout","time":"2019-01-18T07:49:27.374718459Z"}
{"log":"\u0009at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)\n","stream":"stdout","time":"2019-01-18T07:49:27.374732919Z"}
{"log":"\u0009at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)\n","stream":"stdout","time":"2019-01-18T07:49:27.374750799Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n","stream":"stdout","time":"2019-01-18T07:49:27.374764819Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n","stream":"stdout","time":"2019-01-18T07:49:27.374778682Z"}
{"log":"\u0009at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)\n","stream":"stdout","time":"2019-01-18T07:49:27.374792429Z"}
{"log":"\u0009at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n","stream":"stdout","time":"2019-01-18T07:49:27.374805985Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n","stream":"stdout","time":"2019-01-18T07:49:27.374819625Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n","stream":"stdout","time":"2019-01-18T07:49:27.374833335Z"}
{"log":"\u0009at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)\n","stream":"stdout","time":"2019-01-18T07:49:27.374847845Z"}
{"log":"\u0009at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n","stream":"stdout","time":"2019-01-18T07:49:27.374861925Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n","stream":"stdout","time":"2019-01-18T07:49:27.37487589Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n","stream":"stdout","time":"2019-01-18T07:49:27.374890043Z"}
{"log":"\u0009at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)\n","stream":"stdout","time":"2019-01-18T07:49:27.374903813Z"}
{"log":"\u0009at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n","stream":"stdout","time":"2019-01-18T07:49:27.374917793Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n","stream":"stdout","time":"2019-01-18T07:49:27.374931586Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n","stream":"stdout","time":"2019-01-18T07:49:27.374946006Z"}
{"log":"\u0009at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:117)\n","stream":"stdout","time":"2019-01-18T07:49:27.37496104Z"}
{"log":"\u0009at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:106)\n","stream":"stdout","time":"2019-01-18T07:49:27.37498773Z"}
{"log":"\u0009at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n","stream":"stdout","time":"2019-01-18T07:49:27.375003113Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n","stream":"stdout","time":"2019-01-18T07:49:27.375017063Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n","stream":"stdout","time":"2019-01-18T07:49:27.37503086Z"}
{"log":"\u0009at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)\n","stream":"stdout","time":"2019-01-18T07:49:27.3750454Z"}
{"log":"\u0009at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)\n","stream":"stdout","time":"2019-01-18T07:49:27.37505928Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n","stream":"stdout","time":"2019-01-18T07:49:27.37507306Z"}
{"log":"\u0009at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n","stream":"stdout","time":"2019-01-18T07:49:27.375086726Z"}
{"log":"\u0009at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)\n","stream":"stdout","time":"2019-01-18T07:49:27.375100817Z"}
{"log":"\u0009at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)\n","stream":"stdout","time":"2019-01-18T07:49:27.375115354Z"}
{"log":"\u0009at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)\n","stream":"stdout","time":"2019-01-18T07:49:27.375129454Z"}
{"log":"\u0009at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)\n","stream":"stdout","time":"2019-01-18T07:49:27.375144001Z"}
{"log":"\u0009at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)\n","stream":"stdout","time":"2019-01-18T07:49:27.375157464Z"}
{"log":"\u0009at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)\n","stream":"stdout","time":"2019-01-18T07:49:27.375170981Z"}
{"log":"\u0009at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)\n","stream":"stdout","time":"2019-01-18T07:49:27.375184417Z"}
{"log":"\u0009at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)\n","stream":"stdout","time":"2019-01-18T07:49:27.375198024Z"}
{"log":"\u0009at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)\n","stream":"stdout","time":"2019-01-18T07:49:27.375211594Z"}
{"log":"\u0009at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)\n","stream":"stdout","time":"2019-01-18T07:49:27.375225237Z"}
{"log":"\u0009at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)\n","stream":"stdout","time":"2019-01-18T07:49:27.375239487Z"}
{"log":"\u0009at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)\n","stream":"stdout","time":"2019-01-18T07:49:27.375253464Z"}
{"log":"\u0009at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n","stream":"stdout","time":"2019-01-18T07:49:27.375323255Z"}
{"log":"\u0009at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n","stream":"stdout","time":"2019-01-18T07:49:27.375345642Z"}
{"log":"\u0009at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n","stream":"stdout","time":"2019-01-18T07:49:27.375363208Z"}
{"log":"\u0009at java.lang.Thread.run(Thread.java:748)\n","stream":"stdout","time":"2019-01-18T07:49:27.375377695Z"}
{"log":"\n","stream":"stdout","time":"2019-01-18T07:49:27.375391335Z"}
{"log":"\n","stream":"stdout","time":"2019-01-18T07:49:27.375416915Z"}
{"log":"2019-01-18 07:53:06.419 [               ]  INFO 1 --- [vent-bus.prod-1] c.t.listener.CommonListener              : warehousing Dailywarehousing.daily\n","stream":"stdout","time":"2019-01-18T07:53:06.420527437Z"}

i did create a simple image that will produce multiline every 10 second , please use shahbour/java-error:0.0.1

@shahbour Great. So the idea is that it should generate only one log for multiline messages like a java error with its call stack, right?

Yes exactly

On Fri, Jan 18, 2019, 7:00 PM Brian Mayo <notifications@github.com wrote:

@shahbour https://github.com/shahbour Great. So the idea is that it should generate only one log for multiline messages like a java error with its call stack, right?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fluent/fluent-bit/issues/337#issuecomment-455616352, or mute the thread https://github.com/notifications/unsubscribe-auth/ABzk-EVPQUrK3UdNWTfy2_wB5s_YtZ5zks5vEf3FgaJpZM4OjVIY .

We really liked to switch to fluent-bit but as we have many Java Applications this is holding us back...

@ioprotium did you have time to work on this , if you need any testing i can help in that

+1! This would make our life so much easier!

Ideally, if we could have docker mode and multiline work together, and if both are enabled: all parsers defined for the multiline should be applied against the value of the field named after Key rather than the whole event.

Meanwhile, our work-around was to:

• use Parser_Firstline to match multi-line logs together based on a simple regex (eg: if the payload starts with a whitespace char, it should be appended to the previous line)
• the side effect is that any lines that are not matching, are appended to the payload of the field named after Key, but in their docker format
• use a lua filter to parse these lines and re-construct the content of the field named after Key

It does the job but we have to deal with a whole bunch of things that are already handled by fluent-bit in other places (eg: string escape, utf encoding etc...)

Super interested in how you implemetend the above @stang - would you be ok to share the lua filter config and code perhaps? I'm running into the same issue and have been struggling loads with fluent-bit and multi-line and splunk!

Our application generate multiline logs and we also have this issue. Any update for this feature?

Hi @kiich,

Super interested in how you implemetend the above @stang - would you be ok to share the lua filter config and code perhaps?

Here you go:

• gist of the helpers.lua file (called from your lua filter in fluent-bit configuration)
• gist of the JSON.lua file which a slightly modified version of a lua JSON library (original code is linked so you can see what we added)
• and hereafter, an extract of our fluent-bit configuration:

[INPUT]
    Name             tail
    Path             /var/log/containers/*.log
    Parser           docker
    Tag              kube.*
    Refresh_Interval 5
    Mem_Buf_Limit    5MB
    Skip_Long_Lines  On
    DB               /tail-db/tail-containers-state.db
    DB.Sync          Normal
    Ignore_Older 2d
    Multiline On
    Multiline_Flush 5
    Parser_Firstline first_line
...
[FILTER]
    Name lua
    Match kube.*
    script /fluent-bit/etc/helpers.lua
    call process
...
[PARSER]
    Name        first_line
    Format      regex
    Regex       ^{"log":"(?!\\u0009)(?<log>\S(?:(\\")|[^"]){9}(?:(\\")|[^"])*)"

You might want to fine tune the Regex of the parser for your specific use case.

I'm running into the same issue and have been struggling loads with fluent-bit and multi-line and splunk!

In our case, fluent-bit was the only component used to collect and ship log straight to an elasticsearch instance and we didn't want to add more components to the stack, but you might be able to handle such a thing on the Splunk side.

@stang Awesome! thanks so much for this. We are still struggling with fluent-bit and how it handles multiline json logs - we'll try this out! thanks again.

@stang are you running this inside Kubernetes or stand alone ?

@stang are you running this inside Kubernetes or stand alone ?

In kubernetes.

It'd be awesome if that helper was present in the official fluent-bit Docker image by default. I'll be able to easily switch that on using the official Helm chart.

@stang are you running this inside Kubernetes or stand alone ?

In kubernetes.

I asked because i did not see any Kubernetes filter in the example above . Would you mind sharing the full daemonset yaml and configmap you used . Just to have full idea .

@loadbalance-sudachi-kun, @rogerstorm : I saw this issue on Issuehunt. Is this still relevant for you?

I just tested @stang solution and it worked perfectly for me , the only thing i had to change is regex to set first line , for me all lines should start with date so below is what i used

    [PARSER]
        Name        first_line
        Format      regex
        Regex       ^{"log":"(?<log>(?:[12]\d{3}-(?:0[1-9]|1[0-2])-(?:0[1-9]|[12]\d|3[01]))(?:(\\")|[^"]){9}(?:(\\")|[^"])*)"

@shahbour it is work , but the format so ..... , you know , anyway , thank you

@stang does this work with Fluentbit 1.2, where they fix the UTF8 decoding? Or do we need to add the utf8 decoders back?

any progress about this issue，dude

For mutliline in fluentd we use concat. We want to move to fluentbit fir performance reasons and multiline support is the only thing missing for us..please help :(

@stang @shahbour perfect. thanks!

Thanks for the solution, it did work for me while parsing java nultiline log files but for other log outputs it parse them wrong with unwonted encoding. In addition, it ignores the fluentbit.io/parser annotation

Is there a better way to solve it other then splitting the [INPUT]?

Hi @kiich,

Super interested in how you implemetend the above @stang - would you be ok to share the lua filter config and code perhaps?

Here you go:

• gist of the helpers.lua file (called from your lua filter in fluent-bit configuration)
• gist of the JSON.lua file which a slightly modified version of a lua JSON library (original code is linked so you can see what we added)
• and hereafter, an extract of our fluent-bit configuration:

[INPUT]
    Name             tail
    Path             /var/log/containers/*.log
    Parser           docker
    Tag              kube.*
    Refresh_Interval 5
    Mem_Buf_Limit    5MB
    Skip_Long_Lines  On
    DB               /tail-db/tail-containers-state.db
    DB.Sync          Normal
    Ignore_Older 2d
    Multiline On
    Multiline_Flush 5
    Parser_Firstline first_line
...
[FILTER]
    Name lua
    Match kube.*
    script /fluent-bit/etc/helpers.lua
    call process
...
[PARSER]
    Name        first_line
    Format      regex
    Regex       ^{"log":"(?!\\u0009)(?<log>\S(?:(\\")|[^"]){9}(?:(\\")|[^"])*)"

You might want to fine tune the Regex of the parser for your specific use case.

I'm running into the same issue and have been struggling loads with fluent-bit and multi-line and splunk!

In our case, fluent-bit was the only component used to collect and ship log straight to an elasticsearch instance and we didn't want to add more components to the stack, but you might be able to handle such a thing on the Splunk side.

Hi @kiich,

Super interested in how you implemetend the above @stang - would you be ok to share the lua filter config and code perhaps?

Here you go:

• gist of the helpers.lua file (called from your lua filter in fluent-bit configuration)
• gist of the JSON.lua file which a slightly modified version of a lua JSON library (original code is linked so you can see what we added)
• and hereafter, an extract of our fluent-bit configuration:

[INPUT]
    Name             tail
    Path             /var/log/containers/*.log
    Parser           docker
    Tag              kube.*
    Refresh_Interval 5
    Mem_Buf_Limit    5MB
    Skip_Long_Lines  On
    DB               /tail-db/tail-containers-state.db
    DB.Sync          Normal
    Ignore_Older 2d
    Multiline On
    Multiline_Flush 5
    Parser_Firstline first_line
...
[FILTER]
    Name lua
    Match kube.*
    script /fluent-bit/etc/helpers.lua
    call process
...
[PARSER]
    Name        first_line
    Format      regex
    Regex       ^{"log":"(?!\\u0009)(?<log>\S(?:(\\")|[^"]){9}(?:(\\")|[^"])*)"

You might want to fine tune the Regex of the parser for your specific use case.

I'm running into the same issue and have been struggling loads with fluent-bit and multi-line and splunk!

In our case, fluent-bit was the only component used to collect and ship log straight to an elasticsearch instance and we didn't want to add more components to the stack, but you might be able to handle such a thing on the Splunk side.

Here you go:

@stang Hi，seems the lua config files can not be reachable. would you please share me one when you have time? Thanks

Our logs look like:

2019-11-17 07:14:12 +0000 [info]: create client with URL: https://100.64.0.1:443/api and apiVersion: v1
2019-11-17 07:14:13 +0000 [info]: using configuration file: <ROOT>
  <source>
    @type events
    deploy_namespace "demo"
  </source>
  <source>
    @type prometheus
    metrics_path "/metrics"
    port 24231
  </source>
</ROOT>
2019-11-17 07:14:13 +0000 [info]: starting fluentd-1.6.3 pid=8 ruby="2.6.3"

Before turning on the Multiline each line is read a separate line and the JSON is correct.

Eg:

{"log":"2019-11-17 07:14:12 +0000 [info]: create client with URL: https://100.64.0.1:443/api and apiVersion: v1","stream":"stdout","time":"2019-11-17T07:14:12.020572877Z"}

Using the below parser for Parser_Firstline .

(?<log>\d{4}-\d{1,2}-\d{1,2} \d{2}:\d{2}:\d{2}.*)$

But when the multiline feature is turned on, \\n is added which escapes the remaining key value pairs and consider it as part of the "log" itself :

{"log":"2019-11-17 06:53:51 +0000 [info]: create client with URL: https://100.64.0.1:443/api and apiVersion: v1\\n\",\"stream\":\"stdout\",\"time\":\"2019-11-17T06:53:51.792044138Z\"}"}

Can anyone suggest a way to resolve this?

Hello, any news on this? And does filebeat actually support this?

Update: Yes, filebeat supports this and it can have multiple multiline parsers for different containers based on templating using kubernetes metadata.

Has there been any work done in this area yet?

@stang thank you, we followed the Lua approach and it's working well!

@stang any suggestions on below log format . We are having non json messages some times in logs while we have panics. Problem here is json log message single line working well but when we receive panic message each line considering as new line, what we want is wrap all the panic message to single message. Any suggestion please! {"level":"info","ts":"2020-04-07T08:32:01.755-0600","logger":"cmwnext","caller":"middleware/middleware_logging.go:141","msg":"access.log","version":"463.0.0","interactionID":"f6d13193-ebca-4967-423a-07724e4e4d06","sessionID":"","userID":"","request.method":"GET","request.path":"/cmw/v1/panic","response.status.code":200,"response.status.text":"OK"} panic: Bailing out with a panic from main. All is NOT well. goroutine 1 [running]: p-bitbucket.imovetv.com/hydra/cmwnext/pkg/resumes.NotGonnaResume(...) /go/src/p-bitbucket.imovetv.com/hydra/cmwnext/pkg/resumes/resumes_handler.go:64 main.wireMeUpSomeWidespreadPanic(...) /go/src/p-bitbucket.imovetv.com/hydra/cmwnext/cmd/cmwnext/route.go:268 main.ListenAndServe(...) /go/src/p-bitbucket.imovetv.com/hydra/cmwnext/cmd/cmwnext/route.go:264 main.main() /go/src/p-bitbucket.imovetv.com/hydra/cmwnext/cmd/cmwnext/main.go:77 +0x7d2 {"level":"warn","ts":"2020-04-07T08:32:16.964-0600","caller":"go-config/consul.go:136","msg":"CONSUL ACCEPTED:","address":"d-gp2-consul.imovetv.com/k8s/clusters/preview-qak8s/config/cmwnext"}

@stang thank you, we followed the Lua approach and it's working well!

Hi Stang, I am trying to use the Lua approach but i am getting an error message

• Parser docker - cannot be used in docker

• [filter_lua] function process is not found

any idea why this is showing up. Also, just want to let you know that I am working docker logs .

Thanks Eswar

Any update on this thread ?

@stang thank you, we followed the Lua approach and it's working well!

Hi Stang, I am trying to use the Lua approach but i am getting an error message

• Parser docker - cannot be used in docker
• [filter_lua] function process is not found

any idea why this is showing up. Also, just want to let you know that I am working docker logs .

Thanks Eswar

Your case is the same with me! Any update will help me out?

any progress?

all good now :) , thanks everyone!

Multiline Update

As part of Fluent Bit v1.8, we have released a new Multiline core functionality. This new big feature allows you to configure new [MULTILINE_PARSER]s that support multi formats/auto-detection, new multiline mode on Tail plugin, and also on v1.8.2 (to be released on July 20th, 2021) a new Multiline Filter.

For now, you can take at the following documentation resources:

• Multiline Parsers / Concepts and Configuration
• Tail + New Multiline Core Feature
• New Multiline Filter

Documentation pages now point to complete config examples that are available on our repository.

Thanks everyone for supporting this!

Amazing feature! Well done team! 👏