IssueHunt
Bug Bounty (New!)
Browse
Sign in with GitHub
hassio-addons/addon-influxdb

InfluxDB shutdown shortly after start #53

roblandry posted onGitHub

Problem/Motivation

SYSTEM:

  • PI4B 4GB RAM
  • Raspbian Buster
  • Docker version 19.03.2, build 6a30dfc
  • Supervisor: 192
  • HASSIO: 0.100.3

I have been unable to get InfluxDB to run, as it seems to crash shortly after start. It appears after starting chronograph it runs out of memory.

ADDON CONFIG:

{
  "auth": true,
  "reporting": true,
  "ssl": true,
  "certfile": "fullchain.pem",
  "keyfile": "privkey.pem"
}

Expected behavior

Run without memory errors

Actual behavior

-----------------------------------------------------------
 Hass.io Add-on: InfluxDB
 Scalable datastore for metrics, events, and real-time analytics
-----------------------------------------------------------
 Add-on version: 3.3.0
 You are running the latest version of this add-on.
 System: Raspbian GNU/Linux 10 (buster)  (armv7 / raspberrypi4)
 Home Assistant version: 0.100.3
 Supervisor version: 192
-----------------------------------------------------------
 Please, share the above information when looking for help
 or support in, e.g., GitHub, forums or the Discord chat.
-----------------------------------------------------------
[cont-init.d] 00-banner.sh: exited 0.
[cont-init.d] 01-log-level.sh: executing... 
[cont-init.d] 01-log-level.sh: exited 0.
[cont-init.d] create-users.sh: executing... 
[cont-init.d] create-users.sh: exited 0.
[cont-init.d] influxdb.sh: executing... 
[cont-init.d] influxdb.sh: exited 0.
[cont-init.d] kapacitor.sh: executing... 
[cont-init.d] kapacitor.sh: exited 0.
[cont-init.d] nginx.sh: executing... 
[cont-init.d] nginx.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[03:04:27] INFO: Kapacitor is waiting until InfluxDB is available...
[03:04:27] INFO: Chronograf is waiting until InfluxDB is available...
[03:04:27] INFO: Starting the InfluxDB...
[03:05:53] INFO: Starting the Kapacitor

'##:::'##::::'###::::'########:::::'###:::::'######::'####:'########::'#######::'########::
 ##::'##::::'## ##::: ##.... ##:::'## ##:::'##... ##:. ##::... ##..::'##.... ##: ##.... ##:
 ##:'##::::'##:. ##:: ##:::: ##::'##:. ##:: ##:::..::: ##::::: ##:::: ##:::: ##: ##:::: ##:
 #####::::'##:::. ##: ########::'##:::. ##: ##:::::::: ##::::: ##:::: ##:::: ##: ########::
2019/11/01 03:05:55 Using configuration at: /etc/kapacitor/kapacitor.conf
 ##. ##::: #########: ##.....::: #########: ##:::::::: ##::::: ##:::: ##:::: ##: ##.. ##:::
 ##:. ##:: ##.... ##: ##:::::::: ##.... ##: ##::: ##:: ##::::: ##:::: ##:::: ##: ##::. ##::
 ##::. ##: ##:::: ##: ##:::::::: ##:::: ##:. ######::'####:::: ##::::. #######:: ##:::. ##:
..::::..::..:::::..::..:::::::::..:::::..:::......:::....:::::..::::::.......:::..:::::..::

[03:05:55] INFO: Starting Chronograf...
runtime: out of memory: cannot allocate 24576-byte block (589758464 in use)
fatal error: out of memory
runtime: out of memory: cannot allocate 8192-byte block (589758464 in use)
fatal error: out of memory

runtime stack:
runtime.throw(0xc8fe90, 0xd)
    /usr/local/go/src/runtime/panic.go:608 +0x5c
runtime.(*mcache).refill(0xb6efb6c0, 0x304265d)
    /usr/local/go/src/runtime/mcache.go:124 +0xe8
runtime.(*mcache).nextFree.func1()
    /usr/local/go/src/runtime/malloc.go:749 +0x24
runtime.systemstack(0x304e1e0)
    /usr/local/go/src/runtime/asm_arm.s:354 +0x84
runtime.mstart()
    /usr/local/go/src/runtime/proc.go:1229

goroutine 3533 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_arm.s:298 +0x4 fp=0xcab5548 sp=0xcab5544 pc=0x6be48
runtime.(*mcache).nextFree(0xb6efb6c0, 0x1185d, 0x11abc, 0x1185d, 0x11b00)
    /usr/local/go/src/runtime/malloc.go:748 +0xa4 fp=0xcab5574 sp=0xcab5548 pc=0x1a6f4
runtime.mallocgc(0x1800, 0xb32eb0, 0x9ea901, 0xb32eb0)
    /usr/local/go/src/runtime/malloc.go:903 +0x768 fp=0xcab55dc sp=0xcab5574 pc=0x1b014
runtime.makeslice(0xb32eb0, 0x2e8, 0x2e8, 0x2e8, 0x9e9f3c, 0xb32eb0)
    /usr/local/go/src/runtime/slice.go:70 +0x68 fp=0xcab55f0 sp=0xcab55dc pc=0x54714
runtime.makeslice64(0xb32eb0, 0x2e8, 0x0, 0x2e8, 0x0, 0x2, 0x39053, 0x6dc15)
    /usr/local/go/src/runtime/slice.go:85 +0x44 fp=0xcab560c sp=0xcab55f0 pc=0x547e0
github.com/influxdata/influxdb/tsdb/engine/tsm1.integerBatchDecodeAllRLE(0xa36fbe92, 0xb, 0x40e431, 0x0, 0x0, 0x0, 0xf46ba000, 0xbe7edf38, 0x1f, 0x0, ...)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/batch_integer.go:260 +0x2a0 fp=0xcab5668 sp=0xcab560c pc=0x9e84e0
github.com/influxdata/influxdb/tsdb/engine/tsm1.IntegerArrayDecodeAll(0xa36fbe91, 0xc, 0x40e431, 0x0, 0x0, 0x0, 0xbe7ec800, 0x2e8, 0x2e8, 0x0, ...)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/batch_integer.go:150 +0x78 fp=0xcab5698 sp=0xcab5668 pc=0x9e7a04
github.com/influxdata/influxdb/tsdb/engine/tsm1.DecodeIntegerArrayBlock(0xa36fbe83, 0x1a, 0x40e43f, 0xcab5770, 0xa70c90, 0xa1513d97)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/array_encoding.go:68 +0x12c fp=0xcab56e4 sp=0xcab5698 pc=0x9e3ed4
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*tsmBatchKeyIterator).combineInteger(0x34a77a0, 0xd162400, 0x1005b40, 0xd1624d0, 0x34a77e8)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.gen.go:1377 +0xdec fp=0xcab5788 sp=0xcab56e4 pc=0x9f1bc4
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*tsmBatchKeyIterator).mergeInteger(0x34a77a0)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.gen.go:1253 +0x25c fp=0xcab57a0 sp=0xcab5788 pc=0x9f0c98
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*tsmBatchKeyIterator).merge(0x34a77a0)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:1809 +0x40 fp=0xcab57c8 sp=0xcab57a0 pc=0x9fe94c
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*tsmBatchKeyIterator).Next(0x34a77a0, 0x6231ae)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:1792 +0x108 fp=0xcab58dc sp=0xcab57c8 pc=0x9fdb00
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).write(0x30c25a0, 0x92d2e60, 0x45, 0x1008a98, 0x34a77a0, 0x8274b01, 0x0, 0x0)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:1128 +0x148 fp=0xcab596c sp=0xcab58dc pc=0x9fcbb8
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).writeNewFiles(0x30c25a0, 0x10, 0x2, 0xdb28c00, 0x8, 0x8, 0x1008a98, 0x34a77a0, 0x1, 0x0, ...)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:1032 +0x128 fp=0xcab59c4 sp=0xcab596c pc=0x9fc6cc
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).compact(0x30c25a0, 0xeaf600, 0xdb28c00, 0x8, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:940 +0x4a0 fp=0xcab5a50 sp=0xcab59c4 pc=0x9fbe78
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Compactor).CompactFull(0x30c25a0, 0xdb28c00, 0x8, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/compact.go:958 +0x108 fp=0xcab5aa0 sp=0xcab5a50 pc=0x9fc00c
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*compactionStrategy).compactGroup(0xdb28c40)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2152 +0xaac fp=0xcab5f88 sp=0xcab5aa0 pc=0xa1b8d4
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*compactionStrategy).Apply(0xdb28c40)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2129 +0x2c fp=0xcab5fbc sp=0xcab5f88 pc=0xa1ade8
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactFull.func1(0x9aaef70, 0x30ba000, 0xdb28c40)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2098 +0xac fp=0xcab5fdc sp=0xcab5fbc pc=0xa64648
runtime.goexit()
    /usr/local/go/src/runtime/asm_arm.s:867 +0x4 fp=0xcab5fdc sp=0xcab5fdc pc=0x6db90
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactFull
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2094 +0xd0

goroutine 1 [chan receive]:
main.(*Main).Run(0x3181fac, 0x308a0e0, 0x0, 0x0, 0x1a24d08, 0x0)
    /go/src/github.com/influxdata/influxdb/cmd/influxd/main.go:90 +0x234
main.main()
    /go/src/github.com/influxdata/influxdb/cmd/influxd/main.go:45 +0x130

goroutine 18 [syscall, 1 minutes]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:139 +0x130
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:23 +0x14
created by os/signal.init.0
    /usr/local/go/src/os/signal/signal_unix.go:29 +0x30

goroutine 33 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0xa6c5bf80, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:173 +0x44
internal/poll.(*pollDesc).wait(0x3178474, 0x72, 0x3068500, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x7c
internal/poll.(*pollDesc).waitRead(0x3178474, 0xffffff00, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x2c
internal/poll.(*FD).Accept(0x3178460, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:384 +0x17c
net.(*netFD).accept(0x3178460, 0xffffffff, 0x0, 0x1)
    /usr/local/go/src/net/fd_unix.go:238 +0x20
net.(*TCPListener).accept(0x32415f0, 0x3147ec0, 0x40000000, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:139 +0x20
net.(*TCPListener).Accept(0x32415f0, 0xfcbac, 0x11938, 0x19428, 0x3086330)
    /usr/local/go/src/net/tcpsock.go:260 +0x3c
github.com/influxdata/influxdb/tcp.(*Mux).Serve(0x3147ec0, 0x10054a0, 0x32415f0, 0x32415f0, 0x0)
    /go/src/github.com/influxdata/influxdb/tcp/mux.go:75 +0x60
created by github.com/influxdata/influxdb/cmd/influxd/run.(*Server).Open
    /go/src/github.com/influxdata/influxdb/cmd/influxd/run/server.go:387 +0x1ec

goroutine 3224 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3199 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3200 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3185 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3187 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3184 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3173 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3181 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3180 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3182 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3114 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 2859 [select]:
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache(0x30ba000)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1920 +0xd8
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1(0x9aaef60, 0x30ba000)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:499 +0x40
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:499 +0xfc

goroutine 3213 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3210 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3179 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3167 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3223 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3211 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3234 [select]:
github.com/influxdata/influxdb/cmd/influxd/run.(*Server).startServerReporting(0x330b8c0)
    /go/src/github.com/influxdata/influxdb/cmd/influxd/run/server.go:515 +0xc4
created by github.com/influxdata/influxdb/cmd/influxd/run.(*Server).Open
    /go/src/github.com/influxdata/influxdb/cmd/influxd/run/server.go:460 +0x8a0

goroutine 3221 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3212 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3235 [select]:
github.com/influxdata/influxdb/cmd/influxd/run.(*Command).monitorServerErrors(0x319c5a0)
    /go/src/github.com/influxdata/influxdb/cmd/influxd/run/command.go:174 +0x134
created by github.com/influxdata/influxdb/cmd/influxd/run.(*Command).Run
    /go/src/github.com/influxdata/influxdb/cmd/influxd/run/command.go:155 +0x920

goroutine 3164 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3209 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3233 [select]:
github.com/influxdata/influxdb/services/retention.(*Service).run(0x32cfc80)
    /go/src/github.com/influxdata/influxdb/services/retention/service.go:78 +0x8bc
github.com/influxdata/influxdb/services/retention.(*Service).Open.func1(0x32cfc80)
    /go/src/github.com/influxdata/influxdb/services/retention/service.go:51 +0x48
created by github.com/influxdata/influxdb/services/retention.(*Service).Open
    /go/src/github.com/influxdata/influxdb/services/retention/service.go:51 +0x130

goroutine 3119 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3216 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3207 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3215 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3166 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3145 [select]:
github.com/influxdata/influxdb/tsdb.(*Store).monitorShards(0x310a000)
    /go/src/github.com/influxdata/influxdb/tsdb/store.go:1910 +0x100
github.com/influxdata/influxdb/tsdb.(*Store).Open.func1(0x310a000)
    /go/src/github.com/influxdata/influxdb/tsdb/store.go:231 +0x30
created by github.com/influxdata/influxdb/tsdb.(*Store).Open
    /go/src/github.com/influxdata/influxdb/tsdb/store.go:229 +0x20c

goroutine 3115 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3156 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3174 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3169 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3162 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3172 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3165 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3197 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3190 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3198 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3193 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3186 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3191 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3188 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3159 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3168 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3220 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3178 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3225 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3219 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3222 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3153 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3218 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3158 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3118 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3150 [select]:
github.com/influxdata/influxdb/tcp.(*listener).Accept(0x32cfcb0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/influxdata/influxdb/tcp/mux.go:236 +0xd8
github.com/influxdata/influxdb/services/snapshotter.(*Service).serve(0x32cfb00)
    /go/src/github.com/influxdata/influxdb/services/snapshotter/service.go:94 +0x54
created by github.com/influxdata/influxdb/services/snapshotter.(*Service).Open
    /go/src/github.com/influxdata/influxdb/services/snapshotter/service.go:68 +0x74

goroutine 3149 [select]:
github.com/influxdata/influxdb/services/precreator.(*Service).runPrecreation(0x32cfaa0)
    /go/src/github.com/influxdata/influxdb/services/precreator/service.go:76 +0xc8
created by github.com/influxdata/influxdb/services/precreator.(*Service).Open
    /go/src/github.com/influxdata/influxdb/services/precreator/service.go:54 +0x164

goroutine 2860 [select]:
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compact(0x30ba000, 0x9aaef70)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1967 +0x1a8
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions.func1(0x9aaef70, 0x30ba000)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:420 +0x48
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:420 +0x114

goroutine 3206 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3208 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3217 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3214 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3146 [select]:
github.com/influxdata/influxdb/services/subscriber.(*Service).run(0x308cb00)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:239 +0x1d8
github.com/influxdata/influxdb/services/subscriber.(*Service).Open.func1(0x308cb00)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:98 +0x48
created by github.com/influxdata/influxdb/services/subscriber.(*Service).Open
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:96 +0x13c

goroutine 3157 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3116 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3176 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3175 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3152 [IO wait]:
internal/poll.runtime_pollWait(0xa6c5be80, 0x72, 0x0)
    /usr/local/go/src/runtime/netpoll.go:173 +0x44
internal/poll.(*pollDesc).wait(0xe736e24, 0x72, 0xa7bb000, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x7c
internal/poll.(*pollDesc).waitRead(0xe736e24, 0xffffff00, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x2c
internal/poll.(*FD).Accept(0xe736e10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:384 +0x17c
net.(*netFD).accept(0xe736e10, 0x1a36b60, 0x0, 0x2cd33c)
    /usr/local/go/src/net/fd_unix.go:238 +0x20
net.(*TCPListener).accept(0xdf93288, 0xa11d300, 0xbbd95300, 0x46344)
    /usr/local/go/src/net/tcpsock_posix.go:139 +0x20
net.(*TCPListener).Accept(0xdf93288, 0xb518f28, 0xc, 0x33e9ea0, 0x2d1410)
    /usr/local/go/src/net/tcpsock.go:260 +0x3c
net/http.(*Server).Serve(0x7241c80, 0x10054a0, 0xdf93288, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2826 +0x1e0
net/http.Serve(0x10054a0, 0xdf93288, 0xffbf28, 0x308cc00, 0x6, 0xb518fc8)
    /usr/local/go/src/net/http/server.go:2423 +0x58
github.com/influxdata/influxdb/services/httpd.(*Service).serve(0x319cea0, 0x10054a0, 0xdf93288)
    /go/src/github.com/influxdata/influxdb/services/httpd/service.go:249 +0x38
github.com/influxdata/influxdb/services/httpd.(*Service).serveTCP(0x319cea0)
    /go/src/github.com/influxdata/influxdb/services/httpd/service.go:237 +0x2c
created by github.com/influxdata/influxdb/services/httpd.(*Service).Open
    /go/src/github.com/influxdata/influxdb/services/httpd/service.go:186 +0x474

goroutine 3151 [select]:
github.com/influxdata/influxdb/services/continuous_querier.(*Service).backgroundLoop(0x319ce40)
    /go/src/github.com/influxdata/influxdb/services/continuous_querier/service.go:215 +0x114
created by github.com/influxdata/influxdb/services/continuous_querier.(*Service).Open
    /go/src/github.com/influxdata/influxdb/services/continuous_querier/service.go:133 +0x154

goroutine 3117 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 2922 [select]:
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compact(0x319a210, 0x9aaf350)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1967 +0x1a8
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions.func1(0x9aaf350, 0x319a210)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:420 +0x48
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:420 +0x114

goroutine 3170 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3161 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3171 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3155 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3163 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3148 [select]:
github.com/influxdata/influxdb/monitor.(*Monitor).storeStatistics(0x330b950)
    /go/src/github.com/influxdata/influxdb/monitor/service.go:434 +0x5b4
created by github.com/influxdata/influxdb/monitor.(*Monitor).Open
    /go/src/github.com/influxdata/influxdb/monitor/service.go:126 +0x2b8

goroutine 3147 [select]:
github.com/influxdata/influxdb/services/subscriber.(*Service).waitForMetaUpdates(0x308cb00)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:165 +0x8c
github.com/influxdata/influxdb/services/subscriber.(*Service).Open.func2(0x308cb00)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:102 +0x48
created by github.com/influxdata/influxdb/services/subscriber.(*Service).Open
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:100 +0x158

goroutine 3160 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3120 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 2858 [select]:
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compact(0x3e984d0, 0x9aaef50)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1967 +0x1a8
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions.func1(0x9aaef50, 0x3e984d0)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:420 +0x48
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:420 +0x114

goroutine 3154 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 2596 [runnable]:
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compact(0x40dc160, 0x6519070)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1967 +0x1a8
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions.func1(0x6519070, 0x40dc160)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:420 +0x48
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableLevelCompactions
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:420 +0x114

goroutine 2921 [select]:
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache(0x319a210)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1920 +0xd8
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1(0x9aaf340, 0x319a210)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:499 +0x40
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:499 +0xfc

goroutine 2857 [select]:
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache(0x3e984d0)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1920 +0xd8
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1(0x9aaef40, 0x3e984d0)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:499 +0x40
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:499 +0xfc

goroutine 2595 [runnable]:
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactCache(0x40dc160)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:1920 +0xd8
github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions.func1(0x6519060, 0x40dc160)
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:499 +0x40
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).enableSnapshotCompactions
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:499 +0xfc

goroutine 3192 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3189 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3177 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3183 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6f00, 0xffc120, 0xc5ed290, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3204 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3203 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3202 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3201 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3194 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3195 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3196 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3205 [chan receive]:
github.com/influxdata/influxdb/services/subscriber.chanWriter.Run(0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:372 +0x38
github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs.func1(0x3e5dca0, 0xdfe6fc0, 0xffc120, 0xc5ed2c0, 0x340ef08, 0x340ef10, 0x32cfe90)
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:314 +0x44
created by github.com/influxdata/influxdb/services/subscriber.(*Service).updateSubs
    /go/src/github.com/influxdata/influxdb/services/subscriber/service.go:312 +0x858

goroutine 3541 [IO wait]:
internal/poll.runtime_pollWait(0xa6c5be00, 0x72, 0xb9e98)
    /usr/local/go/src/runtime/netpoll.go:173 +0x44
internal/poll.(*pollDesc).wait(0xe7375a4, 0x72, 0xffffff00, 0xffea90, 0x18d0198)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x7c
internal/poll.(*pollDesc).waitRead(0xe7375a4, 0x3523000, 0x1000, 0x1000)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x2c
internal/poll.(*FD).Read(0xe737590, 0x3523000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:169 +0x14c
net.(*netFD).Read(0xe737590, 0x3523000, 0x1000, 0x1000, 0xffffffff, 0x0, 0x2c91d0)
    /usr/local/go/src/net/fd_unix.go:202 +0x38
net.(*conn).Read(0x32aaca0, 0x3523000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:177 +0x58
net/http.(*connReader).Read(0xdca6ba0, 0x3523000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:786 +0x14c
bufio.(*Reader).fill(0xe75a000)
    /usr/local/go/src/bufio/bufio.go:100 +0x10c
bufio.(*Reader).Peek(0xe75a000, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0x2c
net/http.(*conn).readRequest(0xa11d2c0, 0x1006f00, 0xdca6b80, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:963 +0xa84
net/http.(*conn).serve(0xa11d2c0, 0x1006f00, 0xdca6b80)
    /usr/local/go/src/net/http/server.go:1788 +0x3f0
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2851 +0x290

goroutine 3228 [select]:
net/http.(*persistConn).writeLoop(0xdcb61e0)
    /usr/local/go/src/net/http/transport.go:1885 +0xb8
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1339 +0x814

goroutine 3227 [IO wait]:
internal/poll.runtime_pollWait(0xa6c5bf00, 0x72, 0xb9e98)
    /usr/local/go/src/runtime/netpoll.go:173 +0x44
internal/poll.(*pollDesc).wait(0xeb09144, 0x72, 0xffffff00, 0xffea90, 0x18d0198)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x7c
internal/poll.(*pollDesc).waitRead(0xeb09144, 0xc5ef400, 0x400, 0x400)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x2c
internal/poll.(*FD).Read(0xeb09130, 0xc5ef400, 0x400, 0x400, 0x0, 0x0, 0x0)
    /usr/local/go/src/internal/poll/fd_unix.go:169 +0x14c
net.(*netFD).Read(0xeb09130, 0xc5ef400, 0x400, 0x400, 0x18de4, 0x3022000, 0xb63160)
    /usr/local/go/src/net/fd_unix.go:202 +0x38
net.(*conn).Read(0xd9a0058, 0xc5ef400, 0x400, 0x400, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:177 +0x58
crypto/tls.(*block).readFromUntil(0x58d0d60, 0xffd3c8, 0xd9a0058, 0x5, 0xd9a0058, 0xb05b0)
    /usr/local/go/src/crypto/tls/conn.go:492 +0x70
crypto/tls.(*Conn).readRecord(0x30da480, 0xeb1217, 0x30da524, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:593 +0x9c
crypto/tls.(*Conn).Read(0x30da480, 0xde3b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1145 +0xcc
net/http.(*persistConn).Read(0xdcb61e0, 0xde3b000, 0x1000, 0x1000, 0xdcb61e0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1497 +0x170
bufio.(*Reader).fill(0xc9af7a0)
    /usr/local/go/src/bufio/bufio.go:100 +0x10c
bufio.(*Reader).Peek(0xc9af7a0, 0x1, 0x2, 0x0, 0x0, 0xdcfe540, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0x2c
net/http.(*persistConn).readLoop(0xdcb61e0)
    /usr/local/go/src/net/http/transport.go:1645 +0x164
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1338 +0x7f8

goroutine 3527 [select]:
net/http.setRequestCancel.func3(0x0, 0xcbbe120, 0xb5c36e0, 0x73b09fc, 0xb17bc40)
    /usr/local/go/src/net/http/client.go:321 +0x78
created by net/http.setRequestCancel
    /usr/local/go/src/net/http/client.go:320 +0x210

goroutine 3537 [running]:
    goroutine running on other thread; stack unavailable
created by github.com/influxdata/influxdb/tsdb/engine/tsm1.(*Engine).compactFull
    /go/src/github.com/influxdata/influxdb/tsdb/engine/tsm1/engine.go:2094 +0xd0

runtime stack:
runtime.throw(0xc8fe90, 0xd)
    /usr/local/go/src/runtime/panic.go:608 +0x5c
runtime.(*mcache).refill(0xb6efb000, 0x3040065)
    /usr/local/go/src/runtime/mcache.go:124 +0xe8
runtime.(*mcache).nextFree.func1()
    /usr/local/go/src/runtime/malloc.go:749 +0x24
runtime.systemstack(0x304e5a0)
    /usr/local/go/src/runtime/asm_arm.s:354 +0x84
runtime.mstart()
    /usr/local/go/src/runtime/proc.go:1229
[cont-finish.d] executing container finish scripts...
[cont-finish.d] 99-message.sh: executing... 
[cont-finish.d] 99-message.sh: exited 0.
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.

Steps to reproduce

Not sure

Proposed changes

Not sure

:wave: Thanks for opening your first issue here! If you're reporting a :bug: bug, please make sure you include steps to reproduce it. Also, logs, error messages and information about your hardware might be usefull.

posted by addons-assistant[bot] over 5 years ago

This looks like it may possibly be resolved in the upcoming 1.7.9 Influx release, either way it is likely an Influx specific issue rather than the add-on itself. Will leave it open to be closed when that release is available.

posted by sinclairpaul over 5 years ago

I had the same error as roblandry. I restored the influxdb add-on from a snapshot back to version 3.2.1 and it was working again. As suggested by sinclairpaul I will wait until version 1.7.9 of Influx will be available before my next try. :-)

posted by kwertie01 over 5 years ago

I've also had this problem for 6 weeks or so. I hope the compaction bug fixes in 1.7.9 eliminate the issue.

posted by rychardeh over 5 years ago

1.7.9 is in the latest release, closing for now.

posted by sinclairpaul over 5 years ago

Sadly, this problem still persists after upgrading to 1.7.9.

posted by rychardeh over 5 years ago

Sadly, this problem still persists after upgrading to 1.7.9.

Confirm.

Just now upgraded to add-on v3.4.1 containing InfluxDB 1.7.9 and I also get the above shown crash/shutdown directly after/during startup, as I did before the add-on upgrade.

(on RPi3B+, Hassio, HA 0.100.3)

posted by ge2rt over 5 years ago

You will need to log the issue at https://github.com/influxdata/influxdb

<blockquote><img src="https://avatars2.githubusercontent.com/u/5713248?s=400&v=4" width="48" align="right"><div><img src="https://github.githubassets.com/favicon.ico" height="14"> GitHub</div><div><strong><a href="https://github.com/influxdata/influxdb">influxdata/influxdb</a></strong></div><div>Scalable datastore for metrics, events, and real-time analytics - influxdata/influxdb</div></blockquote>

posted by sinclairpaul over 5 years ago

This previous report of an issue was closed with no action: https://github.com/influxdata/influxdb/issues/12902.

There are various reports of memory issues like this one: https://community.influxdata.com/t/influxdb-1-7-4-fails-after-9-months-without-issues/9157 and https://stackoverflow.com/questions/56925881/influxdb-1-7-x-high-cpu-usage-on-rasperry-pi-3

It's not clear to me whether "our" specific issue is due to compaction on startup or not, but this is a ongoing issue that results in out of memory errors: https://github.com/influxdata/influxdb/issues/11339

The most sobering read, however, is this issue which relates to a solution for 32-bit systems which has not been picked up by influxdb: https://github.com/influxdata/influxdb/pull/12362:

For anyone still following this particular issue, specifically when running on a IoT device such as a Raspberry Pi with raspbian (a 32 bit OS) - if you get this dreaded error, the only solution is to simply load up the database on another 64-bit device just to run the compaction periodically, then transfer it back. It somewhat defeats the benefits of a small IoT device, but seems very much that InfluxDB are just not interested in the next wave IoT capabilities that are beginning to emerge, at this time. In short... don't waste too much time trying to get InfluxDB running on a 32-bit platform with even a moderate size DB, as it will inevitably fail with this issue.

My database is currently 1.7GB and presumably not compacted so I think I may have some way to go before the issue is terminal based on that what I've read but the only possible interim solution I could find was to switch from tsm to tsi to reduce memory burden. I'm not whether it will work or not, but I can't see any way of following the procedure since influxdb is inside the addon's container https://docs.influxdata.com/influxdb/v1.7/concepts/tsi-details/. Specifically the use of buildtsi:

If you want to convert an existing shard from an in-memory index to a TSI index, or if you have an existing TSI index which has become corrupt, you can use the buildtsi command to create the index from the underlying TSM data. If you have an existing TSI index that you want to rebuild, first delete the index directory within your shard.

Any thoughts, other than time to find another time-series database?

posted by rychardeh over 5 years ago

This thread has been automatically locked because it has not had recent activity. Please open a new issue for related bugs and link to relevant comments in this thread.

posted by addons-assistant[bot] over 5 years ago

Fund this Issue

$0.00
Funded

Pull requests

Company
AboutMission
Community
SpectrumEmbed
Support
FAQTerms of usePrivacy policyCode of conductCredits
Connect
FacebookTwitter

© 2019 BoostIO, Inc.