The issue has been closed

Very high CPU usage after upgrade to 3.7.1 #81

L2v2P posted onGitHub

Problem/Motivation

After I upgraded from version 3.6.2 to 3.7.1 Influxdb started to use a lot of CPU cycles. According to docker stats it hits 300% CPU every couple of seconds which basically makes my RPi based HA unusable.

I run Influxdb with the following config (which worked fine up until 3.6.2):

auth: true
reporting: true
ssl: true
certfile: fullchain.pem
keyfile: privkey.pem
envvars:
  - name: INFLUXDB_MONITOR_STORE_ENABLED
    value: 'false'
  - name: INFLUXDB_DATA_CACHE_MAX_MEMORY_SIZE
    value: 100m
  - name: INFLUXDB_DATA_CACHE_SNAPSHOT_MEMORY_SIZE
    value: 50m
  - name: INFLUXDB_DATA_INDEX_VERSION
    value: tsi1
log_level: debug

Proposed changes

Other than somehow fixing this, can I downgrade Influxdb to 3.6.2 somehow until then? I can't seem to find any instructions on how to do this.

I'm not able to reproduce this behavior on my instances, so from my end that would be hard to fix at this point. Anything in de logs?

Downgrading can be done by restoring a partial snapshot.

posted by frenck almost 5 years ago

Also experiencing this since upgrade to 3.7.1. Of note - I'm also on dev channel. What I experience is as L2v2P describes, soon after startup, CPU spikes, and I also notice memory is high. It then crashes with a message that it ran out of memory. I have plenty of memory on my pi still free according to Glances (2.4 GB prior to trying to start add-on). I followed the log messages of the container, here is a dump of the entire contents.

https://pastebin.com/n2RstivM

<blockquote><img src="https://pastebin.com/i/facebook.png" width="48" align="right"><div>Pastebin</div><div><strong><a href="https://pastebin.com/n2RstivM">[s6-init] making user provided files available at /var/run/s6/etc...exited 0. [ - Pastebin.com</a></strong></div></blockquote>

posted by RonSpawnson almost 5 years ago

I wonder if it has something to do with the size of the Influxdb database as there might be a correlation between its size and what the RPi can stomach. Mine is about 1.7 Gigabyte. What about yours @RonSpawnson?

posted by L2v2P almost 5 years ago

Take a look at #53. It looks like your issue is similar/same in that it seems to fall over during compaction. I didn't find a solution since it seems to be specific to influxdb on 32-bit arm . Had to move influxdb to another machine. I have heard a report that influxdb doesn't have problem if you use a 64-bit OS with your RPi but not sure how you would get HASS running on that...

posted by rychardeh almost 5 years ago

Downgrading can be done by restoring a partial snapshot.

Okay, so I'm gonna try restoring. However, I don't want to lose my data since the upgrade so I want to make a remote backup first. Therefore I need to open up an additional port to the Influx container. @frenck How can I do that? I'm running Hassio on a RPi3.

posted by L2v2P almost 5 years ago

I downgraded to 3.6.2 which solved the issue. So, there seems to have been some change in the 3.7.1 that causes this. Will hold off upgrading for a while.

posted by L2v2P almost 5 years ago

I just wanted to put in I'm also experiencing this issue. Since 8th of June 06:00 (+2 GMT Amsterdam) my cpu usage skyrocketed to 50-60% average (unfortunately I use auto-update) from the regular 10-20% on Pi 4b. I was also seeing swap usage (had none until the 8th of June) and Garbage collecting. Weird thing was that swap was used, while at least 1.3gb of ram was free. Influxdb instance itself was using almost a GB. All in all performance dropped drastically.

Since I'm on auto-update I didn't notice (or rather ignored it) at first. But since I had some spare time over the weekend I looked into it. At first I couldn't pinpoint the issue (as I was using influxdb for my data) until I realized it was influxdb itself.

At first I though it was a sensor I installed, generating too much data. But downgrading to 3.6.1 (had no more recent backup) fixed the issue for me so I'm guessing somethings up with an update.

Is it possible there is some soft cap on the add-on not being able to use more than 1GB of memory? Forcing it to use swap (which was limited to 100mb and was filled to the brim degrading performance even more?)

Currently, I unfortunately have no data as I rolled back by influxdb instance. However I also made a back-up of the malperforming add-on before rolling back.

Is there any way I might be able to help resolve this issue? maybe provide the partial backup? I'm willing to share it, however it probably will contain personal data so i'd like to handle it with care.

My setup:

Pi 4b influxdb addon 3.7.1 raspbian stretch lite (32bit) PRETTY_NAME="Raspbian GNU/Linux 10 (buster)" NAME="Raspbian GNU/Linux" VERSION_ID="10" Home assistant supervised 0.111.2 supervisor 227

posted by MaartenKleijwegt almost 5 years ago

This thread has been automatically locked because it has not had recent activity. Please open a new issue for related bugs and link to relevant comments in this thread.

posted by addons-assistant[bot] almost 5 years ago

Fund this Issue

$0.00

Funded

Pull requests

Very high CPU usage after upgrade to 3.7.1 #81

Problem/Motivation

Proposed changes

Company

Community

Support

Connect