Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Couldn't write data for continuous query #1120

Closed
odolbeau opened this issue Nov 14, 2014 · 5 comments
Closed

Couldn't write data for continuous query #1120

odolbeau opened this issue Nov 14, 2014 · 5 comments

Comments

@odolbeau
Copy link

Hi!

I have a lot of errors with my continuous queries. Unfortunately, I cannot explain why I encounter this problem and I didn't find anyone who have the same issue.

Environment:

  • Single VM (8 cores / 16Go RAM)
  • Debian 7.6
  • InfluxDB v0.8.5 (git: 9485e99) (leveldb: 1.15)

Our applications send events to an logstash server, configured with the influxdb output.
We receive (I think) between 100k & 150k events / minute.

Here is the error:

[2014/11/13 17:24:01 UTC] [EROR] (github.com/influxdb/influxdb/coordinator.(*Coordinator).CommitSeriesData:552) COORD error writing: %!(EXTRA *errors.errorString=proto: Marshal called with nil)
[2014/11/13 17:24:01 UTC] [EROR] (github.com/influxdb/influxdb/coordinator.(*Coordinator).InterpolateValuesAndCommit:482) Couldn't write data for continuous query: %!(EXTRA *errors.errorString=proto: Marshal called with nil)

If needed I can give you more informations, just tell me. :)

@odolbeau
Copy link
Author

I updated to the version 0.8.6.

Here is the updated error:

[2014/11/20 09:57:01 UTC] [EROR] (github.com/influxdb/influxdb/coordinator.(*Coordinator).CommitSeriesData:567) COORD error writing: %!(EXTRA *errors.errorString=proto: Marshal called with nil)
[2014/11/20 09:57:01 UTC] [EROR] (github.com/influxdb/influxdb/coordinator.(*Coordinator).InterpolateValuesAndCommit:486) Couldn't write data for continuous query: %!(EXTRA *errors.errorString=proto: Marshal called with nil)

@toddboom
Copy link
Contributor

@odolbeau Do you have other continuous queries that are working correctly? Can you provide any other details about the continuous queries themselves?

@odolbeau
Copy link
Author

We use InfluxDB to store all http queries made on our website.
We sent all informations to logstash and we use this output.

Here is the json used to create the DB.
With these continuous queries, I have the problem I explained before.

{
    "spaces": [
        {
            "name": "everything",
            "retentionPolicy": "6h",
            "shardDuration": "3h",
            "regex": "/.*/",
            "replicationFactor": 1,
            "split": 1
        },
        {
            "name": "1h",
            "retentionPolicy": "30d",
            "shardDuration": "1d",
            "regex": "/.*1h/",
            "replicationFactor": 1,
            "split": 1
        },
        {
            "name": "1m",
            "retentionPolicy": "7d",
            "shardDuration": "1d",
            "regex": "/.*1m/",
            "replicationFactor": 1,
            "split": 1
        }
    ],
    "continuousQueries": [
        "select * from http.response into http.response.by_media.[media]",
        "select * from http.response into http.response.by_locale.[locale]",
        "select * from http.response into http.response.by_locale_and_media.[locale].[media]",
        "select * from http.response into http.response.by_route.[route]",

        "select count(duration), median(duration) as median, percentile(duration, 95) as percentile_95, percentile(duration, 99) as percentile_99 from /^http.response.by_.*/ group by time(1m) into agg.:series_name.duration.1m",
        "select count(duration), median(duration) as median, percentile(duration, 95) as percentile_95, percentile(duration, 99) as percentile_99 from /^http.response.by_.*/ group by time(1h) into agg.:series_name.duration.1h"

    ]
}

All associated logs:

[2014/11/28 10:24:00 UTC] [INFO] (github.com/influxdb/influxdb/coordinator.(*Coordinator).RunQuery:41) Start Query: db: metrics, u: root, q: select count(duration),median(duration) as median,percentile(duration,95) as percentile_95,percentile(duration,99) as percentile_99 from /^http.response.by_.*/ where (time < 1417170240000000000) AND (time > 1417170180000000000) group by time(1m)
[2014/11/28 10:24:00 UTC] [INFO] (github.com/influxdb/influxdb/cluster.(*ShardData).Query:325) Processor chain:  AggregatorEngine > PassthroughEngine > ShardIdInserterProcessor (577) > ResponseChannelProcessor

[2014/11/28 10:24:00 UTC] [EROR] (github.com/influxdb/influxdb/coordinator.(*Coordinator).CommitSeriesData:567) COORD error writing: %!(EXTRA *errors.errorString=proto: Marshal called with nil)
[2014/11/28 10:24:00 UTC] [EROR] (github.com/influxdb/influxdb/coordinator.(*Coordinator).InterpolateValuesAndCommit:486) Couldn't write data for continuous query: %!(EXTRA *errors.errorString=proto: Marshal called with nil)
[2014/11/28 10:24:01 UTC] [EROR] (github.com/influxdb/influxdb/coordinator.(*Coordinator).CommitSeriesData:567) COORD error writing: %!(EXTRA *errors.errorString=proto: Marshal called with nil)
[2014/11/28 10:24:01 UTC] [EROR] (github.com/influxdb/influxdb/coordinator.(*Coordinator).InterpolateValuesAndCommit:486) Couldn't write data for continuous query: %!(EXTRA *errors.errorString=proto: Marshal called with nil)
[2014/11/28 10:24:01 UTC] [EROR] (github.com/influxdb/influxdb/coordinator.(*Coordinator).CommitSeriesData:567) COORD error writing: %!(EXTRA *errors.errorString=proto: Marshal called with nil)
[2014/11/28 10:24:01 UTC] [EROR] (github.com/influxdb/influxdb/coordinator.(*Coordinator).InterpolateValuesAndCommit:486) Couldn't write data for continuous query: %!(EXTRA *errors.errorString=proto: Marshal called with nil)
[2014/11/28 10:24:01 UTC] [EROR] (github.com/influxdb/influxdb/coordinator.(*Coordinator).CommitSeriesData:567) COORD error writing: %!(EXTRA *errors.errorString=proto: Marshal called with nil)
[2014/11/28 10:24:01 UTC] [EROR] (github.com/influxdb/influxdb/coordinator.(*Coordinator).InterpolateValuesAndCommit:486) Couldn't write data for continuous query: %!(EXTRA *errors.errorString=proto: Marshal called with nil)

Maybe this problem is due to invalid data ? (a missing locale, media or route for example ?)

@smartynov
Copy link

I faced the same problem today. It's a newly created database (on a clean setup, after crash mentioned in #1226). It is filled with just a couple hours of data (roughly 1M records), and when I create a continuous query, it does not work and log says:

[2014/12/14 23:23:43 UTC] [INFO] (github.com/influxdb/influxdb/coordinator.(*Coordinator).RunQuery:41) Start Query: db: updog, u: root, q: SELECT pop_id, count(site_id) as poll_count, count(distinct(hoster_id)) as hoster_count
, count(distinct(hoster2_id)) as hoster2_count, mean(up) as up_mean, mean(overtime) as overtime_mean, percentile(overtime, 50) as overtime_p50, percentile(overtime, 80) as overtime_p80 FROM site_poll_history GROUP BY time(1m),
 pop_id INTO agg.pop.1m
[2014/12/14 23:23:44 UTC] [INFO] (github.com/influxdb/influxdb/coordinator.(*Coordinator).RunQuery:41) Start Query: db: updog, u: root, q: select pop_id,count(site_id) as poll_count,count(distinct(hoster_id)) as hoster_count,c
ount(distinct(hoster2_id)) as hoster2_count,mean(up) as up_mean,mean(overtime) as overtime_mean,percentile(overtime,50) as overtime_p50,percentile(overtime,80) as overtime_p80 from "site_poll_history" where (time < 14185993800
00000000) AND (time > -6795364578871345152) group by time(1m),pop_id
[2014/12/14 23:23:44 UTC] [INFO] (github.com/influxdb/influxdb/cluster.(*ShardData).Query:325) Processor chain:  AggregatorEngine > PassthroughEngine > ShardIdInserterProcessor (1) > ResponseChannelProcessor

[2014/12/14 23:23:52 UTC] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).GetShardToWriteToBySeriesAndTime:834) No matching shards for write at time 1418599320000000u, creating...
[2014/12/14 23:23:52 UTC] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).createShards:865) createShards for space agg: start: Thu Dec 11 00:00:00 +0000 UTC 2014. end: Thu Dec 18 00:00:00 +0000 UTC 2014
[2014/12/14 23:23:52 UTC] [INFO] (github.com/influxdb/influxdb/datastore.(*ShardDatastore).GetOrCreateShard:158) DATASTORE: opening or creating shard /opt/influxdb/shared/data/db/shard_db_v2/00003
[2014/12/14 23:23:52 UTC] [INFO] (github.com/influxdb/influxdb/cluster.(*ClusterConfiguration).AddShards:1090) Adding shard to agg: 3 - start: Thu Dec 11 00:00:00 +0000 UTC 2014 (1418256000). end: Thu Dec 18 00:00:00 +0000 UTC
 2014 (1418860800). isLocal: true. servers: [1]
[2014/12/14 23:23:52 UTC] [EROR] (github.com/influxdb/influxdb/coordinator.(*Coordinator).CommitSeriesData:567) COORD error writing: %!(EXTRA *errors.errorString=proto: Marshal called with nil)
[2014/12/14 23:23:52 UTC] [EROR] (github.com/influxdb/influxdb/coordinator.(*Coordinator).InterpolateValuesAndCommit:486) Couldn't write data for continuous query: %!(EXTRA *errors.errorString=proto: Marshal called with nil)
[2014/12/14 23:24:00 UTC] [EROR] (github.com/influxdb/influxdb/coordinator.(*Coordinator).CommitSeriesData:567) COORD error writing: %!(EXTRA *errors.errorString=proto: Marshal called with nil)
[2014/12/14 23:24:00 UTC] [EROR] (github.com/influxdb/influxdb/coordinator.(*Coordinator).InterpolateValuesAndCommit:486) Couldn't write data for continuous query: %!(EXTRA *errors.errorString=proto: Marshal called with nil)
[2014/12/14 23:24:09 UTC] [EROR] (github.com/influxdb/influxdb/coordinator.(*Coordinator).CommitSeriesData:567) COORD error writing: %!(EXTRA *errors.errorString=proto: Marshal called with nil)
[2014/12/14 23:24:09 UTC] [EROR] (github.com/influxdb/influxdb/coordinator.(*Coordinator).InterpolateValuesAndCommit:486) Couldn't write data for continuous query: %!(EXTRA *errors.errorString=proto: Marshal called with nil)
[2014/12/14 23:24:17 UTC] [EROR] (github.com/influxdb/influxdb/coordinator.(*Coordinator).CommitSeriesData:567) COORD error writing: %!(EXTRA *errors.errorString=proto: Marshal called with nil)
[2014/12/14 23:24:17 UTC] [EROR] (github.com/influxdb/influxdb/coordinator.(*Coordinator).InterpolateValuesAndCommit:486) Couldn't write data for continuous query: %!(EXTRA *errors.errorString=proto: Marshal called with nil)
[2014/12/14 23:24:26 UTC] [EROR] (github.com/influxdb/influxdb/coordinator.(*Coordinator).CommitSeriesData:567) COORD error writing: %!(EXTRA *errors.errorString=proto: Marshal called with nil)

@smartynov
Copy link

Well, the funny part is that this continuous query started working after re-creation. I dropped both continuous query and the series, and created it again:

[2014/12/14 23:41:17 UTC] [INFO] (github.com/influxdb/influxdb/coordinator.(*Coordinator).RunQuery:41) Start Query: db: updog, u: root, q: drop continuous query 1
[2014/12/14 23:44:10 UTC] [INFO] (github.com/influxdb/influxdb/coordinator.(*Coordinator).RunQuery:41) Start Query: db: updog, u: root, q: drop series agg.pop.1m
[2014/12/14 23:45:01 UTC] [INFO] (github.com/influxdb/influxdb/coordinator.(*Coordinator).RunQuery:41) Start Query: db: updog, u: root, q: select count(pop_id) from /.*/
[2014/12/14 23:45:01 UTC] [INFO] (github.com/influxdb/influxdb/cluster.(*ShardData).Query:325) Processor chain:  PassthroughEngine > ShardIdInserterProcessor (1) > ResponseChannelProcessor

[2014/12/14 23:45:58 UTC] [INFO] (github.com/influxdb/influxdb/coordinator.(*Coordinator).RunQuery:41) Start Query: db: updog, u: root, q: select pop_id,count(site_id) as poll_count,mean(up) as up_mean,mean(overtime) as overti
me_mean,percentile(overtime,50) as overtime_p50,percentile(overtime,80) as overtime_p80 from "site_poll_history" group by time(1m),pop_id into agg.pop.1m
[2014/12/14 23:45:58 UTC] [INFO] (github.com/influxdb/influxdb/coordinator.(*Coordinator).RunQuery:41) Start Query: db: updog, u: root, q: select pop_id,count(site_id) as poll_count,mean(up) as up_mean,mean(overtime) as overti
me_mean,percentile(overtime,50) as overtime_p50,percentile(overtime,80) as overtime_p80 from "site_poll_history" where (time < 1418600700000000000) AND (time > -6795364578871345152) group by time(1m),pop_id
[2014/12/14 23:45:58 UTC] [INFO] (github.com/influxdb/influxdb/cluster.(*ShardData).Query:325) Processor chain:  AggregatorEngine > PassthroughEngine > ShardIdInserterProcessor (1) > ResponseChannelProcessor

[2014/12/14 23:45:59 UTC] [INFO] (github.com/influxdb/influxdb/coordinator.(*Coordinator).RunQuery:41) Start Query: db: updog, u: root, q: select pop_id,count(site_id) as poll_count,mean(up) as up_mean,mean(overtime) as overti
me_mean,percentile(overtime,50) as overtime_p50,percentile(overtime,80) as overtime_p80 from "site_poll_history" where (time < 1418600700000000000) AND (time > 1418600460000000000) group by time(1m),pop_id
[2014/12/14 23:45:59 UTC] [INFO] (github.com/influxdb/influxdb/cluster.(*ShardData).Query:325) Processor chain:  AggregatorEngine > PassthroughEngine > ShardIdInserterProcessor (1) > ResponseChannelProcessor

No more errors in log, it works as expected.

Probably the reason for this is related to shard creation. When I was creating CQ first time, there were no shard existing suitable for this series. And the second time shard already existed (I did not deleted the shard, just series).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants