Recommended approach for creating views in live environment

[subin94]

In a couchdb Server containing more than 1500 databases(Cluster with 4 nodes), tried creating a design document with 4 views in one database(with doc_count more than 450K). Before creating this design doc more than 4GiB RAM was available. But once the view's indexing process started it consumed entire RAM in all nodes.

Is there any recommended way to create heavy views in live environments without causing downtime.

Thanks.

[janl]

building views is normally does not put much strain on RAM, so we need to look for another reason:

• do you have very large documents (multi-MB)
• do you have documents with conflicts (if yes, how many)
• do your map and/or reduce functions make heavy use of loops?
• do you have a custom reduce function? (if yes, please share it)

[subin94]

• do you have very large documents (multi-MB)

Average size of documents used in this design document views: 4KB per doc
No document in the eniter database is greater than 50KB

• do you have documents with conflicts (if yes, how many)

Yes. 260 conflicted documents satisfying this map function. 1127 conflicting documents in the entire database.

• do your map and/or reduce functions make heavy use of loops?

No. Design document is something similar to this { "_id": "_design/design_doc_name", "_rev": "1-xxxxxxx", "views": { "view1": { "map": "function (doc) {\r\n if(doc.type == \"type\" && doc.internal_key.key1 && doc.key4 && doc.internal_key.key2 && doc.internal_key.key3) {\r\n emit([doc.internal_key.key1, doc.key5, doc.internal_key.key3, doc.internal_key.key2, doc.key4], 1)\r\n }\r\n}" }, "view2": { "map": "function (doc) {\r\n if(doc.type == \"type\" && doc.internal_key.key1 && doc.key6 && doc.internal_key.key2 && doc.internal_key.key3) {\r\n emit([doc.internal_key.key1, doc.key5, doc.internal_key.key3, doc.internal_key.key2, doc.key6], 1)\r\n }\r\n}" }, "view3": { "map": "function (doc) {\r\n if(doc.type == \"type\" && doc.internal_key.key7 && doc.key4 && doc.internal_key.key2 && doc.internal_key.key3) {\r\n emit([doc.internal_key.key7, doc.key5, doc.internal_key.key3, doc.internal_key.key2, doc.key4], 1)\r\n }\r\n}" }, "view4": { "map": "function (doc) {\r\n if(doc.type == \"type\" && doc.internal_key.key7 && doc.key6 && doc.internal_key.key2 && doc.internal_key.key3) {\r\n emit([doc.internal_key.key7, doc.key5, doc.internal_key.key3, doc.internal_key.key2, doc.key6], 1)\r\n }\r\n}" } }, "language": "javascript" }

• do you have a custom reduce function? (if yes, please share it)

No Reduce function used at all.

[janl]

thanks, how many conflicts does the most conflicted doc have?

[subin94]

The most conflicted doc has 57 conflicts

No. of conflicts	No. of documents
1	619
2	76
3	407
4	10
5	3
6	4
9	1
14	2
16	1
18	1
30	1
35	1
57	1

[janl]

can you resolve all these conflicts and see if the problem persists. You should really not keep these many conflicts around.

[ermouth]

What is your CouchDB version btw, and what is q value (number of shards) for the given DB? Did OS reclaim memory after indexing finished?

[subin94]

Couchdb Version 3.1.0 and q value is 2

Did OS reclaimed memory after indexing finished?

Yes. Memory reclaimed after the task was killed.