Skip to content

Commit 1b523e2

Browse files
committed
Add caching and pagination examples
1 parent f1ab96f commit 1b523e2

File tree

7 files changed

+211
-18
lines changed

7 files changed

+211
-18
lines changed

Gemfile

+3
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,9 @@ gem 'jbuilder', '~> 2.5'
2828
# gem 'rack-cors'
2929

3030
gem 'jsonapi-resources'
31+
# gem 'jsonapi-resources', '~>0.10.0.beta7'
32+
33+
gem 'faker', group: [:development, :test]
3134

3235
# Reduces boot times through caching; required in config/boot.rb
3336
gem 'bootsnap', '>= 1.1.0', require: false

Gemfile.lock

+19-17
Original file line numberDiff line numberDiff line change
@@ -47,18 +47,18 @@ GEM
4747
archive-zip (0.12.0)
4848
io-like (~> 0.3.0)
4949
arel (9.0.0)
50-
bindex (0.7.0)
50+
bindex (0.8.1)
5151
bootsnap (1.4.4)
5252
msgpack (~> 1.0)
5353
builder (3.2.3)
5454
byebug (11.0.1)
55-
capybara (3.19.1)
55+
capybara (3.25.0)
5656
addressable
5757
mini_mime (>= 0.1.3)
5858
nokogiri (~> 1.8)
5959
rack (>= 1.6.0)
6060
rack-test (>= 0.6.3)
61-
regexp_parser (~> 1.2)
61+
regexp_parser (~> 1.5)
6262
xpath (~> 3.2)
6363
childprocess (1.0.1)
6464
rake (< 13.0)
@@ -68,15 +68,17 @@ GEM
6868
concurrent-ruby (1.1.5)
6969
crass (1.0.4)
7070
erubi (1.8.0)
71-
ffi (1.10.0)
71+
faker (1.9.6)
72+
i18n (>= 0.7)
73+
ffi (1.11.1)
7274
globalid (0.4.2)
7375
activesupport (>= 4.2.0)
7476
i18n (1.6.0)
7577
concurrent-ruby (~> 1.0)
7678
io-like (0.3.0)
77-
jbuilder (2.9.0)
79+
jbuilder (2.9.1)
7880
activesupport (>= 4.2.0)
79-
jsonapi-resources (0.9.7)
81+
jsonapi-resources (0.9.10)
8082
activerecord (>= 4.1)
8183
concurrent-ruby
8284
railties (>= 4.1)
@@ -93,14 +95,14 @@ GEM
9395
mimemagic (~> 0.3.2)
9496
method_source (0.9.2)
9597
mimemagic (0.3.3)
96-
mini_mime (1.0.1)
98+
mini_mime (1.0.2)
9799
mini_portile2 (2.4.0)
98100
minitest (5.11.3)
99-
msgpack (1.2.10)
100-
nio4r (2.3.1)
101+
msgpack (1.3.0)
102+
nio4r (2.4.0)
101103
nokogiri (1.10.3)
102104
mini_portile2 (~> 2.4.0)
103-
public_suffix (3.0.3)
105+
public_suffix (3.1.1)
104106
puma (3.12.1)
105107
rack (2.0.7)
106108
rack-test (1.1.0)
@@ -133,9 +135,9 @@ GEM
133135
rb-fsevent (0.10.3)
134136
rb-inotify (0.10.0)
135137
ffi (~> 1.0)
136-
regexp_parser (1.4.0)
138+
regexp_parser (1.5.1)
137139
ruby_dep (1.5.0)
138-
rubyzip (1.2.2)
140+
rubyzip (1.2.3)
139141
sass (3.7.4)
140142
sass-listen (~> 4.0.0)
141143
sass-listen (4.0.0)
@@ -147,11 +149,10 @@ GEM
147149
sprockets (>= 2.8, < 4.0)
148150
sprockets-rails (>= 2.0, < 4.0)
149151
tilt (>= 1.1, < 3)
150-
selenium-webdriver (3.142.2)
152+
selenium-webdriver (3.142.3)
151153
childprocess (>= 0.5, < 2.0)
152154
rubyzip (~> 1.2, >= 1.2.2)
153-
spring (2.0.2)
154-
activesupport (>= 4.2)
155+
spring (2.1.0)
155156
spring-watcher-listen (2.0.1)
156157
listen (>= 2.7, < 4.0)
157158
spring (>= 1.2, < 3.0)
@@ -173,9 +174,9 @@ GEM
173174
activemodel (>= 5.0)
174175
bindex (>= 0.4.0)
175176
railties (>= 5.0)
176-
websocket-driver (0.7.0)
177+
websocket-driver (0.7.1)
177178
websocket-extensions (>= 0.1.0)
178-
websocket-extensions (0.1.3)
179+
websocket-extensions (0.1.4)
179180
xpath (3.2.0)
180181
nokogiri (~> 1.8)
181182

@@ -187,6 +188,7 @@ DEPENDENCIES
187188
byebug
188189
capybara (>= 2.15)
189190
chromedriver-helper
191+
faker
190192
jbuilder (~> 2.5)
191193
jsonapi-resources
192194
listen (>= 3.0.5, < 3.2)

README.md

+151-1
Original file line numberDiff line numberDiff line change
@@ -306,4 +306,154 @@ curl -i -H "Accept: application/vnd.api+json" "http://localhost:3000/contacts?in
306306
Test a validation Error
307307
```bash
308308
curl -i -H "Accept: application/vnd.api+json" -H 'Content-Type:application/vnd.api+json' -X POST -d '{ "data": { "type": "contacts", "attributes": { "name-first": "John Doe", "email": "[email protected]" } } }' http://localhost:3000/contacts
309-
```
309+
```
310+
311+
## Handling More Data
312+
313+
The earlier responses seem pretty snappy, but they are not really returning a lot of data. In a real world system there will be a lot more data. Lets mock some with the faker gem.
314+
315+
### Add fake data for testing
316+
317+
Add the `faker` gem to your Gemfile
318+
319+
```ruby
320+
gem 'faker', group: [:development, :test]
321+
```
322+
323+
And add some seed data using the seeds file
324+
325+
```ruby
326+
# This file should contain all the record creation needed to seed the database with its default values.
327+
# The data can then be loaded with the rails db:seed command (or created alongside the database with db:setup).
328+
#
329+
# Examples:
330+
#
331+
# movies = Movie.create([{ name: 'Star Wars' }, { name: 'Lord of the Rings' }])
332+
# Character.create(name: 'Luke', movie: movies.first)
333+
334+
contacts = []
335+
20000.times do
336+
contacts << Contact.create({
337+
name_first: Faker::Name.first_name,
338+
name_last: Faker::Name.last_name,
339+
email: Faker::Internet.safe_email,
340+
twitter: "@#{Faker::Internet.user_name}"
341+
})
342+
end
343+
344+
contacts.each do |contact|
345+
contact.phone_numbers.create({
346+
name: 'cell',
347+
phone_number: Faker::PhoneNumber.cell_phone
348+
})
349+
350+
contact.phone_numbers.create({
351+
name: 'home',
352+
phone_number: Faker::PhoneNumber.phone_number
353+
})
354+
end
355+
356+
```
357+
358+
Now lets add the seed data (note this may run for a while):
359+
360+
```bash
361+
bundle install
362+
rails db:seed
363+
```
364+
365+
### Large requests take to long to complete
366+
367+
Now if we query our contacts we will get a large (20K contacts) dataset back, and it may run for many seconds (about 8 on my system)
368+
369+
```bash
370+
curl -i -H "Accept: application/vnd.api+json" "http://localhost:3000/contacts"
371+
```
372+
373+
### Options
374+
375+
There are some things we can do to work around this. First we should add a config file to our initializers. Add a file named `jsonapi_resources.rb` to the `config/initializers` directory and add this:
376+
377+
```ruby
378+
JSONAPI.configure do |config|
379+
# Config setting will go here
380+
end
381+
```
382+
383+
#### Caching
384+
385+
We can enable caching so the next request will not require the system to process all 20K records again.
386+
387+
We first need to turn on caching for the rails portion of the application with the following:
388+
389+
```bash
390+
rails dev:cache
391+
```
392+
393+
To enable caching of JSONAPI responses we need to specify which cache to use (and in version v0.10.x and later that we want all resources cached by default). So add the following to the initializer you created earlier:
394+
395+
```ruby
396+
JSONAPI.configure do |config|
397+
config.resource_cache = Rails.cache
398+
# The following option works in versions v0.10 and later
399+
#config.default_caching = true
400+
end
401+
```
402+
403+
If using an earlier version than v0.10.x we need to enable caching for each resource type we want the system to cache. Add the following line to the `contacts` ressource:
404+
405+
```ruby
406+
class ContactResource < JSONAPI::Resource
407+
caching
408+
#...
409+
end
410+
```
411+
412+
If we restart the application and make the same request it will still take the same amount of time (actually a tiny bit more as the resources are added to the cache). However if we perform the same request the time should drop significantly, going from ~8s to ~1.6s on my system for the same 20K contacts.
413+
414+
We might be able to live with performance of the cached results, but we should plan for the worst case. So we need another solution to keep our responses snappy.
415+
416+
#### Pagination
417+
418+
Instead of returning the full result set when the user asks for it, we can break it into smaller pages of data. That way the server never needs to serialize every resource in the system at once.
419+
420+
We can add pagination with a config option in the initializer. Add the following to `config/initializers/jsonapi_resources.rb`:
421+
422+
```ruby
423+
JSONAPI.configure do |config|
424+
config.resource_cache = Rails.cache
425+
# config.default_caching = true
426+
427+
# Options are :none, :offset, :paged, or a custom paginator name
428+
config.default_paginator = :paged # default is :none
429+
430+
config.default_page_size = 50 # default is 10
431+
config.maximum_page_size = 100 # default is 20
432+
end
433+
```
434+
435+
Restart the system and try the request again:
436+
437+
```bash
438+
curl -i -H "Accept: application/vnd.api+json" "http://localhost:3000/contacts"
439+
```
440+
441+
442+
Now we only get the first 50 contacts back, and the request is much faster (about 80ms). And you will now see a `links` key with links to get the remaining resources in your set. This should look like this:
443+
444+
```json
445+
{
446+
"data":[...],
447+
"links": {
448+
"first":"http://localhost:3000/contacts?page%5Bnumber%5D=1&page%5Bsize%5D=50",
449+
"next":"http://localhost:3000/contacts?page%5Bnumber%5D=2&page%5Bsize%5D=50",
450+
"last":"http://localhost:3000/contacts?page%5Bnumber%5D=401&page%5Bsize%5D=50",
451+
}
452+
}
453+
```
454+
455+
This will allow your client to iterate over the `next` links to fetch the full results set without putting extreme pressure on your server.
456+
457+
The `default_page_size` setting is used if the request does not specify a size, and the `maximum_page_size` is used to limit the size the client may request.
458+
459+
*Note:* The default page sizes are very conservative. There is significant overhead in making many small requests, and tuning the page sizes should be considered essential.

app/resources/contact_resource.rb

+2
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
class ContactResource < JSONAPI::Resource
2+
caching
3+
24
attributes :name_first, :name_last, :email, :twitter
35
has_many :phone_numbers
46
end

app/resources/phone_number_resource.rb

+2
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
class PhoneNumberResource < JSONAPI::Resource
2+
caching
3+
24
attributes :name, :phone_number
35
has_one :contact
46

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
JSONAPI.configure do |config|
2+
config.resource_cache = Rails.cache
3+
4+
# For v0.10.x and later you can use
5+
# config.default_caching = true
6+
7+
# Options are :none, :offset, :paged, or a custom paginator name
8+
config.default_paginator = :paged # default is :none
9+
10+
config.default_page_size = 50 # default is 10
11+
config.maximum_page_size = 100 # default is 20
12+
end

db/seeds.rb

+22
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,25 @@
55
#
66
# movies = Movie.create([{ name: 'Star Wars' }, { name: 'Lord of the Rings' }])
77
# Character.create(name: 'Luke', movie: movies.first)
8+
9+
contacts = []
10+
20000.times do
11+
contacts << Contact.create({
12+
name_first: Faker::Name.first_name,
13+
name_last: Faker::Name.last_name,
14+
email: Faker::Internet.safe_email,
15+
twitter: "@#{Faker::Internet.user_name}"
16+
})
17+
end
18+
19+
contacts.each do |contact|
20+
contact.phone_numbers.create({
21+
name: 'cell',
22+
phone_number: Faker::PhoneNumber.cell_phone
23+
})
24+
25+
contact.phone_numbers.create({
26+
name: 'home',
27+
phone_number: Faker::PhoneNumber.phone_number
28+
})
29+
end

0 commit comments

Comments
 (0)