Skip to content

Commit

Permalink
Improve Telemetry/Stats definitions and handling
Browse files Browse the repository at this point in the history
  • Loading branch information
@cabol
cabol committed Mar 29, 2023
1 parent d6159da commit 0fb0452
Show file tree
Hide file tree
Showing 32 changed files with 827 additions and 585 deletions.
14 changes: 6 additions & 8 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -66,10 +66,12 @@ file:
def deps do
[
{:nebulex, "~> 3.0"},
#=> When using the local cache adapter
{:nebulex_adapters_local, "~> 3.0"},
{:decorator, "~> 1.4"}, #=> When using Caching Annotations
{:telemetry, "~> 1.0"}, #=> When using the Telemetry events (Nebulex stats)
{:shards, "~> 1.0"}, #=> When using :shards as backend
#=> When using Caching Annotations
{:decorator, "~> 1.4"},
#=> When using the Telemetry events (Nebulex stats)
{:telemetry, "~> 1.0"}
]
end
```
Expand All @@ -85,18 +87,14 @@ makes all dependencies optional. For example:
you have to add `:telemetry` to the dependency list.
See [telemetry guide][telemetry].

* For intensive workloads, you may want to use `:shards` as the backend for
the local adapter and having partitioned tables. In such a case, you have
to add `:shards` to the dependency list.

[telemetry]: http://hexdocs.pm/nebulex/telemetry.html

Then run `mix deps.get` in your shell to fetch the dependencies. If you want to
use another cache adapter, just choose the proper dependency from the table
above.

Finally, in the cache definition, you will need to specify the `adapter:`
respective to the chosen dependency. For the local built-in cache it is:
respective to the chosen dependency. For the local cache would be:

```elixir
defmodule MyApp.Cache do
Expand Down
14 changes: 7 additions & 7 deletions guides/getting-started.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -418,9 +418,9 @@ iex> Blog.Cache.count_all(spec)
_num_of_matched_entries
```

> The previous example assumes you are using the built-in local adapter.
> The previous example assumes you are using `Nebulex.Adapters.Local` adapter.
Also, if you are using the built-in local adapter, you can use the queries
Also, if you are using `Nebulex.Adapters.Local` adapter, you can use the queries
`:expired` and `:unexpired` too, like so:

```elixir
Expand All @@ -443,8 +443,8 @@ _num_of_removed_entries
```

And just like `count_all/2`, you can also provide a custom query to delete only
the matched entries, or if you are using the built-in local adapter you can also
use the queries `:expired` and `:unexpired`. For example:
the matched entries, or if you are using `Nebulex.Adapters.Local` adapter, you
can also use the queries `:expired` and `:unexpired`. For example:

```elixir
iex> expired_entries = Blog.Cache.delete_all(:expired)
Expand All @@ -460,7 +460,7 @@ iex> Blog.Cache.delete_all(spec)
_num_of_matched_entries
```

> These examples assumes you are using the built-in local adapter.
> These examples assumes you are using `Nebulex.Adapters.Local` adapter.
### Stream all entries from cache matching the given query

Expand Down Expand Up @@ -582,8 +582,8 @@ mix nbx.gen.cache -c Blog.NearCache -a Nebulex.Adapters.Multilevel
```

By default, the command generates a 2-level near-cache topology. The first
level or `L1` using the built-in local adapter, and the second one or `L2`
using the built-in partitioned adapter.
level or `L1` using `Nebulex.Adapters.Local` adapter, and the second one or `L2`
using `Nebulex.Adapters.Partitioned` adapter.

The generated cache module `lib/blog/near_cache.ex`:

Expand Down
60 changes: 28 additions & 32 deletions guides/telemetry.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,16 @@ events:
adapter before an adapter callback is executed.

* Measurement: `%{system_time: System.monotonic_time()}`
* Metadata: `%{adapter_meta: map, function_name: atom, args: [term]}`
* Metadata:

```elixir
%{
adapter_meta: map,
function_name: atom,
args: [term],
extra_metadata: map
}
```

* `[:my_app, :cache, :command, :stop]` - Dispatched by the underlying cache
adapter after an adapter callback has been successfully executed.
Expand All @@ -48,6 +57,7 @@ events:
adapter_meta: map,
function_name: atom,
args: [term],
extra_metadata: map,
result: term
}
```
Expand All @@ -64,6 +74,7 @@ events:
adapter_meta: map,
function_name: atom,
args: [term],
extra_metadata: map,
kind: :error | :exit | :throw,
reason: term,
stacktrace: term
Expand Down Expand Up @@ -120,7 +131,7 @@ Telemetry.Metrics.summary(
tag_values:
&Map.merge(&1, %{
cache: &1.adapter_meta.cache,
adapter: &1.adapter_meta.cache.__adapter__()
adapter: &1.adapter_meta.adapter
})
)
```
Expand All @@ -133,7 +144,7 @@ transformation on the event metadata in order to get to the values we need.
Each adapter is responsible for providing stats by implementing
`Nebulex.Adapter.Stats` behaviour. However, Nebulex provides a simple default
implementation using [Erlang counters][erl_counters], which is used by
the built-in local adapter. The local adapter uses
`Nebulex.Adapters.Local` adapter. The local adapter uses
`Nebulex.Telemetry.StatsHandler` to aggregate the stats and keep
them updated, therefore, it requires the Telemetry events are dispatched
by the adapter, otherwise, it won't work properly.
Expand Down Expand Up @@ -201,8 +212,7 @@ defmodule MyApp.Telemetry do
last_value("my_app.cache.stats.misses", tags: [:cache]),
last_value("my_app.cache.stats.writes", tags: [:cache]),
last_value("my_app.cache.stats.updates", tags: [:cache]),
last_value("my_app.cache.stats.evictions", tags: [:cache]),
last_value("my_app.cache.stats.expirations", tags: [:cache])
last_value("my_app.cache.stats.evictions", tags: [:cache])
]
end

Expand All @@ -225,24 +235,24 @@ children = [
]
```

Now start an IEx session and call the server:
Now start an IEx session and make some cache calls:

```
iex(1)> MyApp.Cache.get 1
iex(1)> MyApp.Cache.get! 1
nil
iex(2)> MyApp.Cache.put 1, 1, ttl: 10
iex(2)> MyApp.Cache.put! 1, 1, ttl: 10
:ok
iex(3)> MyApp.Cache.get 1
iex(3)> MyApp.Cache.get! 1
1
iex(4)> MyApp.Cache.put 2, 2
iex(4)> MyApp.Cache.put! 2, 2
:ok
iex(5)> MyApp.Cache.delete 2
iex(5)> MyApp.Cache.delete! 2
:ok
iex(6)> Process.sleep(20)
:ok
iex(7)> MyApp.Cache.get 1
iex(7)> MyApp.Cache.get! 1
nil
iex(2)> MyApp.Cache.replace 1, 11
iex(2)> MyApp.Cache.replace! 1, 11
true
```

Expand All @@ -251,7 +261,7 @@ and you should see something like the following output:
```
[Telemetry.Metrics.ConsoleReporter] Got new event!
Event name: my_app.cache.stats
All measurements: %{evictions: 2, expirations: 1, hits: 1, misses: 2, updates: 1, writes: 2}
All measurements: %{evictions: 2, hits: 1, misses: 2, updates: 1, writes: 2}
All metadata: %{cache: MyApp.Cache}
Metric measurement: :hits (last_value)
Expand All @@ -273,10 +283,6 @@ Tag values: %{cache: MyApp.Cache}
Metric measurement: :evictions (last_value)
With value: 2
Tag values: %{cache: MyApp.Cache}
Metric measurement: :expirations (last_value)
With value: 1
Tag values: %{cache: MyApp.Cache}
```

### Custom metrics
Expand Down Expand Up @@ -325,8 +331,7 @@ defp metrics do
last_value("my_app.cache.stats.misses", tags: [:cache, :node]),
last_value("my_app.cache.stats.writes", tags: [:cache, :node]),
last_value("my_app.cache.stats.updates", tags: [:cache, :node]),
last_value("my_app.cache.stats.evictions", tags: [:cache, :node]),
last_value("my_app.cache.stats.expirations", tags: [:cache, :node]),
last_value("my_app.cache.stats.evictions", tags: [:cache, :node])

# Nebulex custom Metrics
last_value("my_app.cache.size.value", tags: [:cache, :node])
Expand All @@ -339,7 +344,7 @@ If you start an IEx session like previously, you should see the new metric too:
```
[Telemetry.Metrics.ConsoleReporter] Got new event!
Event name: my_app.cache.stats
All measurements: %{evictions: 0, expirations: 0, hits: 0, misses: 0, updates: 0, writes: 0}
All measurements: %{evictions: 0, hits: 0, misses: 0, updates: 0, writes: 0}
All metadata: %{cache: MyApp.Cache, node: :nonode@nohost}
Metric measurement: :hits (last_value)
Expand All @@ -362,10 +367,6 @@ Metric measurement: :evictions (last_value)
With value: 0
Tag values: %{cache: MyApp.Cache, node: :nonode@nohost}
Metric measurement: :expirations (last_value)
With value: 0
Tag values: %{cache: MyApp.Cache, node: :nonode@nohost}
[Telemetry.Metrics.ConsoleReporter] Got new event!
Event name: my_app.cache.size
All measurements: %{value: 0}
Expand Down Expand Up @@ -408,8 +409,8 @@ Then, when you run `MyApp.Multilevel.stats()` you get something like:
```elixir
%Nebulex.Stats{
measurements: %{
l1: %{evictions: 0, expirations: 0, hits: 0, misses: 0, updates: 0, writes: 0},
l2: %{evictions: 0, expirations: 0, hits: 0, misses: 0, updates: 0, writes: 0}
l1: %{evictions: 0, hits: 0, misses: 0, updates: 0, writes: 0},
l2: %{evictions: 0, hits: 0, misses: 0, updates: 0, writes: 0}
},
metadata: %{
l1: %{
Expand Down Expand Up @@ -459,11 +460,6 @@ metrics in this way:
measurement: &get_in(&1, [:l1, :evictions]),
tags: [:cache]
),
last_value("my_app.cache.stats.l1.expirations",
event_name: "my_app.cache.stats",
measurement: &get_in(&1, [:l1, :expirations]),
tags: [:cache]
),

# L2 metrics
last_value("my_app.cache.stats.l2.hits",
Expand Down
82 changes: 4 additions & 78 deletions lib/nebulex/adapter.ex
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,6 @@ defmodule Nebulex.Adapter do
Specifies the minimal API required from adapters.
"""

alias Nebulex.Telemetry

@typedoc "Adapter"
@type t :: module

Expand All @@ -20,7 +18,7 @@ defmodule Nebulex.Adapter do
* `:adapter` - The defined cache adapter.
"""
@type adapter_meta :: %{optional(term) => term}
@type adapter_meta() :: %{optional(term) => term}

## Callbacks

Expand All @@ -33,7 +31,7 @@ defmodule Nebulex.Adapter do
Initializes the adapter supervision tree by returning the children
and adapter metadata.
"""
@callback init(config :: keyword) :: {:ok, :supervisor.child_spec(), adapter_meta}
@callback init(config :: keyword()) :: {:ok, :supervisor.child_spec(), adapter_meta()}

# Define optional callbacks
@optional_callbacks __before_compile__: 1
Expand All @@ -53,7 +51,7 @@ defmodule Nebulex.Adapter do
Nebulex.Adapter.lookup_meta(cache.get_dynamic_cache())
"""
@spec lookup_meta(atom | pid) :: {:ok, adapter_meta} | {:error, Nebulex.Error.t()}
@spec lookup_meta(atom() | pid()) :: {:ok, adapter_meta()} | {:error, Nebulex.Error.t()}
defdelegate lookup_meta(name_or_pid), to: Nebulex.Cache.Registry, as: :lookup

@doc """
Expand All @@ -62,82 +60,10 @@ defmodule Nebulex.Adapter do
It expects a name or a PID representing the cache.
"""
@spec with_meta(atom | pid, (adapter_meta -> term)) :: term | {:error, Nebulex.Error.t()}
@spec with_meta(atom() | pid(), (adapter_meta() -> any())) :: any() | {:error, Nebulex.Error.t()}
def with_meta(name_or_pid, fun) do
with {:ok, adapter_meta} <- lookup_meta(name_or_pid) do
fun.(adapter_meta)
end
end

# FIXME: ExCoveralls does not mark most of this section as covered
# coveralls-ignore-start

@doc """
Helper macro for the adapters so they can add the logic for emitting the
recommended Telemetry events.
See the built-in adapters for more information on how to use this macro.
"""
defmacro defspan(fun, opts \\ [], do: block) do
{name, [adapter_meta | args_tl], as, [_ | as_args_tl] = as_args} = build_defspan(fun, opts)

quote do
def unquote(name)(unquote_splicing(as_args))

def unquote(name)(%{telemetry: false} = unquote(adapter_meta), unquote_splicing(args_tl)) do
unquote(block)
end

def unquote(name)(unquote_splicing(as_args)) do
metadata = %{
adapter_meta: unquote(adapter_meta),
function_name: unquote(as),
args: unquote(as_args_tl)
}

Telemetry.span(
unquote(adapter_meta).telemetry_prefix ++ [:command],
metadata,
fn ->
result =
unquote(name)(
Map.merge(unquote(adapter_meta), %{telemetry: false, in_span?: true}),
unquote_splicing(as_args_tl)
)

{result, Map.put(metadata, :result, result)}
end
)
end
end
end

## Private Functions

defp build_defspan(ast, opts) when is_list(opts) do
{name, args} =
case Macro.decompose_call(ast) do
{_, _} = parts -> parts
_ -> raise ArgumentError, "invalid syntax in defspan #{Macro.to_string(ast)}"
end

as = Keyword.get(opts, :as, name)
as_args = build_as_args(args)

{name, args, as, as_args}
end

defp build_as_args(args) do
for {arg, idx} <- Enum.with_index(args) do
arg
|> Macro.to_string()
|> build_as_arg({arg, idx})
end
end

# sobelow_skip ["DOS.BinToAtom"]
defp build_as_arg("_" <> _, {{_e1, e2, e3}, idx}), do: {:"var#{idx}", e2, e3}
defp build_as_arg(_, {arg, _idx}), do: arg

# coveralls-ignore-stop
end
Loading

0 comments on commit 0fb0452

Please sign in to comment.