From 4e11ffde6a3c179ca74bfc3d68ae0c584e5e43af Mon Sep 17 00:00:00 2001 From: Peter Deffebach Date: Tue, 31 Oct 2023 18:23:00 -0400 Subject: [PATCH 01/11] no more with --- docs/src/index.md | 44 ++++++++++++++--------------- src/DataFramesMeta.jl | 2 +- src/eachrow.jl | 4 +-- src/macros.jl | 54 ++++++++++++++++++------------------ src/parsing.jl | 2 +- test/byrow.jl | 10 +++---- test/dataframes.jl | 26 ++++++++--------- test/function_compilation.jl | 26 ++++++++--------- test/parsing.jl | 4 +-- 9 files changed, 86 insertions(+), 86 deletions(-) diff --git a/docs/src/index.md b/docs/src/index.md index 062aaee2..a4016cdb 100644 --- a/docs/src/index.md +++ b/docs/src/index.md @@ -17,7 +17,7 @@ In addition, DataFramesMeta provides `@rselect`, `@rselect!`, `@rorderby`, `@rsubset`, and `@rsubset!`. * `@rename` and `@rename!` for renaming columns * `@by`, for grouping and combining a data frame in a single step -* `@with`, for working with the columns of a data frame with high performance and +* `@attach`, for working with the columns of a data frame with high performance and convenient syntax * `@eachrow` and `@eachrow!` for looping through rows in data frame, again with high performance and convenient syntax. @@ -183,19 +183,19 @@ df = DataFrame(x = [1, 1, 2, 2], y = [1, 2, 101, 102]); end ``` -## `@with` +## `@attach` -`@with` creates a scope in which all symbols that appear are aliases for the columns +`@attach` creates a scope in which all symbols that appear are aliases for the columns in a DataFrame. ```julia df = DataFrame(x = 1:3, y = [2, 1, 2]) x = [2, 1, 0] -@with(df, :y .+ 1) -@with(df, :x + x) # the two x's are different +@attach(df, :y .+ 1) +@attach(df, :x + x) # the two x's are different -x = @with df begin +x = @attach df begin res = 0.0 for i in 1:length(:x) res += :x[i] * :y[i] @@ -203,12 +203,12 @@ x = @with df begin res end -@with(df, df[:x .> 1, ^(:y)]) # The ^ means leave the :y alone +@attach(df, df[:x .> 1, ^(:y)]) # The ^ means leave the :y alone ``` !!! note - `@with` creates a function, so scope within `@with` is a local scope. + `@attach` creates a function, so scope within `@attach` is a local scope. Variables in the parent can be read. Writing to variables in the parent scope differs depending on the type of scope of the parent. If the parent scope is a global scope, then a variable cannot be assigned without using the `global` keyword. @@ -216,12 +216,12 @@ end the `global` keyword is not needed to assign to that parent scope. !!! note - Because `@with` creates a function, be careful with the use of `return`. + Because `@attach` creates a function, be careful with the use of `return`. ``` function data_transform(df; returnearly = true) if returnearly - @with df begin + @attach df begin z = :x + :y return z end @@ -233,10 +233,10 @@ end end ``` - The above function will return `[4, 5, 6]` because the `return` inside the `@with` - applies to the anonymous function created by `@with`. + The above function will return `[4, 5, 6]` because the `return` inside the `@attach` + applies to the anonymous function created by `@attach`. - Given that `@eachrow` (below) is implemented with `@with`, the same caveat applies to + Given that `@eachrow` (below) is implemented with `@attach`, the same caveat applies to `@eachrow` blocks. @@ -245,7 +245,7 @@ end Act on each row of a data frame. Includes support for control flow and `begin end` blocks. Since the "environment" induced by `@eachrow df` is implicitly a single row of `df`, one uses regular operators and comparisons instead of -their elementwise counterparts as in `@with`. Does not change the input data +their elementwise counterparts as in `@attach`. Does not change the input data frame argument. `@eachrow!` is identical to `@eachrow` but acts on a data frame in-place, modifying @@ -326,8 +326,8 @@ The following macros accept `@byrow`: `@byrow` can be used in the left hand side of expressions, e.g. `@select(df, @byrow z = :x * :y)`. * `@subset`, `@subset!` and `@orderby`, with syntax of the form `@subset(df, @byrow :x > :y)` -* `@with`, where the anonymous function created by `@with` is wrapped in - `ByRow`, as in `@with(df, @byrow :x * :y)`. +* `@attach`, where the anonymous function created by `@attach` is wrapped in + `ByRow`, as in `@attach(df, @byrow :x * :y)`. To avoid writing `@byrow` multiple times when performing multiple operations, it is allowed to use `@byrow` at the beginning of a block of @@ -723,9 +723,9 @@ transform(df, [:A, "B"] => (+) => :y) will error in DataFrames. -For consistency, this restriction in the input column types also applies to `@with` +For consistency, this restriction in the input column types also applies to `@attach` and `@eachrow`. You cannot mix integer column references with `Symbol` or string column -references in `@with` and `@eachrow` in any part of the expression, but you can mix +references in `@attach` and `@eachrow` in any part of the expression, but you can mix `Symbol`s and strings. The following will fail: ```julia @@ -734,7 +734,7 @@ df = DataFrame(A = 1:3, B = [2, 1, 2]) :A = $2 end -@with df begin +@attach df begin $1 + $"A" end ``` @@ -746,7 +746,7 @@ while the following will work without error $1 + $2 end -@with df begin +@attach df begin $1 + $2 end ``` @@ -860,7 +860,7 @@ julia> @subset df $(:a => t -> t .>= 2) ``` !!! warning - The macros `@orderby` and `@with` do not transparently call underlying DataFrames.jl functions. Escaping entire transformations should be considered unstable and may change in future versions. + The macros `@orderby` and `@attach` do not transparently call underlying DataFrames.jl functions. Escaping entire transformations should be considered unstable and may change in future versions. !!! warning Row-wise macros such as `@rtransform` and `@rsubset` will not automatically wrap functions in `src => fun => dest` in `ByRow`. @@ -880,7 +880,7 @@ In summary * Regular expressions, `$(r"^a")` * Filtering column selectors, such as `$(Not(:x))` and `$(Between(:a, :z))` - The macros `@with`, `@subset`, and `@orderby` do not support multi-column selectors. + The macros `@attach`, `@subset`, and `@orderby` do not support multi-column selectors. * Advanced users of DataFramesMeta.jl and DataFrames.jl may wrap an argument entirely in `$()` to pass `src => fun => dest` pairs directly to DataFrames.jl functions. However this is discouraged and it's behavior may change in future versions. diff --git a/src/DataFramesMeta.jl b/src/DataFramesMeta.jl index a16cca5a..4d810b9f 100644 --- a/src/DataFramesMeta.jl +++ b/src/DataFramesMeta.jl @@ -11,7 +11,7 @@ using OrderedCollections: OrderedCollections @reexport using Chain # Basics: -export @with, +export @attach, @subset, @subset!, @rsubset, @rsubset!, @orderby, @rorderby, @by, @combine, diff --git a/src/eachrow.jl b/src/eachrow.jl index 8f4ce932..d9079711 100644 --- a/src/eachrow.jl +++ b/src/eachrow.jl @@ -91,7 +91,7 @@ end Includes support for control flow and `begin end` blocks. Since the "environment" induced by `@eachrow df` is implicitly a single row of `df`, use regular operators and comparisons instead of their elementwise counterparts -as in `@with`. Note that the scope within `@eachrow` is a hard scope. +as in `@attach`. Note that the scope within `@eachrow` is a hard scope. `eachrow` also supports special syntax for allocating new columns. The syntax `@newcol x::Vector{Int}` allocates a new uninitialized column `:x` with an `Vector` container @@ -246,7 +246,7 @@ end Includes support for control flow and `begin end` blocks. Since the "environment" induced by `@eachrow! df` is implicitly a single row of `df`, use regular operators and comparisons instead of their elementwise counterparts -as in `@with`. Note that the scope within `@eachrow!` is a hard scope. +as in `@attach`. Note that the scope within `@eachrow!` is a hard scope. `eachrow!` also supports special syntax for allocating new columns. The syntax `@newcol x::Vector{Int}` allocates a new uninitialized column `:x` with an `Vector` container diff --git a/src/macros.jl b/src/macros.jl index ce18a7d8..c4c05707 100644 --- a/src/macros.jl +++ b/src/macros.jl @@ -12,7 +12,7 @@ DataFrames's `source => fun => destination` syntax. ### Details -Parsing follows the same convention as other DataFramesMeta.jl macros, such as `@with`. All +Parsing follows the same convention as other DataFramesMeta.jl macros, such as `@attach`. All terms in the expression that are `Symbol`s are treated as columns in the data frame, except `Symbol`s wrapped in `^`. To use a variable representing a column name, wrap the variable in `$DOLLAR`. @@ -70,7 +70,7 @@ to indicate that the anonymous function created by DataFramesMeta to represent an operation should be applied "by-row". If an expression starts with `@byrow`, either of the form `@byrow :y = f(:x)` -in transformations or `@byrow f(:x)` in `@orderby`, `@subset`, and `@with`, +in transformations or `@byrow f(:x)` in `@orderby`, `@subset`, and `@attach`, then the anonymous function created by DataFramesMeta is wrapped in the `DataFrames.ByRow` function wrapper, which broadcasts the function so that it run on each row. @@ -144,7 +144,7 @@ Rather, `@eachrow` and `@eachrow!` return data frames. Now consider `@byrow`. `@byrow` transforms ```julia -@with df @byrow begin +@attach df @byrow begin :a * :b end ``` @@ -156,7 +156,7 @@ tempfun(a, b) = a * b tempfun.(df.a, df.b) ``` -In contrast to `@eachrow`, `@with` combined with `@byrow` returns a vector of the +In contrast to `@eachrow`, `@attach` combined with `@byrow` returns a vector of the broadcasted multiplication and not a data frame. Additionally, transformations applied using `@eachrow!` modify the input @@ -165,7 +165,7 @@ data frame. On the contrary, `@byrow` does not update columns. ```julia julia> df = DataFrame(a = [1, 2], b = [3, 4]); -julia> @with df @byrow begin +julia> @attach df @byrow begin :a = 500 end 2-element Vector{Int64}: @@ -198,7 +198,7 @@ df = DataFrame(a = [1, 2], b = [3, 4]) is not possible in Julia versions below 1.7. ``` -julia> @with df @byrow begin +julia> @attach df @byrow begin if :a == 1 5 else @@ -209,7 +209,7 @@ julia> @with df @byrow begin 5 10 -julia> @with df @. begin +julia> @attach df @. begin if :a == 1 5 else @@ -225,7 +225,7 @@ julia> @with df @. begin ```julia julia> df = DataFrame(a = [1, 2], b = [3, 4]); -julia> @with df @byrow :x + [5, 6] +julia> @attach df @byrow :x + [5, 6] ``` will error, because the `:x` in the above expression refers @@ -234,7 +234,7 @@ julia> @with df @byrow :x + [5, 6] On the other hand ```julia -@with df @. :x + [5, 6] +@attach df @. :x + [5, 6] ``` will succeed, as `df.x` is a 2-element vector as is `[5, 6]`. @@ -257,10 +257,10 @@ julia> function expensive() return 1 end; -julia> @time @with df @byrow :a + expensive(); +julia> @time @attach df @byrow :a + expensive(); 1.037073 seconds (51.67 k allocations: 3.035 MiB, 3.19% compilation time) -julia> @time @with df :a .+ expensive(); +julia> @time @attach df :a .+ expensive(); 0.539900 seconds (110.67 k allocations: 6.525 MiB, 7.05% compilation time) ``` @@ -271,7 +271,7 @@ julia> @time @with df :a .+ expensive(); `@byrow` or in DataFramesMeta.jl at large. The best solution is simply ``` -@with df begin +@attach df begin x = expensive() :a + x end @@ -531,7 +531,7 @@ end ############################################################################## ## -## @with +## @attach ## ############################################################################## @@ -555,9 +555,9 @@ function with_helper(d, body) end """ - @with(d, expr) + @attach(d, expr) -`@with` allows DataFrame columns keys to be referenced as symbols. +`@attach` allows DataFrame columns keys to be referenced as symbols. ### Arguments @@ -566,7 +566,7 @@ end ### Details -`@with` works by parsing the expression body for all columns indicated +`@attach` works by parsing the expression body for all columns indicated by symbols (e.g. `:colA`). Then, a function is created that wraps the body and passes the columns as function arguments. This function is then called. Operations are efficient because: @@ -577,7 +577,7 @@ then called. Operations are efficient because: The following ```julia -@with(d, :a .+ :b .+ 1) +@attach(d, :a .+ :b .+ 1) ``` becomes @@ -591,8 +591,8 @@ If an expression is wrapped in `^(expr)`, `expr` gets passed through untouched. If an expression is wrapped in `$DOLLAR(expr)`, the column is referenced by the variable `expr` rather than a symbol. -If the expression provide to `@with` begins with `@byrow`, the function -created by the `@with` block is broadcasted along the columns of the +If the expression provide to `@attach` begins with `@byrow`, the function +created by the `@attach` block is broadcasted along the columns of the data frame. ### Examples @@ -606,19 +606,19 @@ julia> df = DataFrame(x = 1:3, y = [2, 1, 2]); julia> x = [2, 1, 0]; -julia> @with(df, :y .+ 1) +julia> @attach(df, :y .+ 1) 3-element Vector{Int64}: 3 2 3 -julia> @with(df, :x + x) +julia> @attach(df, :x + x) 3-element Vector{Int64}: 3 3 3 -julia> @with df begin +julia> @attach df begin res = 0.0 for i in 1:length(:x) res += :x[i] * :y[i] @@ -627,20 +627,20 @@ julia> @with df begin end 10.0 -julia> @with(df, df[:x .> 1, ^(:y)]) # The ^ means leave the :y alone +julia> @attach(df, df[:x .> 1, ^(:y)]) # The ^ means leave the :y alone 2-element Vector{Int64}: 1 2 julia> colref = :x; -julia> @with(df, :y + $(DOLLAR)colref) # Equivalent to df[!, :y] + df[!, colref] +julia> @attach(df, :y + $(DOLLAR)colref) # Equivalent to df[!, :y] + df[!, colref] 3-element Vector{Int64}: 3 3 5 -julia> @with df @byrow :x * :y +julia> @attach df @byrow :x * :y 3-element Vector{Int64}: 2 2 @@ -649,7 +649,7 @@ julia> @with df @byrow :x * :y ``` !!! note - `@with` creates a function, so the scope within `@with` is a local scope. + `@attach` creates a function, so the scope within `@attach` is a local scope. Variables in the parent can be read. Writing to variables in the parent scope differs depending on the type of scope of the parent. If the parent scope is a global scope, then a variable cannot be assigned without using the `global` keyword. @@ -657,7 +657,7 @@ julia> @with df @byrow :x * :y the `global` keyword is not needed to assign to that parent scope. !!! note - Using `AsTable` inside `@with` block is currently not supported. + Using `AsTable` inside `@attach` block is currently not supported. """ macro with(d, body) esc(with_helper(d, body)) diff --git a/src/parsing.jl b/src/parsing.jl index 5c48f90b..b436b71f 100644 --- a/src/parsing.jl +++ b/src/parsing.jl @@ -318,7 +318,7 @@ function fun_to_vec(ex::Expr; # classify the type of expression # :x # handled via dispatch # $:x # handled as though above - # f(:x) # requires no_dest, for `@with` and `@subset` in future + # f(:x) # requires no_dest, for `@attach` and `@subset` in future # :y = :x # Simple pair # :y = $:x # Extract and return simple pair (no function) # $:y = :x # Simple pair diff --git a/test/byrow.jl b/test/byrow.jl index 61d5dafa..27d38324 100644 --- a/test/byrow.jl +++ b/test/byrow.jl @@ -393,18 +393,18 @@ end @test d ≅ @select!(copy(df), :n1 = :i .* :g, :n2 = :i .* :g) end -@testset "@with with @byrow" begin +@testset "@attach with @byrow" begin df = DataFrame(A = 1:3, B = [2, 1, 2]) - @test @with(df, @byrow :A * 1) == df.A .* 1 - @test @with(df, @byrow :A * :B) == df.A .* df.B + @test @attach(df, @byrow :A * 1) == df.A .* 1 + @test @attach(df, @byrow :A * :B) == df.A .* df.B - t = @with df @byrow begin + t = @attach df @byrow begin :A * 1 end @test t == df.A .* 1 - t = @with df @byrow begin + t = @attach df @byrow begin :A * :B end @test t == df.A .* df.B diff --git a/test/dataframes.jl b/test/dataframes.jl index 5393b25e..7dd68409 100644 --- a/test/dataframes.jl +++ b/test/dataframes.jl @@ -558,11 +558,11 @@ end x = [2, 1, 0] - @test @with(df, :A .+ 1) == df.A .+ 1 - @test @with(df, :A .+ :B) == df.A .+ df.B - @test @with(df, :A .+ x) == df.A .+ x + @test @attach(df, :A .+ 1) == df.A .+ 1 + @test @attach(df, :A .+ :B) == df.A .+ df.B + @test @attach(df, :A .+ x) == df.A .+ x - x = @with df begin + x = @attach df begin res = 0.0 for i in 1:length(:A) res += :A[i] * :B[i] @@ -570,20 +570,20 @@ end res end idx = :A - @test @with(df, $idx .+ :B) == df.A .+ df.B + @test @attach(df, $idx .+ :B) == df.A .+ df.B idx2 = :B - @test @with(df, $idx .+ $idx2) == df.A .+ df.B - @test @with(df, $:A .+ $"B") == df.A .+ df.B + @test @attach(df, $idx .+ $idx2) == df.A .+ df.B + @test @attach(df, $:A .+ $"B") == df.A .+ df.B - @test_throws ArgumentError @with(df, :A + $2) + @test_throws ArgumentError @attach(df, :A + $2) @test x == sum(df.A .* df.B) - @test @with(df, df[:A .> 1, ^([:B, :A])]) == df[df.A .> 1, [:B, :A]] - @test @with(df, DataFrame(a = :A * 2, b = :A .+ :B)) == DataFrame(a = df.A * 2, b = df.A .+ df.B) + @test @attach(df, df[:A .> 1, ^([:B, :A])]) == df[df.A .> 1, [:B, :A]] + @test @attach(df, DataFrame(a = :A * 2, b = :A .+ :B)) == DataFrame(a = df.A * 2, b = df.A .+ df.B) - @test @with(df, :A) === df.A - @test @with(df, $:A) === df.A - @test @with(df, $"A") === df.A + @test @attach(df, :A) === df.A + @test @attach(df, $:A) === df.A + @test @attach(df, $"A") === df.A end @testset "orderby" begin diff --git a/test/function_compilation.jl b/test/function_compilation.jl index 471f07f9..27f922c4 100644 --- a/test/function_compilation.jl +++ b/test/function_compilation.jl @@ -162,29 +162,29 @@ using DataFramesMeta slowtime = @timed combine(gd, :b => (b -> testnt(b)) => AsTable) (slowtime[2] > fasttime[2]) || @warn("Slow compilation") - @test @with df (:a + :b) == [3] + @test @attach df (:a + :b) == [3] - fasttime = @timed @with df (:a + :b) - slowtime = @timed @with df ((a, b) -> a + b)(df.a, df.b) + fasttime = @timed @attach df (:a + :b) + slowtime = @timed @attach df ((a, b) -> a + b)(df.a, df.b) (slowtime[2] > fasttime[2]) || @warn("Slow compilation") - @test @with df (:a .* :b) == [2] + @test @attach df (:a .* :b) == [2] - @with df (:a .* :b) - fasttime = @timed @with df (:a .* :b) - slowtime = @timed @with df ((a, b) -> a .* b)(df.a, df.b) + @attach df (:a .* :b) + fasttime = @timed @attach df (:a .* :b) + slowtime = @timed @attach df ((a, b) -> a .* b)(df.a, df.b) (slowtime[2] > fasttime[2]) || @warn("Slow compilation") - @test @with df testfun(:a, :b) == [2] + @test @attach df testfun(:a, :b) == [2] - fasttime = @timed @with df testfun(:a, :b) - slowtime = @timed @with df ((a, b) -> testfun(a, b))(df.a, df.b) + fasttime = @timed @attach df testfun(:a, :b) + slowtime = @timed @attach df ((a, b) -> testfun(a, b))(df.a, df.b) (slowtime[2] > fasttime[2]) || @warn("Slow compilation") - @test @with df testdotfun.(:a, :b) == [2] + @test @attach df testdotfun.(:a, :b) == [2] - fasttime = @timed @with df testdotfun.(:a, :b) - slowtime = @timed @with df ((a, b) -> testdotfun.(a, b))(df.a, df.b) + fasttime = @timed @attach df testdotfun.(:a, :b) + slowtime = @timed @attach df ((a, b) -> testdotfun.(a, b))(df.a, df.b) (slowtime[2] > fasttime[2]) || @warn("Slow compilation") end end diff --git a/test/parsing.jl b/test/parsing.jl index 4723c2e3..db71be8f 100644 --- a/test/parsing.jl +++ b/test/parsing.jl @@ -124,9 +124,9 @@ end @eval df = DataFrame(a = 1) # Some errors when we dont have keyword arguments, since # the following gets parsed as (a^b) and we want - # (a, b...) in @subset and (a, b) in @with + # (a, b...) in @subset and (a, b) in @attach @test_throws MethodError @eval @rsubset df ^(true) - @test_throws LoadError @eval @with df ^(true) + @test_throws LoadError @eval @attach df ^(true) end end # module \ No newline at end of file From abbe1aeff44a8ed2536badd29354c7ec974f2fbb Mon Sep 17 00:00:00 2001 From: Peter Deffebach Date: Tue, 31 Oct 2023 18:26:25 -0400 Subject: [PATCH 02/11] no more with --- src/macros.jl | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/macros.jl b/src/macros.jl index c4c05707..cdd46d74 100644 --- a/src/macros.jl +++ b/src/macros.jl @@ -546,7 +546,7 @@ getsinglecolumn(df, s::DataFrames.ColumnIndex) = df[!, s] getsinglecolumn(df, s) = throw(ArgumentError("Only indexing with Symbols, strings and integers " * "is currently allowed with $DOLLAR")) -function with_helper(d, body) +function attach_helper(d, body) # Make body an expression to force the # complicated method of fun_to_vec # in the case of QuoteNode @@ -659,8 +659,8 @@ julia> @attach df @byrow :x * :y !!! note Using `AsTable` inside `@attach` block is currently not supported. """ -macro with(d, body) - esc(with_helper(d, body)) +macro attach(d, body) + esc(attach_helper(d, body)) end ASTABLE_RHS_ORDERBY_DOCS = """ From 25302da4a8813c0909eee43524b42956059d3034 Mon Sep 17 00:00:00 2001 From: Peter Deffebach Date: Wed, 1 Nov 2023 10:22:49 -0400 Subject: [PATCH 03/11] deprecation --- src/macros.jl | 11 +++++++++++ test/dataframes.jl | 2 +- test/deprecated.jl | 34 ++++++++++++++++++++++++++++++++++ 3 files changed, 46 insertions(+), 1 deletion(-) diff --git a/src/macros.jl b/src/macros.jl index cdd46d74..600b2d5d 100644 --- a/src/macros.jl +++ b/src/macros.jl @@ -663,6 +663,17 @@ macro attach(d, body) esc(attach_helper(d, body)) end +""" + @with(x, args...) + +Deprecated version of `@attach`, see `?@with` for details. +""" +macro with(x, args...) + @warn "`@with is deprecated, use `@with` instead." + esc(attach_helper(d, body)) +end + + ASTABLE_RHS_ORDERBY_DOCS = """ In operations, it is also allowed to use `AsTable(cols)` to work with multiple columns at once, where the columns are grouped together in a diff --git a/test/dataframes.jl b/test/dataframes.jl index 7dd68409..4d2058be 100644 --- a/test/dataframes.jl +++ b/test/dataframes.jl @@ -553,7 +553,7 @@ cr = "c" @test_throws MethodError @eval @select(df, :n = sum(Between(:i, :t))) end -@testset "with" begin +@testset "attach" begin df = DataFrame(A = 1:3, B = [2, 1, 2]) x = [2, 1, 0] diff --git a/test/deprecated.jl b/test/deprecated.jl index 126ec441..feac7741 100644 --- a/test/deprecated.jl +++ b/test/deprecated.jl @@ -198,4 +198,38 @@ end @test (@combine(gd, n = first(:i))) ≅ newdf end +@testset "with" begin + df = DataFrame(A = 1:3, B = [2, 1, 2]) + + x = [2, 1, 0] + + @test @with(df, :A .+ 1) == df.A .+ 1 + @test @with(df, :A .+ :B) == df.A .+ df.B + @test @with(df, :A .+ x) == df.A .+ x + + x = @with df begin + res = 0.0 + for i in 1:length(:A) + res += :A[i] * :B[i] + end + res + end + idx = :A + @test @with(df, $idx .+ :B) == df.A .+ df.B + idx2 = :B + @test @with(df, $idx .+ $idx2) == df.A .+ df.B + @test @with(df, $:A .+ $"B") == df.A .+ df.B + + @test_throws ArgumentError @with(df, :A + $2) + + @test x == sum(df.A .* df.B) + @test @with(df, df[:A .> 1, ^([:B, :A])]) == df[df.A .> 1, [:B, :A]] + @test @with(df, DataFrame(a = :A * 2, b = :A .+ :B)) == DataFrame(a = df.A * 2, b = df.A .+ df.B) + + @test @with(df, :A) === df.A + @test @with(df, $:A) === df.A + @test @with(df, $"A") === df.A +end + + end # module \ No newline at end of file From a0bdcca0b110914efd18bac17e20bb6779ae6553 Mon Sep 17 00:00:00 2001 From: Peter Deffebach Date: Wed, 1 Nov 2023 11:06:32 -0400 Subject: [PATCH 04/11] tests pass --- src/DataFramesMeta.jl | 2 +- src/eachrow.jl | 4 ++-- src/linqmacro.jl | 2 +- src/macros.jl | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/src/DataFramesMeta.jl b/src/DataFramesMeta.jl index 4d810b9f..c424b2ba 100644 --- a/src/DataFramesMeta.jl +++ b/src/DataFramesMeta.jl @@ -21,7 +21,7 @@ export @attach, @distinct, @rdistinct, @distinct!, @rdistinct!, @eachrow, @eachrow!, @byrow, @passmissing, @astable, @kwarg, - @based_on, @where # deprecated + @based_on, @where, @with # deprecated const DOLLAR = raw"$" diff --git a/src/eachrow.jl b/src/eachrow.jl index d9079711..d6c4ef50 100644 --- a/src/eachrow.jl +++ b/src/eachrow.jl @@ -68,7 +68,7 @@ function eachrow_helper(df, body, deprecation_warning) let $_df = $df local _N = nrow($_df) local _DF = @transform($_df, $(e_newcols...)) - $(with_helper(:_DF, :(for row = 1:_N + $(attach_helper(:_DF, :(for row = 1:_N $(eachrow_replace(e_body)) end))) _DF @@ -224,7 +224,7 @@ function eachrow!_helper(df, body) let $_df = $df local _N = nrow($_df) local _DF = @transform!($_df, $(e_newcols...)) - $(with_helper(:_DF, :(for row = 1:_N + $(attach_helper(:_DF, :(for row = 1:_N $(eachrow_replace(e_body)) end))) _DF diff --git a/src/linqmacro.jl b/src/linqmacro.jl index 9e77dd34..108b4eb5 100644 --- a/src/linqmacro.jl +++ b/src/linqmacro.jl @@ -111,7 +111,7 @@ end linq(::SymbolParameter{s}, args...) where {s} = Expr(:call, s, args...) function linq(::SymbolParameter{:with}, d, body) - with_helper(d, body) + attach_helper(d, body) end function linq(::SymbolParameter{:where}, d, args...) diff --git a/src/macros.jl b/src/macros.jl index 600b2d5d..73bcfd45 100644 --- a/src/macros.jl +++ b/src/macros.jl @@ -668,7 +668,7 @@ end Deprecated version of `@attach`, see `?@with` for details. """ -macro with(x, args...) +macro with(d, body) @warn "`@with is deprecated, use `@with` instead." esc(attach_helper(d, body)) end From fd16687dd121a0fe649be280c03f0c631c72ccad Mon Sep 17 00:00:00 2001 From: pdeffebach <23196228+pdeffebach@users.noreply.github.com> Date: Wed, 1 Nov 2023 11:07:04 -0400 Subject: [PATCH 05/11] Update src/macros.jl MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Bogumił Kamiński --- src/macros.jl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/macros.jl b/src/macros.jl index 73bcfd45..43d53d5a 100644 --- a/src/macros.jl +++ b/src/macros.jl @@ -669,7 +669,7 @@ end Deprecated version of `@attach`, see `?@with` for details. """ macro with(d, body) - @warn "`@with is deprecated, use `@with` instead." + @warn "`@with is deprecated, use `@attach` instead." esc(attach_helper(d, body)) end From d274cc613edf1b019d985a0915e02331ee2ef6b1 Mon Sep 17 00:00:00 2001 From: pdeffebach <23196228+pdeffebach@users.noreply.github.com> Date: Wed, 1 Nov 2023 11:07:10 -0400 Subject: [PATCH 06/11] Update src/macros.jl MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Bogumił Kamiński --- src/macros.jl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/macros.jl b/src/macros.jl index 43d53d5a..871f8407 100644 --- a/src/macros.jl +++ b/src/macros.jl @@ -666,7 +666,7 @@ end """ @with(x, args...) -Deprecated version of `@attach`, see `?@with` for details. +Deprecated version of `@attach`, see `?@attach` for details. """ macro with(d, body) @warn "`@with is deprecated, use `@attach` instead." From aee6b505db7562d9188023aa1f730258a2d75b7c Mon Sep 17 00:00:00 2001 From: Peter Deffebach Date: Wed, 1 Nov 2023 11:33:51 -0400 Subject: [PATCH 07/11] exporting with --- test/deprecated.jl | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/test/deprecated.jl b/test/deprecated.jl index feac7741..5be576fc 100644 --- a/test/deprecated.jl +++ b/test/deprecated.jl @@ -203,11 +203,11 @@ end x = [2, 1, 0] - @test @with(df, :A .+ 1) == df.A .+ 1 - @test @with(df, :A .+ :B) == df.A .+ df.B - @test @with(df, :A .+ x) == df.A .+ x + @test DataFramesMeta.@with(df, :A .+ 1) == df.A .+ 1 + @test DataFramesMeta.@with(df, :A .+ :B) == df.A .+ df.B + @test DataFramesMeta.@with(df, :A .+ x) == df.A .+ x - x = @with df begin + x = DataFramesMeta.@with df begin res = 0.0 for i in 1:length(:A) res += :A[i] * :B[i] @@ -215,20 +215,20 @@ end res end idx = :A - @test @with(df, $idx .+ :B) == df.A .+ df.B + @test DataFramesMeta.@with(df, $idx .+ :B) == df.A .+ df.B idx2 = :B - @test @with(df, $idx .+ $idx2) == df.A .+ df.B - @test @with(df, $:A .+ $"B") == df.A .+ df.B + @test DataFramesMeta.@with(df, $idx .+ $idx2) == df.A .+ df.B + @test DataFramesMeta.@with(df, $:A .+ $"B") == df.A .+ df.B - @test_throws ArgumentError @with(df, :A + $2) + @test_throws ArgumentError DataFramesMeta.@with(df, :A + $2) @test x == sum(df.A .* df.B) - @test @with(df, df[:A .> 1, ^([:B, :A])]) == df[df.A .> 1, [:B, :A]] - @test @with(df, DataFrame(a = :A * 2, b = :A .+ :B)) == DataFrame(a = df.A * 2, b = df.A .+ df.B) + @test DataFramesMeta.@with(df, df[:A .> 1, ^([:B, :A])]) == df[df.A .> 1, [:B, :A]] + @test DataFramesMeta.@with(df, DataFrame(a = :A * 2, b = :A .+ :B)) == DataFrame(a = df.A * 2, b = df.A .+ df.B) - @test @with(df, :A) === df.A - @test @with(df, $:A) === df.A - @test @with(df, $"A") === df.A + @test DataFramesMeta.@with(df, :A) === df.A + @test DataFramesMeta.@with(df, $:A) === df.A + @test DataFramesMeta.@with(df, $"A") === df.A end From 875517c87e1feae29a111571454beb66ab27009a Mon Sep 17 00:00:00 2001 From: Peter Deffebach Date: Tue, 7 Nov 2023 11:58:10 -0500 Subject: [PATCH 08/11] update doctests --- src/macros.jl | 53 +++++++++++++++++++++++++++++---------------------- 1 file changed, 30 insertions(+), 23 deletions(-) diff --git a/src/macros.jl b/src/macros.jl index 871f8407..13fa9b2b 100644 --- a/src/macros.jl +++ b/src/macros.jl @@ -899,6 +899,9 @@ julia> @subset df begin :y .== 3 end 0×2 DataFrame + Row │ x y + │ Int64 Int64 +─────┴────────────── julia> df = DataFrame(n = 1:20, x = [3, 3, 3, 3, 1, 1, 1, 2, 1, 1, 2, 1, 1, 2, 2, 2, 3, 1, 1, 2]); @@ -1011,12 +1014,12 @@ julia> @rsubset df :A > 3 julia> @rsubset df :A > 3 || :B == "pear" 3×2 DataFrame - Row │ A B - │ Int64 String - ─────┼─────────────── - 1 │ 2 pear - 2 │ 4 orange - 3 │ 5 pear + Row │ A B + │ Int64 String +─────┼─────────────── + 1 │ 2 pear + 2 │ 4 orange + 3 │ 5 pear ``` """ macro rsubset(x, args...) @@ -1167,6 +1170,9 @@ julia> @subset! copy(df) begin :y .== 3 end 0×2 DataFrame + Row │ x y + │ Int64 Int64 +─────┴────────────── julia> df = DataFrame(n = 1:20, x = [3, 3, 3, 3, 1, 1, 1, 2, 1, 1, 2, 1, 1, 2, 2, 2, 3, 1, 1, 2]); @@ -1369,15 +1375,15 @@ julia> @orderby d begin Row │ x n c │ Int64 Int64 String ─────┼────────────────────── - 1 │ 1 5 e - 2 │ 1 6 f - 3 │ 1 7 g - 4 │ 1 9 i - 5 │ 1 10 j - 6 │ 2 4 d - 7 │ 2 8 h - 8 │ 3 3 c - 9 │ 3 2 b + 1 │ 1 5 d + 2 │ 1 6 g + 3 │ 1 7 f + 4 │ 1 9 j + 5 │ 1 10 h + 6 │ 2 4 e + 7 │ 2 8 i + 8 │ 3 3 b + 9 │ 3 2 c 10 │ 3 1 a julia> @orderby d @byrow :x^2 @@ -1434,6 +1440,7 @@ julia> df = DataFrame(x = [8,8,-8,7,7,-7], y = [-1, 1, -2, 2, -3, 3]) 6 │ -7 3 julia> @rorderby df abs(:x) (:x * :y^3) +6×2 DataFrame Row │ x y │ Int64 Int64 ─────┼────────────── @@ -2528,19 +2535,19 @@ julia> df = DataFrame(x = 1:10, y = 10:-1:1); julia> @distinct(df, :x .+ :y) 1×2 DataFrame - Row │ x y - │ Int64 Int64 -─────┼─────────────── - 1 │ 1 10 + Row │ x y + │ Int64 Int64 +─────┼────────────── + 1 │ 1 10 julia> @distinct df begin :x .+ :y end 1×2 DataFrame - Row │ x y - │ Int64 Int64 -─────┼─────────────── - 1 │ 1 10 + Row │ x y + │ Int64 Int64 +─────┼────────────── + 1 │ 1 10 ``` """ macro distinct(d, args...) From cd227acffd0c6d6131102091e73287848b751f84 Mon Sep 17 00:00:00 2001 From: Peter Deffebach Date: Tue, 7 Nov 2023 12:03:01 -0500 Subject: [PATCH 09/11] fix doctes failures --- src/macros.jl | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/src/macros.jl b/src/macros.jl index 13fa9b2b..85d3865e 100644 --- a/src/macros.jl +++ b/src/macros.jl @@ -1391,16 +1391,16 @@ julia> @orderby d @byrow :x^2 Row │ x n c │ Int64 Int64 String ─────┼────────────────────── - 1 │ 1 5 e - 2 │ 1 6 f - 3 │ 1 7 g - 4 │ 1 9 i - 5 │ 1 10 j - 6 │ 2 4 d - 7 │ 2 8 h + 1 │ 1 5 d + 2 │ 1 6 g + 3 │ 1 7 f + 4 │ 1 9 j + 5 │ 1 10 h + 6 │ 2 4 e + 7 │ 2 8 i 8 │ 3 1 a - 9 │ 3 2 b - 10 │ 3 3 c + 9 │ 3 2 c + 10 │ 3 3 b ``` """ macro orderby(d, args...) From 49c79706be2fb9f3d63ebc7a2be63769a085333f Mon Sep 17 00:00:00 2001 From: Peter Deffebach Date: Thu, 14 Dec 2023 12:53:05 -0500 Subject: [PATCH 10/11] version bump --- Project.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Project.toml b/Project.toml index f5b09a67..7be81dec 100644 --- a/Project.toml +++ b/Project.toml @@ -1,6 +1,6 @@ name = "DataFramesMeta" uuid = "1313f7d8-7da2-5740-9ea0-a2ca25f37964" -version = "0.14.0" +version = "0.14.2" [deps] Chain = "8be319e6-bccf-4806-a6f7-6fae938471bc" From 750e881c38dac7791e383555c708f5f44aa4e955 Mon Sep 17 00:00:00 2001 From: Peter Deffebach Date: Thu, 14 Dec 2023 13:40:55 -0500 Subject: [PATCH 11/11] add news entry --- NEWS.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/NEWS.md b/NEWS.md index b41627f1..2af1fff3 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,3 +1,6 @@ +# DataFramesMeta v0.14.2 Release notes +* `@with` is deprecated and functionality is moved to the new `@attach` macro. ([#367](https://github.com/JuliaData/DataFramesMeta.jl/pull/367)). Julia version 1.11 introduces a macro named `@with` which provides a syntax for dynamic scoping. To avoid conflicts with `@with` when Julia version 1.11 is released, we rename `@with` to `@attach` and begin the depreaction process. + # DataFramesMeta v0.14.1 Release notes * Fixed a bug where `ByRow` was not properly identified if DataFramesMeta.jl was `import`ed ([#366](https://github.com/JuliaData/DataFramesMeta.jl/pull/366))