implement dot_product, cosine_similarity/matrix for tensor ops; add shape mismatch e…

[Force1ess]

feat: implement vector operations and error handling

Resolves #136

• Add dot_product function for tensor multiplication
• Implement cosine_similarity and cosine_distance metrics
• Add ShapeMismatch error type for tensor dimension validation
• Include comprehensive tests with edge cases for all operations

This supersedes previous work in #138 and #157

[Force1ess]

Hi @edgarriba ,
I've implemented the optimized cosine_similarity function as described in the blog post and successfully reproduced comparable performance results. The benchmarks confirm significant speed improvements on my Mac M1 Pro, as shown in the attached screenshots.
One question: Since this optimized implementation doesn't use the dot_product function, should we remove that function from the codebase to reduce redundancy?

[edgarriba]

Great ! So what do you think about providing a low level kernel function like: fn cosine_similariy_kernel_float<T>(a: &[T], b: &[T]) -> T. This will make easy to have a very clean low level api to expose to raw python types (numpy comes with an overhead to pass data from/to) and avoid for now to expose the whole Tensor API. We can do this in a future PR

[Force1ess]

Sounds great to me!

[Force1ess]

By the way, should I put it in the 'tensor-ops' crate too?

[edgarriba]

yep, create a new mod kernels and place it there. This is my suggestion

fn dot_product1_float_kernel<T>(a: &[T], b: &[T]) -> T
where
    T: Zero + Copy + Clone + std::ops::Add<Output = T> + std::ops::Mul<Output = T>,
{
    let mut result = T::zero();
    for (a_val, b_val) in a.iter().zip(b.iter()) {
        result = result + *a_val * *b_val;
    }
    result
}

or

fn product1_float_kernel<T>(a: &[T], b: &[T]) -> T
where
    T: Zero + Copy + Clone + std::ops::Add<Output = T> + std::ops::Mul<Output = T>,
{
    let result = a
        .iter()
        .zip(b.iter())
        .fold(T::zero(), |acc, (a_val, b_val)| acc + *a_val * *b_val);
    result
}

probably the second should leverage better rust internals to vectorise when it's possible

[edgarriba]

Hi @edgarriba , I've implemented the optimized cosine_similarity function as described in the blog post and successfully reproduced comparable performance results. The benchmarks confirm significant speed improvements on my Mac M1 Pro, as shown in the attached screenshots. One question: Since this optimized implementation doesn't use the dot_product function, should we remove that function from the codebase to reduce redundancy?

Results are amazing ! I’m pretty we can even optimise a bit more later. I would like to keep the two functions, specially if we can make them very performant as they are very useful for ml in general

[Force1ess]

Hi @edgarriba , I've implemented the optimized cosine_similarity function as described in the blog post and successfully reproduced comparable performance results. The benchmarks confirm significant speed improvements on my Mac M1 Pro, as shown in the attached screenshots. One question: Since this optimized implementation doesn't use the dot_product function, should we remove that function from the codebase to reduce redundancy?

Results are amazing ! I’m pretty we can even optimise a bit more later. I would like to keep the two functions, specially if we can make them very performant as they are very useful for ml in general

Hi, I've finished the improvements we mentioned before.
Here's the newest benchmark result:

[Force1ess]

By the way, I am pretty interested in optimizing these functions under your guides.
If you have any ideas, please feel free to let me know.

[edgarriba]

By the way, I am pretty interested in optimizing these functions under your guides. If you have any ideas, please feel free to let me know.

lot's of things to do. Curating all this low level kernels needed for ml could be an interesting direction and then efficiently exposing to python

[edgarriba]

yep, create a new mod kernels and place it there. This is my suggestion

fn dot_product1_float_kernel<T>(a: &[T], b: &[T]) -> T
where
    T: Zero + Copy + Clone + std::ops::Add<Output = T> + std::ops::Mul<Output = T>,
{
    let mut result = T::zero();
    for (a_val, b_val) in a.iter().zip(b.iter()) {
        result = result + *a_val * *b_val;
    }
    result
}

or

fn product1_float_kernel<T>(a: &[T], b: &[T]) -> T
where
    T: Zero + Copy + Clone + std::ops::Add<Output = T> + std::ops::Mul<Output = T>,
{
    let result = a
        .iter()
        .zip(b.iter())
        .fold(T::zero(), |acc, (a_val, b_val)| acc + *a_val * *b_val);
    result
}

probably the second should leverage better rust internals to vectorise when it's possible

[edgarriba]

same kernel strategy here

fn cosine_similarity_kernel<T>(a: &[T], b: &[T]) -> T
where
    T: num_traits::Float,
{
    let (dot_product, magnitude_a, magnitude_b) = a.iter().zip(b.iter()).fold(
        (T::zero(), T::zero(), T::zero()),
        |(dot_product, magnitude_a, magnitude_b), (a_val, b_val)| {
            let a = *a_val;
            let b = *b_val;
            (
                dot_product + a * b,
                magnitude_a + a * a,
                magnitude_b + b * b,
            )
        },
    );

    if magnitude_a == T::zero() || magnitude_b == T::zero() {
        return T::zero();
    }

    let denominator = magnitude_a.sqrt() * magnitude_b.sqrt();
    dot_product / denominator
}

[Force1ess]

Hi @edgarriba ,
I've relocated the dependency as per your guides.
Is there anything else we should address in this PR?

[Force1ess]

By the way, I am pretty interested in optimizing these functions under your guides. If you have any ideas, please feel free to let me know.

lot's of things to do. Curating all this low level kernels needed for ml could be an interesting direction and then efficiently exposing to python

While I plan to work on implementing low-level kernels for machine learning.
Maybe I should focus on resolving GSoC-related issues first.

[edgarriba]

By the way, I am pretty interested in optimizing these functions under your guides. If you have any ideas, please feel free to let me know.

lot's of things to do. Curating all this low level kernels needed for ml could be an interesting direction and then efficiently exposing to python

While I plan to work on implementing low-level kernels for machine learning. Maybe I should focus on resolving GSoC-related issues first.

definitely ! but feel free to open any ticket if you think of some other interesting things to be done later. Maybe other can help too. BTW, are you in the Discord?

[Force1ess]

By the way, I am pretty interested in optimizing these functions under your guides. If you have any ideas, please feel free to let me know.

lot's of things to do. Curating all this low level kernels needed for ml could be an interesting direction and then efficiently exposing to python

While I plan to work on implementing low-level kernels for machine learning. Maybe I should focus on resolving GSoC-related issues first.

definitely ! but feel free to open any ticket if you think of some other interesting things to be done later. Maybe other can help too. BTW, are you in the Discord?

Of course, my nickname is forceless!
Moreover, please check if there's anything else we should complete in this PR.

[edgarriba]

let just move the internal functions in a kernel.rs and good to go

[Force1ess]

let just move the internal functions in a kernel.rs and good to go

Hi, I've implemented and placed them in the kernel crate.

[edgarriba]

ok, update then the tensor-ops to use the new kernels functions

[edgarriba]

check that the test pass in you local machine

[Force1ess]

@edgarriba, I've used the kernel functions and all tests passed in my machine

check that the test pass in you local machine

[edgarriba]

Suggested change

	#[error("Error with the kernel: {0}")]
	#[transparent]

I recently learnt this trick, to forward directly the error

[edgarriba]

now seeing this test, I think we should limit also to Tensor1 this operator, as here i don't think it makes a lot of sense as it's applying to the whole tensor instead to a specific axis dimension ?