diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index a3bdb5fb8..07b9b3c95 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -25,7 +25,7 @@ jobs: strategy: fail-fast: false matrix: - python-version: [3.8, 3.9, '3.10', 3.11] + python-version: [3.9, '3.10', 3.11, 3.12] os: [ubuntu-latest, macos-latest, windows-latest] platform: [x64] install-type: [pip, ] # conda] @@ -34,13 +34,13 @@ jobs: use-pre: [false] include: - os: macos-latest # ubuntu-latest - python-version: '3.10' + python-version: 3.11 install-type: pip depends: OPTIONAL_DEPS coverage: true use-pre: false - os: ubuntu-latest - python-version: '3.10' + python-version: 3.12 install-type: pip depends: OPTIONAL_DEPS coverage: false diff --git a/.gitignore b/.gitignore index 1f8a10c9c..9c8b8e457 100644 --- a/.gitignore +++ b/.gitignore @@ -60,6 +60,7 @@ docs/source/api docs/source/auto_examples docs/source/auto_tutorials docs/source/reference +docs/source/sg_execution_times.rst docs/examples/*.png docs/examples/*.vtk docs/examples/*.gif diff --git a/docs/source/_static/images/gsoc-robin-3-fury-discord-bot-references-url.jpg b/docs/source/_static/images/gsoc-robin-3-fury-discord-bot-references-url.jpg new file mode 100644 index 000000000..d9bdec1f4 Binary files /dev/null and b/docs/source/_static/images/gsoc-robin-3-fury-discord-bot-references-url.jpg differ diff --git a/docs/source/_static/images/gsoc_llm_robin_week5.jpg b/docs/source/_static/images/gsoc_llm_robin_week5.jpg new file mode 100644 index 000000000..71be43086 Binary files /dev/null and b/docs/source/_static/images/gsoc_llm_robin_week5.jpg differ diff --git a/docs/source/posts/2024/2024-06-06-week-1-robin.rst b/docs/source/posts/2024/2024-06-06-week-1-robin.rst new file mode 100644 index 000000000..c6b5a8495 --- /dev/null +++ b/docs/source/posts/2024/2024-06-06-week-1-robin.rst @@ -0,0 +1,115 @@ +Week 1: It officially begins... +=============================== + +.. post:: June 06 2024 + :author: Robin Roy + :tags: google + :category: gsoc + +Hi, I'm `Robin `_ and this is my blog about week 1. + +My goal for week 1 was to start with `Retrieval-Augmented Generation (RAG) `_, check different databases and host every endpoint. My week1 and week2 are very intertwined because I applied everything I did during week1 on week2. (I'm writing this blog midway through week2) + +why phi-3-mini-4k-instruct? +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Before I detail everything I've done this week, I'll explain why `phi-3 mini 4k `_ was chosen as the LLM, I forgot to mention this in the last blog. Phi-3 is a small 3.8B 4k context model, it means it can work with 4k tokens(similar to words) at a time. Due to its small size, it runs fast both locally and on Huggingface. Performance wise comparatively with other opensource models, it performs decently well. In the `LMSYS LLM leaderboard `_ phi-3 mini 4k comes with an ELO of 1066 (59th position). But it achieves this as a small model. +I also tried Llama3-8B, it performs better than phi-3 mini with an ELO of 1153 and rank 22. But it is considerably slower for inference. Due to this, I chose phi-3 mini for now. + + +Things I did week-1 (and some week2) +------------------------------------ + +1) **Choosing the vector database** + +I decided to choose `Pinecone `_ as the vector DB because it had a very generous free tier. Other options on consideration were `pgvector `_ and `chromadb `_, but they didn't have a free tier. + +2) **PR Submissions and Review** + +I also merged a `PR `_ on FURY which fixes a CI issue. I also spent time doing review of other PRs from my fellow GSoC mates. + +3) **Deciding which embedding model to use** + +A good embedding model is necessary to generate embeddings which we then upsert into the DB. Ollama had embedding model support, but I found the catalogue very small and the models they provided were not powerful enough. Therefore I decided to try using HuggingFace Sentence Transformers. +Sentence Transformers have a very vibrant catalogue of models available of various sizes. I chose `gte-large-en-v1.5 `_ from Alibaba-NLP, an 8k context, 434 million parameter model. It only had a modest memory requirement of 1.62 GB. +Performance wise, it ranks 11th on the `MTEB leaderboard `_. It is a very interesting model due to its size:performance ratio. + +4) **Hosting the embedding model** + +Hosting this sentence-transformer model was confusing. For some reason, the HF spaces were blocking the Python script from writing on ``.cache`` folder. Docker container inside spaces runs with user id 1000 (non-root user), therefore I had to give it permission to download and store files. + +I've hosted 5 gunicorn workers to serve 5 parallel requests at a time. Since the model is small, this is possible. + +5) **Hosting the database endpoint** + +I wrapped the pinecone DB API into an endpoint so it'll be easy to query and receive the results. +It is also configured to accept 5 concurrent requests although I could increase it a lot more. + +I upserted docstrings from ``fury/actor.py`` into the vector DB for testing. So now, whenever you ask a question the model will use some ``actor.py`` function to give you an answer. For now, it could be used like a semantic function search engine. + +I decided to abstract the DB endpoint to reduce the dependency on one provider. We can swap the providers as required and keep all other features running. + +6) **Hosting Discord Bot** + +So with this, all the endpoints are finally online. The bot has some issues, it is going offline midway for some reason. I'll have to see why that happens. + +For some reason, Huggingface spaces decided to not start the bot script. Later a community admin from Huggingface told me to use their official bot implementation as a reference. This is why I had to use threading and gradio to get the bot running (migrating to docker can be done, but this is how they did it and I just took that for now). + +Huggingface spaces need a script to satisfy certain criteria to allow them to run, one of them is a non-blocking I/O on the main loop. So I had to move the discord bot to a new thread. + +Connecting all of them together! +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +So now we have 4 hosted services, all hosted on HuggingFace spaces: + - Discord Bot + - LLM API + - Embeddings API + - Database API + +Now we'll have to connect them all to get an answer to the user query. + +This is the current architecture, there's a lot of room for improvement here. + + + .. raw:: html + + + +The Language Model takes the context and the user query, combines them to form an answer and returns to the user through discord (for now). Maybe moving the core logic from discord bot to a separate node might be good, and connect discord/github/X to that node. +The database takes embeddings and do an Approximate Nearest Neighbor search (a variant of KNN) and returns top-k results (k=3 for now). + + .. raw:: html + + + +What is coming up next week? +---------------------------- + +Answer quality improvements. Also, the discord bot dies randomly, have to fix that also. + +Did you get stuck anywhere? +--------------------------- + +Was stuck in hosting models on Huggingface spaces, fixed it later. + +LINKS: + +- `Discord Bot `_ + +- `Database Repo `_ + +- `Embedding Repo `_ + +- `LLM Repo `_ + +- `Retrieval-Augmented Generation (RAG) `_ +- `phi-3 mini 4k `_ +- `LMSYS LLM leaderboard `_ +- `Pinecone `_ +- `pgvector `_ +- `chromadb `_ +- `PR `_ +- `gte-large-en-v1.5 `_ +- `MTEB leaderboard `_ + +Thank you for reading! diff --git a/docs/source/posts/2024/2024-06-06-week1-wachiou-bouraima.rst b/docs/source/posts/2024/2024-06-06-week1-wachiou-bouraima.rst new file mode 100644 index 000000000..b16a3f960 --- /dev/null +++ b/docs/source/posts/2024/2024-06-06-week1-wachiou-bouraima.rst @@ -0,0 +1,61 @@ +WEEK 1: Progress and challenges at Google Summer of Code (GSoC) 2024 +==================================================================== + +.. post:: June 06, 2024 + :author: Wachiou BOURAIMA + :tags: google + :category: gsoc + +Hello👋🏾, + +Welcome back to my Google Summer of Code (GSoC) 2024 journey! +This week has been filled with progress and challenges as I continue to work on modernizing the FURY code base. + + +Applying the keyword_only decorator +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +My main task this week was to apply the keyword_only decorator to several functions. +The decorator ensures that all arguments except the first are keyword-only, +which helps to make the code clearer and parameter passing more explicit. +Some warnings appeared after applying this decorator, and to resolve them, +I updated all the code where these functions were called with the necessary format. This was a very important step in maintaining the integrity and functionality of the code base. + + +Managing the challenges of Git rebasing +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Rebasing the branch I was working on was the other major activity of my week. +It was a real challenge because of the conflicts that arose and had to be resolved. +It involved a lot of research and problem-solving on how to resolve these conflicts, +which greatly enhanced my understanding of Git. It was a challenging but satisfying experience of version control management and complex mergers. + + +Peer code review +~~~~~~~~~~~~~~~~ + +In addition to my duties, I was also tasked with reviewing the code of my peers. +This exercise was very rewarding, as it enabled me to understand different coding styles and approaches. +The constructive comments and suggestions were beneficial not only for teammates, +but also for improving my own coding and reviewing skills. + + +Acknowledgements +~~~~~~~~~~~~~~~~~ + +I would like to thank all my classmates: `Iñigo Tellaetxe Elorriaga `_, `Robin Roy `_, `Kaustav Deka `_ and my guide: `Serge Koudoro `_ for their constructive suggestions on my work. +Their ideas and suggestions were of great help to me and I am grateful for their support and advice. + + +What happens next? +~~~~~~~~~~~~~~~~~~ + +Here's a summary of what I plan to do in week two: + +- Apply the keyword_only decorator to all other necessary functions. +- Update the calling of these functions in the code to ensure consistency and avoid raising warnings. +- Rename the decorator with a more descriptive name. +- Add two parameters to the decorator, specifying from which version of FURY it will work. + + +🥰Thanks for reading! Your comments are most welcome, and I look forward to giving you a sneak preview of my work next week. diff --git a/docs/source/posts/2024/2024-06-15-week2-wachiou-bouraima.rst b/docs/source/posts/2024/2024-06-15-week2-wachiou-bouraima.rst new file mode 100644 index 000000000..fd87a11c2 --- /dev/null +++ b/docs/source/posts/2024/2024-06-15-week2-wachiou-bouraima.rst @@ -0,0 +1,52 @@ +WEEK 2: Refinements and Further Enhancements +============================================ + +.. post:: June 15, 2024 + :author: Wachiou BOURAIMA + :tags: google + :category: gsoc + +Hello again, +~~~~~~~~~~~~~ + +Welcome back to my Google Summer of Code (GSoC) 2024 journey! This week has been dedicated to refining and improving the work done so far, with a particular focus on the keyword_only decorator. + + +Renaming and Updating the Decorator +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This week, I've updated `this Pull Request `_ by renaming the ``keyword_only`` decorator to ``warn_on_args_to_kwargs`` for greater clarity. The updated decorator now includes version parameters from_version and until_version. This enhancement ensures that the decorator will raise a RuntimeError if the current version of FURY is greater than until_version. + + +Peer Code Review +~~~~~~~~~~~~~~~~~ + +I also spent time reviewing `Kaustav Deka's `_ code. This exercise remains rewarding, as it helps me understand different coding styles and approaches. Constructive feedback and suggestions from my classmates were invaluable, not only in helping my teammates but also in improving my own coding and reviewing skills. + + +Research into lazy loading +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In parallel, I started researching the lazy loading feature and thinking about how to implement it. This feature will optimize performance by loading resources only when they're needed, which is crucial to improving the efficiency of FURY's code base. + + +Acknowledgements +~~~~~~~~~~~~~~~~ + +I am deeply grateful to my classmates `Iñigo Tellaetxe Elorriaga `_, `Robin Roy `_, and `Kaustav Deka `_ for their insightful suggestions and comments on my work. +Special thanks to my mentor, `Serge Koudoro `_, whose guidance and support enabled me to meet the challenges of this project. +Their combined efforts have greatly contributed to my progress, and I appreciate their continued help. + + +What happens next? +~~~~~~~~~~~~~~~~~~ + +For week 3, I plan to : + +- Ensure that the ``warn_on_args_to_kwargs`` decorator is applied consistently in all necessary functions. +- Continue to update the calling of these functions in the code to maintain consistency and avoid warnings. +- Refine decorator as necessary based on feedback and testing. +- Start implementing lazy loading functionality based on my research to optimize performance. + + +🥰 Thank you for taking the time to follow my progress. Your feedback is always welcome and I look forward to sharing more updates with you next week. diff --git a/docs/source/posts/2024/2024-06-16-week2-robin.rst b/docs/source/posts/2024/2024-06-16-week2-robin.rst new file mode 100644 index 000000000..cfcb5235b --- /dev/null +++ b/docs/source/posts/2024/2024-06-16-week2-robin.rst @@ -0,0 +1,79 @@ +Week 2: The first iteration! +============================ + +.. post:: June 16 2024 + :author: Robin Roy + :tags: google + :category: gsoc + +Hi, I'm `Robin `_ and this is my blog about week 2. + +My goal for week 2 was to connect everything and make a prototype. So now we have a bot working 24x7 to answer all your doubts :) + +Apart from the things mentioned in my `week 1 blog `_, the things I did in week 2 are basically: + - Chunking the files for embedding. + - Upserting the chunks into the database. + - Connecting everything together. + - Making the discord bot async. + - Merging a PR. + +1) **Chunking the files for embedding** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the context of building LLM-related applications, chunking is the process of breaking down large pieces of text into smaller segments. It's an essential technique that helps optimize the relevance of the content we get back from a vector database once we use an embedding model to embed content. For our case with FURY, our data is entirely code. So one approach I tried was to take docstrings and the function/class signature. + +I used a naive parser during week 2, which used a combination of regex and common pattern matching to do this splitting. Later my mentors `Mohamed `_ and `Serge `_ told me to use a better approach, using the python ``inspect`` module. + +Another thing to consider was the chunk size. It is shown that smaller chunks outperform larger chunks. This can be intuitively thought of like this: An embedding model can compress a smaller text to 1024 vectors without much data loss compared to compressing a larger text to 1024 vectors. + +This also introduces another important issue, we need a way to test it based on our model. So we need benchmarking. + + +2) **Upserting chunks into the database** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +I upserted all the chunks into the database, along with the vectors I gave metadata which was the function signature and docstrings. Later in week 3, we'll modify this to show links instead of the big wall of text. + + +3) **Connecting everything together** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +I took the 4 key parts - Discord Bot, LLM API, Embeddings API and the Database API and connected them together. This was explained on the `week 1 blog `_ itself. + + +4) **Making the Discord Bot async** +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +One of the biggest challenges I faced this week was to get everything running properly. LLM output takes a lot of time to generate (we'll fix this amazingly well in week 3 BTW). +I made a big mistake, I used ``requests`` library to do the REST API calls. It occurred to me later that it is synchronous and does blocking calls. This was the reason my Discord bot was dying randomly. I fixed it by migrating to ``aiohttp``. + +This also made me realize I can use async in a lot of other places. A lot of these tasks are I/O bound. If I make them async we might be able to take many more concurrent requests. + +5) **Merging a PR** +~~~~~~~~~~~~~~~~~~~ + +I merged a `PR `_ which modifies `.gitignore`. I found this while generating the Sphinx docs. + + +What is coming up next week? +---------------------------- + +- A faster LLM inference. +- Better pipeline for data collection. +- Links for citation. + +Did you get stuck anywhere? +--------------------------- + +Took me some time to realize I was using synchronous code inside async. Fixed it later. + + +LINKS: + +- `Week 1 Blog `_ +- `PR `_ +- `Serge Koudoro `_ +- `Mohamed Abouagour `_ +- `Robin :) `_ + +Thank you for reading! diff --git a/docs/source/posts/2024/2024-06-16-week3-robin.rst b/docs/source/posts/2024/2024-06-16-week3-robin.rst new file mode 100644 index 000000000..ba6c2e2e9 --- /dev/null +++ b/docs/source/posts/2024/2024-06-16-week3-robin.rst @@ -0,0 +1,82 @@ +Week 3: Data Data Data! +======================= + +.. post:: June 16 2024 + :author: Robin Roy + :tags: google + :category: gsoc + +Hi, I'm `Robin `_ and this is my blog about week 3. + +My goal for week 3 was to collect data more efficiently and improve the citations. I also had my mid-terms during this week so I had to get things done fast. + +Things I did in week 3 +---------------------- + +1) **A better data parsing technique** + +My initial approach was naive, it was just regex and some common filtrations. Later, my mentors told me to use the `inspect` module. I studied that module and realized that I needed to parse data using an AST. I didn't use the `inspect` module to do the parsing, since I only had to get the function/class signature and docstrings. So instead I used the ``ast`` module from python stdlib. My mentors gave me the general direction to go through - which was using ASTs to parse data effectively. + +So now we have a script which you run like `python extractor.py fury` and it'll generate the appropriate JSON files. + +`{"path": "../..", "function/class name": "name", "docstring": "..", "class_methods": ["method1", "..."]}` + +I also changed the upserting chunk format. Earlier it was just strings, now it is JSON (same thing above). I do not have a scientific reason for this, but empirically it looks like it helped. Benchmarking is something I'm planning to do next week. + +Metadata format: + +`metadata: {"path": "../..", "function/class name": "name", "docstring": "..", "methods": [(method1, docstring), (method2, docstring), ...]}` + +2) **Links for citation** + +Now the bot shows links for citations. Because of the new parsing, I was able to do that pretty efficiently. + +.. image:: /_static/images/gsoc-robin-3-fury-discord-bot-references-url.jpg + :alt: Link based references for the LLM output. + + +3) **Faster Inference** + +So this is something about the Generative AI field. There are too many things happening you might miss some stuff. `Groq` is a company providing free APIs for the llama and other opensource models (free for now, at least). Its inference speed is also super high. So I decided to integrate that also into our infrastructure. +Since everything is a microservice in our architecture, it is easy to add new things. + +Our architecture: + .. raw:: html + + + +So now, along with Ollama, we have Groq inference also. I aim to make a `router` so that we can swap different providers as required. I'm also very interested in integrating Google Gemini 1.5 Flash and other models. Groq does not support fine-tuning, but Flash supports it and is `free of cost `_ (for now). Our architecture is platform agnostic, so we can try out different things without being locked into any of them. We will also fine-tune our phi3 model since we have the data with us. + + .. raw:: html + + + +4) **Dockerizing Discord Bot** + +I earlier used the huggingface implementation (copied their implementation demo). It was bad. My mentors suggested to dockerize the bot so I did that. + + +What is coming up next week? +---------------------------- + +- Benchmarking. Now we have the data, but we need to properly benchmark to see whether the modifications I make every day are making the bot dumber or smarter. +- Study different techniques to improve model answer accuracy such as `HyDE `_. +- Study how to go forward with fine-tuning. +- Improved references. +- Collect more data. + + +Did you get stuck anywhere? +--------------------------- + +No, everything went well this week. Exam preparation was a pain though😢. + +LINKS: + +- `Gemini Blog `_ + +- `HyDE `_ + +- `Robin :) `_ + +Thank you for reading! diff --git a/docs/source/posts/2024/2024-06-26-week3-wachiou-bouraima.rst b/docs/source/posts/2024/2024-06-26-week3-wachiou-bouraima.rst new file mode 100644 index 000000000..cca107b68 --- /dev/null +++ b/docs/source/posts/2024/2024-06-26-week3-wachiou-bouraima.rst @@ -0,0 +1,74 @@ +WEEK 3: Refinements and Further Enhancements +============================================ + +.. post:: June 26, 2024 + :author: Wachiou BOURAIMA + :tags: google + :category: gsoc + +Hello everyone, +--------------- + +Welcome to the fourth week of my Google Summer of Code (GSoC) 2024 journey! +This week I've been delving into the technical aspects of my project, +focusing on the consistent application of the ``warn_on_args_to_kwargs`` decorator and the initial implementation of lazy loading. + + +Consistent application of ``warn_on_args_to_kwargs`` +---------------------------------------------------- + +This week I continued to apply the decorator to functions. +To ensure consistency across the code base, I audited all functions that could benefit from the ``warn_on_args_to_kwargs`` decorator. +To do this, I had to: + +1. Identify target functions: + + * Identify functions that could benefit from the decorator. + * continue reviewing the code base to identify functions that accept both positional and keyword arguments. + +2. Applying the Decorator: + + * For each identified function, I added the ``warn_on_args_to_kwargs`` decorator. + * Example: + +.. code-block:: python + + @warn_on_args_to_kwargs() + def get_actor_from_primitive( + vertices, + triangles, + *, + colors=None, + normals=None, + backface_culling=True, + prim_count=1, + ): + +3. Updating Unit Tests: + +* updated all the unit tests for the functions where the ``warn_on_args_to_kwargs`` decorator is applied to ensure they respect the new format. +* Example: + +.. code-block:: python + + actr = get_actor_from_primitive(big_verts, big_faces, colors=big_colors) + +- You can find more details and the implementation in my pull request: `https://github.com/fury-gl/fury/pull/888 `_. + + +What Happens Next? +------------------ + +For week 4, I plan to: + +* Continue refining the ``warn_on_args_to_kwargs`` decorator based on feedback from my Peers `Iñigo Tellaetxe Elorriaga `_, `Robin Roy `_, `Kaustav Deka `_, my guide: `Serge Koudoro `_ and the other community members. +* Apply the ``warn_on_args_to_kwargs`` decorator to all the remaining modules and update all the unit tests of these modules too, to respect the desired format. +* Dive deep into the lazy loading functionality based on my research to optimize performance. +* Further engage in code reviews to support my peers and improve our project. + +Did I get stuck? +---------------- + +I didn't get stuck. + +Thank you for following my progress. Your feedback is always welcome. diff --git a/docs/source/posts/2024/2024-06-26-week4-wachiou-bouraima.rst b/docs/source/posts/2024/2024-06-26-week4-wachiou-bouraima.rst new file mode 100644 index 000000000..2bb6def37 --- /dev/null +++ b/docs/source/posts/2024/2024-06-26-week4-wachiou-bouraima.rst @@ -0,0 +1,112 @@ +WEEK 4: Updating Decorator, Exploring Lazy Loading, and Code Reviews +==================================================================== + +.. post:: June 26, 2024 + :author: Wachiou BOURAIMA + :tags: google + :category: gsoc + +Hello everyone, +--------------- + +Welcome again to my Google summer of code 2024 (GSoC' 2024) journey 2024!. +This week, I focused on updating the ``warn_on_args_to_kwargs`` decorator, applying it across multiple modules, exploring lazy loading, and continuing with code reviews. + + +Updating the ``warn_on_args_to_kwargs`` decorator +------------------------------------------------- + +Based on feedback from my mentor `Serge Koudoro `_ and peers `Iñigo Tellaetxe Elorriaga `_, `Robin Roy `_, `Kaustav Deka `_, I refined the ``warn_on_args_to_kwargs`` decorator and its associated unit tests: + +1. Improvements: + + - Added conditions to verify if the values of ``from_version``, ``until_version``, and the current version of FURY are respected. This includes handling cases where ``from_version`` is greater than the current version of FURY, ``until_version`` is less than the current version of FURY, and ``until_version`` is greater than or equal to the current version of FURY. + - Ensured the decorator and tests cover a broader range of edge cases. + - Enhanced the warning messages for better clarity and guidance. + +2. Doctest Updates: + + - Updated the doctest considering the values of `from_version` and `until_version`. + - Moved the doctest from the `def decorator()` function to the root function. + +3. Unit Tests: + +.. code-block:: python + + def test_warn_on_args_to_kwargs(): + @warn_on_args_to_kwargs() + def func(a, b, *, c, d=4, e=5): + return a + b + c + d + e + + # if FURY_CURRENT_VERSION is less than from_version + fury.__version__ = "0.0.0" + npt.assert_equal(func(1, 2, 3, 4, 5), 15) + npt.assert_equal(func(1, 2, c=3, d=4, e=5), 15) + npt.assert_raises(TypeError, func, 1, 3) + +- This ensures robust validation and helps catch potential issues early. + + +Applying the ``warn_on_args_to_kwargs`` Decorator +----------------------------------------------- + +This week, I applied the ``warn_on_args_to_kwargs`` decorator to several modules, ensuring consistent usage and improved code quality. The modules updated include: + +- `actors` +- `ui` +- `animation` +- `shares` +- `data` + +For each module, I opened a pull request to track the changes and facilitate reviews: + +- `actors`: https://github.com/fury-gl/fury/pull/898 +- `animation`: https://github.com/fury-gl/fury/pull/899 +- `data`: https://github.com/fury-gl/fury/pull/900 +- `shares`: https://github.com/fury-gl/fury/pull/901 +- `ui`: https://github.com/fury-gl/fury/pull/902 + + +Exploring lazy loading +---------------------- + +In order to optimize performance, I've started exploring and implementing lazy loading. This week, the focus was on the following points: + +- Getting to grips with how the lazy loader works +- Implementing some small script to understand how the lazy loader works +- I also read the SPEC1 document available at `SPEC1 `_ +- Understanding the benefits of lazy loading and how it can be applied to the FURY code base +- Planning the integration of lazy loading into the FURY code base + +Code sample: ``_ + + +Peer Code Review +---------------- + +This week, I continued to dedicate time to reviewing the code of my peers. Specifically, I reviewed Kaustav Deka’s work, providing constructive feedback and suggestions for improvement. You can view the pull request here: `https://github.com/dipy/dipy/pull/3239 `_. + + +Acknowledgements +---------------- + +I am deeply grateful to my classmates `Iñigo Tellaetxe Elorriaga `_, `Robin Roy `_, `Kaustav Deka `_ for their continuous support and insightful suggestions. Special thanks to my mentor, `Serge Koudoro `_ , whose expertise and guidance have been invaluable in navigating these technical challenges. + + +Did I get stuck? +----------------- + +Yes, I was a bit confused about understanding lazy loader, but thanks to the help of my mentor `Serge Koudoro `_ , I was able to understand it better. + + +What's next? +------------ + +For the upcoming week, I plan to: + +- Implement lazy loading in the FURY code base +- Continue refining the ``warn_on_args_to_kwargs`` decorator based on feedback +- Engage in more code reviews to support my peers +- Prepare to working on the FURY website to improve the documentation and user experience + +Thank you for following my progress. Your feedback is always welcome. diff --git a/docs/source/posts/2024/2024-07-01-week-4-robin.rst b/docs/source/posts/2024/2024-07-01-week-4-robin.rst new file mode 100644 index 000000000..0172bff7b --- /dev/null +++ b/docs/source/posts/2024/2024-07-01-week-4-robin.rst @@ -0,0 +1,74 @@ +Week 4: Pipeline Improvements and Taking The Bot Public! +======================================================== + +.. post:: July 1 2024 + :author: Robin Roy + :tags: google + :category: gsoc + +Hi, I'm `Robin `_ and this is my blog about week 4. + +My goals for week 4 were to move my Google colab notes to a proper Python script, improve the existing code, and make a working pipeline to upsert data easily. Also, the bot is public now :) Anyone reading this blog could join this `Discord Server `_ and ask questions right away! + +Things I did in Week 4 +---------------------- + +1) **Chunking tutorials and documentation** + +Earlier, only files fitting the context window of the embedding model were upserted. This was because otherwise, we'd have to split the file in half and lose the overall context. This will lead to information loss and retrieval will be messy. Now, I decided I'd upsert everything by splitting information properly. By "properly", what I mean is it won't be a random split, and there'll be logical reasoning behind every chunk. + +This area is still actively studied, and the whole concept is to find ideal chunks which are self-sufficient and contain the most information. This `notebook `_ details 6 different approaches, I read through them and some of their associated literature and decided we'll use `Recursive Character Text Splitting` and `Document Specific Splitting` for now. There is no major reason for this, I just felt it'll work well for now (a reasoning-backed approach will come in a few weeks). There is a lot of experimentation we could do here, a better chunking will result in better ``references`` generation and so on. + +So this is our current process + - if normal function/class definition: no splitting, chunk as it is. + - if rst files, use the ``rst parser`` and split them with a chunk size of ~8000 tokens (max llama could take). RST files in FURY contain documentation & blog posts. + - if tutorial, try chunking as it is, if not possible split at 8000 tokens. + +Function/class definitions are generally under 8000 so I've not done explicit checks for now, the model will trim the remaining if longer (I found some long classes later). + +2) **Move colab files to a proper Python script** + +I did all the upsertion and experiments on colab. It is messy and can't be used in production. We need a one-click approach to upsertion. Something like point to `fury` directory and it should do everything. So I took the messy colab code and made a python script from it. + +One of my key goals is to separate core application logic from LLMs/Database providers. We should be able to swap them as needed without much fuss. I'll talk more about this in week 5. + +3) **Taking the bot public!** + +The whole point of making the bot is to help improve the productivity of FURY developers. So I decided to take it public on `this discord server `_. You could use it today! (actually, you could've used it from the 20th of last month, this blog got delayed😢) + +I'll observe what people are asking and then iterate towards making the bot better in that area. I think it'll be better than making the bot good on what I believe is the best. + +4) **Minor bugfixes and stuff** + +Did some minor bug fixes on things like the Discord bot generation cutoff and error handling improvements. It was Discord message limit (<=2000) that caused the generation to cut off, I split the message into parts to fix that. Error handling was improved generally everywhere. I'll need to bring logging later. + + +Minor Sidequest +~~~~~~~~~~~~~~~ + +This is in no way related to FURY, but it was fun so I thought I'd add it here :) + +So after midterms, I decided to go back home, to maximally use my time I searched for things to do and found a local FOSS event (`link `_). It was done by FOSS United Kochi and it's one of the major FOSS events in my state (Kerala, India). Met some Pythonistas! Explained what FURY is to them. I also ended up finding some lore (`link `_) about how GNU/Linux spread in Kerala, India. Also found some old FOSS event pictures (`this `_ one is talking about Python, 2003 World of Python). This was my first FOSS event outside campus so it was fun :) + + +What is coming up next week? +---------------------------- + +- Benchmarking +- Architecture Update + +Did you get stuck anywhere? +--------------------------- + +No, I did not get stuck. This week was more of learning and experimentation so I think it's normal what I encountered. + +LINKS: + +- `Discord Server `_ +- `A Text Splitting Guide `_ +- `GNU Case of Kerala `_ +- `2003 World of Python `_ +- `FOSS United Kochi `_ +- `Robin :) `_ + +Thank you for reading! diff --git a/docs/source/posts/2024/2024-07-01-week-5-robin.rst b/docs/source/posts/2024/2024-07-01-week-5-robin.rst new file mode 100644 index 000000000..b785be5f0 --- /dev/null +++ b/docs/source/posts/2024/2024-07-01-week-5-robin.rst @@ -0,0 +1,129 @@ +Week 5: LLM Benchmarking & Architecture Modifications +===================================================== + +.. post:: July 1 2024 + :author: Robin Roy + :tags: google + :category: gsoc + +Hi, I'm `Robin `_ and this is my blog about week 5. + +This week, we'll take all the things we did in the previous weeks, and quantify them. Benchmarking an LLM is the process of grading the LLM answer. To grade properly, we need good rubrics, so that's what I worked on this week. Also, I made some architectural changes, to make the overall development simple. + +Things I did in Week 5 +---------------------- + +1) **Architectural Update** + +Earlier, this was our architecture: + + + .. raw:: html + + + +This had an obvious issue, all the core logic was inside the Discord Bot. So if I want to say, use the LLM inference for making a GitHub bot, or for benchmarking etc, it wasn't possible. So I decided to cut the LLM logic from Discord Bot and made a new ``LLM Router``. It'll handle all the LLM logic from now on, and we do not directly call any other endpoint other than this one. +It makes life simple, every input going into the endpoint goes like this: + +.. code-block:: json + + { + "query": "Render a cube in fury", + "llm": "llama3-70b-8192", + "knn": "3", + "stream": False + } + +Every response coming out will be like this: + +.. code-block:: json + + { + "response": "Yes, this is how it would be done python import fury....", + "references": "1, 2, 3" + } + +What happens on the inside is completely abstracted away. You just call this and it'll + - call the embedding model + - pass embeddings to the database + - return them to LLM (which you can choose) + - returns LLM answer with references to you + +Currently, we support ``ollama``, ``google`` and ``groq`` providers. That itself is 20+ LLM support, and you could swap between them using ``/api/groq or api/google or /api/ollama ...``. Adding another provider is simply adding another endpoint. + +So if you do + +`curl -X POST https://robinroy03-fury-engine.hf.space/api/groq/generate -H "Content-Type: application/json" -d '{"query": "How do I create a sphere in FURY?", "llm": "llama3-70b-8192", "knn": "3", "stream": false}'` + + +You'll get a response from ``llama3-70b-8192`` using ``groq``. If you do ``https://robinroy03-fury-engine.hf.space/api/google/generate`` you can call any google gemini modes like ``gemini-1.5-pro`` or ``gemini-1.5-flash``. Same for ``ollama``. + +This still could be improved, it does not currently account for vision models. I did not add that because we do not use vision models other than for benchmarking now, and that too is done locally. Benchmarking could also be streamlined, I avoided that because benchmarking is still in development so I'll have to rewrite every day. Presently you can use this core ``router`` for a working LLM generation (you'll get the same thing you'll get from the Discord Bot. So if you have a website, all you have to do is call the API). + +This is our present architecture: + +.. image:: /_static/images/gsoc_llm_robin_week5.jpg + :alt: Present LLM architecture. + +It is the same thing as above, except we have two new components - ``LLM Engine`` and a ``Groq & Gemini`` endpoint. When we'll end up having a conversational model setup (right now, it is one question and one answer), this model will be upgraded to accommodate that. My plan is to extend LLM Engine and add that. Other features such as vision also could be added to this as needed. + +2) **Gemini Models added** + +As mentioned above, I added ``Gemini`` models this week. They have a decent free tier. Also, I'm studying the feasibility of fine-tuning using the ``Gemini`` models. + +3) **LLM Benchmarking** + +LLM Benchmarking is the process of evaluating the LLM output and giving a score. With this, making the model better will be simply a function of increasing the score. This area is still under development and the things I've tried here are the current standard procedures. To understand more about benchmarking, you can read `this `_, `this `_ and `this `_. This `course `_ is also amazing. + +I'll anyways give a TL;DR: +LLM benchmarking is essentially like writing an English Literature exam and getting the grades. Your evaluator may give you a 4 or a 5, and the reasoning can be varied. For the same answer, you may even get very varied results from 2 different evaluators! Two common rubrics they use are ``groundedness (whether the answer follows from the material)`` and ``completion (whether the answer is complete, whether it fully answers the question with respect to the material)``. These are the same rubrics we'll use for LLM evaluation. For code, it's different. The code should compile and do exactly what it should. + +Now FURY Bot does 2 things - writing code & writing answers for common questions (on GitHub issues etc). Presently, I've only collected data for coding questions, as they are much easier to evaluate and give a clear sense of direction (also I found more coding data). + +Evaluating FURY code can be done by: + 1) Running the code. + 2) Checking the output. + +Now we do this using ``pytest`` in the FURY repo for tests. But this approach is tedious, as collecting questions and writing test cases take a lot of time, also the orientation of the 3D objects also matters (an LLM generation is not deterministic). So we are using a vision model ``moondream2`` to check the LLM generated output and verify if it is what we actually wanted. +On a high level, this is what we do (for now): + +- Take a QnA pair from the collected dataset (I've collected ~23 questions). +- Ask the LLM to generate a FURY code for that (using the references). +- Run this generated code. +- Check the output using ``moondream2`` and verify whether it is what we wanted. + +There is also ``fast_eval`` which checks whether the code compiles and skips ``moondream2`` entirely. This is obviously faster and is also decently good (is actually a pretty good heuristic). If it runs, assume it works :) + +This is our current stats: (from now on, we can finally talk using numbers) + +Coding benchmark: +~~~~~~~~~~~~~~~~~ +On ``fast_eval`` we have a success rate of ``47.83%`` for ``groq``. + +On ``normal_eval`` we have a success rate of ``13.04%`` for ``groq``. + +Note that ``moondream2`` also sometimes mistakes the output for something else. It is close to ``~45%`` when I checked manually. For now, I'm only going to focus on ``fast_eval`` as fixing ``moondream2`` is a distraction for the moment. (This actually gets very meta, there are projects where they have benchmarks for the evaluator and so on. `Read this `_.) + + +What is coming up next week? +---------------------------- + +- Better benchmark scores :) +- Line number highlighting @ references. +- Some ``references`` improvements. + +Did you get stuck anywhere? +--------------------------- + +No, I did not get stuck anywhere. + +LINKS: + +- `RAG Evaluation `_ +- `LLM Judge `_ +- `Advanced RAG `_ +- `Advanced Retrieval for AI `_ +- `Moondream2 `_ +- `Finding GPT-4 mistakes with GPT-4 `_ + +Thank you for reading! diff --git a/docs/source/posts/2024/2024-07-06-week5-wachiou-bouraima.rst b/docs/source/posts/2024/2024-07-06-week5-wachiou-bouraima.rst new file mode 100644 index 000000000..01a915125 --- /dev/null +++ b/docs/source/posts/2024/2024-07-06-week5-wachiou-bouraima.rst @@ -0,0 +1,112 @@ +WEEK 5: Implementing Lazy Loading in FURY with ``lazy_loader`` +============================================================== + +.. post:: July 6, 2024 + :author: Wachiou BOURAIMA + :tags: google + :category: gsoc + +Hello everyone, +--------------- + +Welcome back to my Google Summer of Code (GSoC) 2024 journey! This week has been particularly exciting as I introduced a significant performance optimization feature: lazy loading. Here's an overview of my progress and contributions. + + +**Introduction of lazy loading** +-------------------------------- + +This week, I focused on implementing the ``lazy_loader`` feature of `Scientific Python `_ to optimize module loading in FURY. Lazy loading improves performance by deferring the loading of modules until they are actually needed, thus reducing start-up times and memory footprint. + +The implementation involved: + +1. Implementation of Lazy Loading: + + - Application of lazy loading in several FURY modules using the ``lazy_loader`` module to improve performance + +2. Update ``__init__.py`` files: + + - Modified ``__init__.py`` files to support lazy loading where necessary. This ensures that modules are only loaded when they are accessed for the first time + +3. Added Type Stubs (``__init__.pyi``): + + - Adding type stubs (``__init__.pyi``) provides type hints for lazy-loading modules, improving code readability and maintainability + +4. **Improved module organization:** + + - Improved module organization in ``__init__.py`` and ``__init__.pyi`` files, to effectively support lazy loading. + + +**Example Implementation** +--------------------------- + +To give you an idea, here's the actual implementation of how lazy loading was done using the ``lazy_loader`` module in FURY: + +``__init__.py`` File: + +.. code-block:: python + + import lazy_loader as lazy + from fury.pkg_info import __version__, pkg_commit_hash + + __getattr__, __dir__, __all__ = lazy.attach_stub(__name__, __file__) + + _all__ += [ + "__version__", + "disable_warnings", + "enable_warnings", + "get_info", + ] + + # ... (functions) + +``__init__.pyi`` File: + +.. code-block:: python + + # This file is a stub type for the fury package. It provides information about types + # to help type-checking tools like mypy and improve the development experience + # with better autocompletion and documentation in code editors. + + __all__ = [ + "actor", + "actors", + "animation", + "colormap", + # ... (other modules) + , + ] + + from . import ( + actor, + actors, + animation, + colormap, + # ... (other modules) + , + ) + # ... (other functions) + +You can review the implementation in `this pull request `_. + + +Reading ``SPEC1`` +----------------- + +To align myself with best practice, I read the `SPEC1 `_ document available at Scientific Python SPEC1. This document provided valuable hints and guidelines that I took into account when implementing the lazy loading feature. + + +Did I get stuck anywhere? +-------------------------- +No, I didn't encounter any major blockers this week. The implementation of lazy loading went smoothly, and I was able to complete the task. + + +**What's Next?** +----------------- + +For the next week, I plan to: + +1. Review all my Pull Requests with my mentor `Serge Koudoro `_, to ensure everything is up to FURY's standards. +2. Start working on the redesign of the FURY website, making it more user-friendly and visually appealing. + + +Thank you for reading. Stay tuned for more updates on my progress! diff --git a/fury/colormap.py b/fury/colormap.py index 66a5221f8..10659bc99 100644 --- a/fury/colormap.py +++ b/fury/colormap.py @@ -675,7 +675,7 @@ def rgb2hsv(rgb): out_v = rgb.max(-1) # -- S channel - delta = rgb.ptp(-1) + delta = np.ptp(rgb, -1) # Ignore warning for zero divided by zero old_settings = np.seterr(invalid="ignore") out_s = delta / out_v