Major Refactor and release on PyPi: Modularization into crates

[ODAncona]

After all improvements and features, I'm proud to release V3.0.0.

This PR concludes a 1 week dev sprint on code2prompt and introduces Four major modifications

1. Introduced a website that merges documentation and landing page.
2. Reorganized the codebase into crates
3. Released code2prompt on Pypi 🚀
4. Introduced a Discord Server for the code2prompt community

Right now, you can install it in your python environnement using:

pip install code2prompt_rs

---

Description of crates

The separation into crates will improve maintainability, performance, and ease of contribution for code2prompt community. The structure is inspired from the polars repository which has a similar use-case: a Rust lib with a python package.

Overview

• code2prompt : The CLI version hosted on Crates.io by @mufeedvh
• code2prompt-core: The core functionalities of code2prompt hosted on Crates.io by @ODAncona
• code2prompt-python: The python bindings for code2prompt hosted on Pypi by @ODAncona

Code2prompt

This crate contains the CLI logic and depends on code2prompt-core and has integration tests. The name was kept as is in order to keep the current behavior: the code2prompt crate is the CLI.

Code2prompt-core

This crate is the base logic of code ingestion and include:

• filtering logic
• git logic
• directory traversal logic
• sort logic
• template logic
• tokenizer logic

This is the place to develop prompt-compression techniques, LSP or MCP.

Code2prompt-python

This crate contains the bindings for python. Rye was used to handle packaging and python environment with rust natively and seamlessly.

Pypi Release:
The name code2prompt was already taken by this repo. Therefore I came up with code2prompt_rs to differentiate it. Both tools have similar functionalities, the former is written in python and the latter is written in rust. The former propose features that are not available "yet" in the latter.

These are:

1. Analyze Command: The Python version provides a command to generate detailed codebase statistics (file counts, directory structures, and extension lists) whereas the Rust version lacks this functionality.
2. Configuration File Loading: The Python tool automatically loads configuration options from .code2promptrc files found in the current and parent directories, while the Rust version requires all options to be set via command-line arguments.
3. Interactive File Selector: Python offers an interactive file selector using prompt_toolkit for browsing and selecting files, but although Rust includes an --interactive flag, it does not implement this feature.
4. Price Calculation and Display: The Python implementation estimates and displays processing costs for various LLMs using a formatted table with pricing data, whereas the Rust version has no such capability.

I concluded that it was better to publish it as another name rather than request the former community to join efforts.

---

I strongly believe code2prompt is a great choice for agentic AI as it's natively fast 🦀
I cannot thanks enough @mufeedvh to have started this project back in march 2024

I’m eager to discuss these improvements further and welcome any suggestions to align our visions for the future of Code2prompt.

It was a thrilling experience to develop this, and I'm all hears for what's next !