more plugins docs

README.md

@@ -90,12 +90,12 @@ Heimdall supports a growing set of pluggable command types:
 
 | Plugin      | Description                            | Execution Mode |
 | ----------- | -------------------------------------- | -------------- |
-| `ping`      | Basic plugin used for testing          | Sync or Async  |
+| `ping`      | [Basic plugin used for testing](https://github.com/patterninc/heimdall/blob/main/plugins/ping/README.md)          | Sync or Async  |
 | `shell`     | [Shell command execution](https://github.com/patterninc/heimdall/blob/main/plugins/shell/README.md)                | Sync or Async  |
-| `glue`      | Pulling Iceberg table metadata         | Sync or Async  |
-| `dynamodb`  | DynamoDB read operation                | Sync or Async  |
+| `glue`      | [Pulling Iceberg table metadata](https://github.com/patterninc/heimdall/blob/main/plugins/glue/README.md)         | Sync or Async  |
+| `dynamo`    | [DynamoDB read operation](https://github.com/patterninc/heimdall/blob/main/plugins/dynamo/README.md)                | Sync or Async  |
 | `snowflake` | Query execution in Snowflake           | Async          |
-| `spark`     | SparkSQL query execution on EMR on EKS | Async          |
+| `spark`     | [SparkSQL query execution on EMR on EKS](https://github.com/patterninc/heimdall/blob/main/plugins/spark/README.md) | Async          |
 
 ---
 

plugins/dynamo/README.md

@@ -0,0 +1,101 @@
+# 📦 DynamoDB Plugin
+
+The **DynamoDB Plugin** enables Heimdall to run read-only PartiQL queries against AWS DynamoDB tables. It securely connects using AWS credentials or an optional assumed IAM role, supports pagination, and returns structured query results.
+
+🔒 **Read-Only:** This plugin only supports data retrieval via PartiQL — no writes or modifications.
+
+---
+
+## 🧩 Plugin Overview
+
+* **Plugin Name:** `dynamo`
+* **Execution Mode:** Sync
+* **Use Case:** Querying DynamoDB tables using PartiQL from within Heimdall workflows
+
+---
+
+## ⚙️ Defining a DynamoDB Command
+
+### 📝 Command Definition
+
+```yaml
+- name: dynamo-0.0.1
+  status: active
+  plugin: dynamo
+  version: 0.0.1
+  description: Read data using PartiQL
+  tags:
+    - type:dynamodb
+  cluster_tags:
+    - type:dynamodb
+```
+
+### 🖥️ Cluster Definition (AWS Authentication)
+
+```yaml
+- name: dynamo-0.0.1
+  status: active
+  version: 0.0.1
+  description: AWS DynamoDB
+  context:
+    role_arn: arn:aws:iam::123456789012:role/HeimdallDynamoQueryRole
+  tags:
+    - type:dynamodb
+    - data:prod
+```
+
+* `role_arn` is optional. If provided, Heimdall will assume this IAM role to authenticate requests.
+
+---
+
+## 🚀 Submitting a DynamoDB Job
+
+Define the PartiQL query and optional result limit in the job context:
+
+```json
+{
+  "name": "list-items-job",
+  "version": "0.0.1",
+  "command_criteria": ["type:dynamo"],
+  "cluster_criteria": ["data:prod"],
+  "context": {
+    "query": "SELECT * FROM my_table WHERE category = 'books'",
+    "limit": 100
+  }
+}
+```
+
+---
+
+## 📊 Returning Job Results
+
+The plugin paginates through query results automatically, returning a structured output like:
+
+```json
+{
+  "columns": [
+    {"name": "id", "type": "string"},
+    {"name": "category", "type": "string"},
+    {"name": "price", "type": "float"}
+  ],
+  "data": [
+    ["123", "books", 12.99],
+    ["124", "books", 15.50]
+  ]
+}
+```
+
+Retrieve results via:
+
+```
+GET /api/v1/job/<job_id>/result
+```
+
+---
+
+## 🧠 Best Practices
+
+* Use IAM roles with **least privilege** for security.
+* Test queries in AWS console or CLI before running via Heimdall.
+* Avoid large result sets by leveraging the `limit` parameter.
+* Validate job input to prevent malformed PartiQL queries.

plugins/glue/README.md

@@ -0,0 +1,69 @@
+# 🍯 Glue Plugin
+
+The **Glue Plugin** enables Heimdall to query the AWS Glue Data Catalog to retrieve metadata about a specific table. It helps you integrate Glue metadata queries directly into your orchestration workflows.
+
+---
+
+## 🧩 Plugin Overview
+
+* **Plugin Name:** `glue`
+* **Execution Mode:** Sync
+* **Use Case:** Fetching AWS Glue table metadata for auditing, validation, or downstream processing
+
+---
+
+## ⚙️ Defining a Glue Command
+
+```yaml
+- name: glue-metadata-0.0.1
+  status: active
+  plugin: glue
+  version: 0.0.1
+  description: Query AWS Glue catalog for table metadata
+  context:
+    catalog_id: 123456789012
+  tags:
+    - type:glue-query
+  cluster_tags:
+    - type:localhost
+```
+
+* `catalog_id` is the AWS Glue catalog identifier (optional; defaults to AWS account ID if omitted).
+
+---
+
+## 🚀 Submitting a Glue Job
+
+Specify the Glue table name to fetch metadata for in the job context:
+
+```json
+{
+  "name": "fetch-glue-table-metadata",
+  "version": "0.0.1",
+  "command_criteria": ["type:glue-query"],
+  "cluster_criteria": ["data:local"],
+  "context": {
+    "table_name": "my_database.my_table"
+  }
+}
+```
+
+---
+
+## 📊 Returning Job Results
+
+The plugin returns raw metadata as a JSON string containing the table schema and properties.
+
+Retrieve results via:
+
+```
+GET /api/v1/job/<job_id>/result
+```
+
+---
+
+## 🧠 Best Practices
+
+* Use appropriate IAM permissions for Glue Catalog access.
+* Validate the fully qualified `table_name` to avoid query errors.
+* Use the metadata for auditing, documentation, or dynamic pipeline decisions.

plugins/ping/README.md

@@ -0,0 +1,102 @@
+# 🏓 Ping Plugin
+
+The **Ping Plugin** is a sample command used for testing Heimdall’s orchestration flow. Instead of sending actual ICMP packets, it responds instantly with a predefined message — perfect for dry runs, plugin testing, or just checking your Heimdall wiring. 🚧
+
+⚠️ **Testing Only:** This plugin is a *no-op*. It does **not** reach out to real hosts. Use it to verify that jobs run through Heimdall correctly.
+
+---
+
+## 🧩 Plugin Overview
+
+* **Plugin Name:** `ping`
+* **Execution Mode:** Sync
+* **Use Case:** Testing job submission, validation, or plugin behavior without side effects
+
+---
+
+## ⚙️ Defining a Ping Command
+
+You don’t need to specify much — just use the `ping` plugin and give it a name.
+
+```yaml
+- name: ping-0.0.1
+  status: active
+  plugin: ping
+  version: 0.0.1
+  description: Check Heimdall wiring
+  tags:
+    - type:ping
+  cluster_tags:
+    - type:localhost
+```
+
+🔹 When this job runs, Heimdall will simulate a ping and respond with a message like:
+
+```
+Hello, <calling user>!
+```
+
+---
+
+## 🖥️ Cluster Configuration
+
+Use a simple localhost cluster (or any compatible test target) to execute ping jobs:
+
+```yaml
+- name: localhost-0.0.1
+  status: active
+  version: 0.0.1
+  description: Localhost
+  tags:
+    - type:localhost
+    - data:local
+```
+
+---
+
+## 🚀 Submitting a Ping Job
+
+Here’s how to submit an example ping command via the Heimdall API:
+
+```json
+{
+  "name": "ping-check-job",
+  "version": "0.0.1",
+  "command_criteria": ["type:ping"],
+  "cluster_criteria": ["data:local"],
+  "context": {}
+}
+```
+
+🟢 This will run the ping plugin and instantly return result.
+
+---
+
+## 📊 Returning Job Results
+
+The plugin returns this result:
+
+```json
+{
+  "columns": [
+    {"name": "message", "type": "string"}
+  ],
+  "data": [
+    ["Hello, alice!"]
+  ]
+}
+```
+
+You can retrieve the result from:
+
+```
+GET /api/v1/job/<job_id>/result
+```
+
+---
+
+## 🧠 Best Practices
+
+* Use this plugin to **test your pipelines** before running real jobs.
+* It’s great for **CI/CD checks**, plugin regression tests, or mocking command behavior.
+* Don't forget: **no real pinging happens** — it's just a friendly "Hello!" 🎯