Merge pull request #11 from Sammyjo20/feature/haystack-state

Feature | Haystack Data

Author: Sammyjo20

README.md

@@ -40,6 +40,7 @@ That's right! Let's just be clear that we're not talking about **Batched Jobs**.
 - They are volatile, meaning if you lose one job in the chain - you lose the whole chain.
 - They do not provide the `then`, `catch`, `finally` callable methods that batched jobs do.
 - Long delays with memory based or SQS queue is not possible as you could lose the jobs due to expiry or if the server shuts down.
+- You can't share data between jobs as there is no "state" across the chain
 
 Laravel Haystack aims to solve this by storing the job chain in the database and queuing one job at a time. When the job is completed, Laravel Haystack listens out for the "job completed" event and queues the next job in the chain from the database.
 
@@ -49,6 +50,7 @@ Laravel Haystack aims to solve this by storing the job chain in the database and
 - It provides callback methods like `then`, `catch` and `finally`.
 - Global middleware that can be applied to every single job in the chain
 - Delay that can be added to every job in the chain
+- You can store and retrieve data/state that is accessible to every job in the chain.
 - You can store the model for later processing.
 
 ### Use Cases
@@ -429,6 +431,102 @@ class ProcessPodcast implements ShouldQueue, StackableJo
     }
 ```
 
+## Shared Job Data
+
+Laravel Haystack has the ability for your jobs to store and retrieve state/data between jobs. This is really useful if you need to store data in the first job and then in the second job, process the data and in the final job, email the processed data. You are able to create a process pipeline since each job is processed in sequential order. This is really exciting because with traditional chained jobs, you cannot share data between jobs.
+
+```php
+<?php
+
+$haystack = Haystack::build()
+   ->addJob(new RetrieveDataFromApi)
+   ->addJob(new ProcessDataFromApi)
+   ->addJob(new StoreDataFromApi)
+   ->dispatch();
+```
+
+### Storing data inside of jobs
+
+Inside your job, you can use the `setHaystackData()` method to store some data. This method accepts a key, value and optional Eloquent cast.
+
+```php
+<?php
+ 
+namespace App\Jobs;
+ 
+use Sammyjo20\LaravelHaystack\Contracts\StackableJob;
+use Sammyjo20\LaravelHaystack\Concerns\Stackable;
+ 
+class RetrieveDataFromApi implements ShouldQueue, StackableJob
+{
+    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels, Stackable
+
+    public function handle()
+    {
+        // Your application code...
+        
+        $this->setHaystackData('username', 'Sammyjo20');
+    }
+```
+
+### Casting data 
+
+The `setHaystackData` method supports any data type. It supports fully casting your data into any of Eloquent's existing casts, or even your custom casts. Just provide a third argument to specify the cast.
+
+```php
+<?php
+ 
+namespace App\Jobs;
+ 
+use Sammyjo20\LaravelHaystack\Contracts\StackableJob;
+use Sammyjo20\LaravelHaystack\Concerns\Stackable;
+ 
+class RetrieveDataFromApi implements ShouldQueue, StackableJob
+{
+    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels, Stackable
+
+    public function handle()
+    {
+        // Your application code...
+        
+        // Array Data, provide third argument to specify cast.
+        
+        $this->setHaystackData('data', ['username' => 'Sammyjo20'], 'array');
+        
+        // Supports custom casts
+        
+        $this->setHaystackData('customData', $object, CustomCast::class);
+    }
+```
+
+### Retrieving data inside of jobs
+
+From one job you can set the data, but that data will be available to every job in the haystack there after. Just use the `getHaystackData` method to get data by key or use the `allHaystackData` to get a collection containing the haystack data.
+
+```php
+<?php
+ 
+namespace App\Jobs;
+ 
+use Sammyjo20\LaravelHaystack\Contracts\StackableJob;
+use Sammyjo20\LaravelHaystack\Concerns\Stackable;
+ 
+class RetrieveDataFromApi implements ShouldQueue, StackableJob
+{
+    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels, Stackable
+
+    public function handle()
+    {
+        // Get data by key
+        
+        $username = $this->getHaystackData('username'); // Sammyjo20
+        
+        // Get all data
+        
+        $allData = $this->allHaystackData(); // Collection: ['username' => 'Sammyjo20']
+    }
+```
+
 ## Manual Processing
 
 By default, Laravel Haystack will use events to automatically process the next job in the haystack. If you would like to disable this functionality, you will need to make sure to disable the `automatic_processing` in your config/haystack.php file. After that, you will need to make sure your jobs tell Haystack when to process the next job.

composer.json

@@ -26,6 +26,7 @@
         "spatie/laravel-package-tools": "^1.9.2"
     },
     "require-dev": {
+        "jessarcher/laravel-castable-data-transfer-object": "^2.2",
         "laravel/pint": "^1.0",
         "orchestra/testbench": "^7.0",
         "pestphp/pest": "^1.21",

config/haystack.php

@@ -2,6 +2,19 @@
 
 return [
 
+    /*
+   |--------------------------------------------------------------------------
+   | Return All Haystack Data When Finished
+   |--------------------------------------------------------------------------
+   |
+   | This value if set to true, will instruct Haystack to query all the
+   | haystack data rows out of the database and return them to the
+   | then/finally/catch blocks as a collection.
+   |
+   */
+
+    'return_all_haystack_data_when_finished' => true,
+
     /*
     |--------------------------------------------------------------------------
     | Queue Haystack Jobs Automatically

database/factories/HaystackDataFactory.php

@@ -0,0 +1,21 @@
+<?php
+
+namespace Sammyjo20\LaravelHaystack\Database\Factories;
+
+use Sammyjo20\LaravelHaystack\Models\HaystackData;
+use Illuminate\Database\Eloquent\Factories\Factory;
+
+class HaystackDataFactory extends Factory
+{
+    protected $model = HaystackData::class;
+
+    /**
+     * Definition
+     *
+     * @return array|mixed[]
+     */
+    public function definition()
+    {
+        return [];
+    }
+}

database/migrations/create_haystack_data_table.php.stub

@@ -0,0 +1,28 @@
+<?php
+
+use Sammyjo20\LaravelHaystack\Models\Haystack;
+use Illuminate\Database\Migrations\Migration;
+use Illuminate\Database\Schema\Blueprint;
+use Illuminate\Support\Facades\Schema;
+
+return new class extends Migration
+{
+    public function up()
+    {
+        Schema::create('haystack_data', function (Blueprint $table) {
+            $table->id();
+            $table->timestamps();
+            $table->foreignIdFor(Haystack::class)->constrained()->cascadeOnDelete();
+            $table->string('key');
+            $table->binary('value');
+            $table->string('cast')->nullable();
+
+            $table->unique(['haystack_id', 'key']);
+        });
+    }
+
+    public function down()
+    {
+        Schema::dropIfExists('haystack_data');
+    }
+};

database/migrations/create_haystacks_table.php.stub

@@ -18,6 +18,7 @@ return new class extends Migration
             $table->dateTime('started_at')->nullable();
             $table->dateTime('resume_at')->nullable();
             $table->dateTime('finished_at')->nullable();
+            $table->boolean('return_data')->default(true);
         });
     }
 

src/Builders/HaystackBuilder.php

@@ -69,6 +69,13 @@ class HaystackBuilder
      */
     protected ?Closure $globalMiddleware = null;
 
+    /**
+     * Should we return the data when the haystack finishes?
+     *
+     * @var bool
+     */
+    protected bool $returnDataOnFinish = true;
+
     /**
      * Constructor
      */
@@ -263,13 +270,26 @@ protected function createHaystack(): Haystack
         $haystack->on_catch = $this->onCatch;
         $haystack->on_finally = $this->onFinally;
         $haystack->middleware = $this->globalMiddleware;
+        $haystack->return_data = $this->returnDataOnFinish;
         $haystack->save();
 
         $haystack->bales()->insert($this->prepareJobsForInsert($haystack));
 
         return $haystack;
     }
 
+    /**
+     * Specify if you do not want haystack to return the data.
+     *
+     * @return $this
+     */
+    public function dontReturnData(): static
+    {
+        $this->returnDataOnFinish = false;
+
+        return $this;
+    }
+
     /**
      * Get all the jobs in the builder.
      *

src/Concerns/ManagesBales.php

@@ -5,10 +5,12 @@
 use Closure;
 use Carbon\CarbonImmutable;
 use Carbon\CarbonInterface;
-use Illuminate\Queue\Jobs\Job;
+use InvalidArgumentException;
+use Illuminate\Support\Collection;
 use Illuminate\Contracts\Queue\ShouldQueue;
 use Sammyjo20\LaravelHaystack\Data\NextJob;
 use Sammyjo20\LaravelHaystack\Models\HaystackBale;
+use Sammyjo20\LaravelHaystack\Models\HaystackData;
 use Sammyjo20\LaravelHaystack\Helpers\CarbonHelper;
 use Sammyjo20\LaravelHaystack\Contracts\StackableJob;
 use Sammyjo20\LaravelHaystack\Data\PendingHaystackBale;
@@ -152,13 +154,17 @@ public function finish(bool $fail = false): void
 
         $this->update(['finished_at' => now()]);
 
+        $returnAllData = config('haystack.return_all_haystack_data_when_finished', false);
+
+        $allData = $this->return_data === true && $returnAllData === true ? $this->allData() : null;
+
         $fail === true
-            ? $this->executeClosure($this->on_catch)
-            : $this->executeClosure($this->on_then);
+            ? $this->executeClosure($this->on_catch, $allData)
+            : $this->executeClosure($this->on_then, $allData);
 
         // Always execute the finally closure.
 
-        $this->executeClosure($this->on_finally);
+        $this->executeClosure($this->on_finally, $allData);
 
         // Now finally delete itself.
 
@@ -213,12 +219,13 @@ public function appendPendingJob(PendingHaystackBale $pendingJob): void
      * Execute the closure.
      *
      * @param  Closure|null  $closure
+     * @param  Collection|null  $data
      * @return void
      */
-    protected function executeClosure(?Closure $closure): void
+    protected function executeClosure(?Closure $closure, ?Collection $data = null): void
     {
         if ($closure instanceof Closure) {
-            $closure();
+            $closure($data);
         }
     }
 
@@ -232,4 +239,52 @@ public function pause(CarbonImmutable $resumeAt): void
     {
         $this->update(['resume_at' => $resumeAt]);
     }
+
+    /**
+     * Store data on the Haystack.
+     *
+     * @param  string  $key
+     * @param  mixed  $value
+     * @param  string|null  $cast
+     * @return ManagesBales|\Sammyjo20\LaravelHaystack\Models\Haystack
+     */
+    public function setData(string $key, mixed $value, string $cast = null): self
+    {
+        if (is_null($cast) && is_string($value) === false && is_int($value) === false) {
+            throw new InvalidArgumentException('You must specify a cast if the value is not a string or integer.');
+        }
+
+        $this->data()->updateOrCreate(['key' => $key], [
+            'cast' => $cast,
+            'value' => $value,
+        ]);
+
+        return $this;
+    }
+
+    /**
+     * Retrieve data by a key from the Haystack.
+     *
+     * @param  string  $key
+     * @param  mixed|null  $default
+     * @return mixed
+     */
+    public function getData(string $key, mixed $default = null): mixed
+    {
+        $data = $this->data()->where('key', $key)->first();
+
+        return $data instanceof HaystackData ? $data->value : $default;
+    }
+
+    /**
+     * Retrieve all the data from the Haystack.
+     *
+     * @return Collection
+     */
+    public function allData(): Collection
+    {
+        return $this->data()->orderBy('id')->get()->mapWithKeys(function ($value, $key) {
+            return [$value->key => $value->value];
+        });
+    }
 }