Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: upstream update from clara-rules 0.24.0 #496

Closed
wants to merge 89 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
afd749a
feat: initial commit removing cljs support and converting over to deps
k13gomez Dec 27, 2023
36d4283
chore: remove old project.clj file
k13gomez Dec 27, 2023
0c1ea12
fix: ensure buidl runs, add linter support
k13gomez Dec 27, 2023
fcff5c7
feat: ensure clean build deploy
k13gomez Dec 27, 2023
2478d16
fix: build add clojure and corretto jvm
k13gomez Dec 27, 2023
ec822d3
feat: use clojure
k13gomez Dec 27, 2023
5f8d726
feat: cleanup
k13gomez Dec 27, 2023
83306cb
feat: add pom file
k13gomez Dec 27, 2023
04ecb7a
chore: update ignore file
k13gomez Dec 27, 2023
c2986af
feat: dont delete pom
k13gomez Dec 27, 2023
892bf79
feat: make minor enhancements to cancelling cache
k13gomez Dec 27, 2023
c6e1dbe
feat: implement update cache using ham fisted mut list
k13gomez Dec 27, 2023
3a79d0a
feat: simply multiple equality wrappers into mostly single JEQ wrapper
k13gomez Dec 27, 2023
84b48ff
chore: formatting
k13gomez Dec 27, 2023
32192b0
feat: progress converting memory to mutable map/coll, add kaocha script
k13gomez Dec 28, 2023
8fa2b85
feat: remove unnecessary mutable flags and set statements
k13gomez Dec 28, 2023
1798006
feat: refactor elements and token fns using mutable map compute
k13gomez Dec 28, 2023
5f94ae0
chore: formatting and cleanup
k13gomez Dec 28, 2023
83fa452
feat: map persistent accum memory using compute
k13gomez Dec 28, 2023
281eef7
fix: lint deps
k13gomez Dec 28, 2023
fed0df7
fix: compute if present for remove accum
k13gomez Dec 28, 2023
9d0f22b
feat: update transient production memory ops using mutable ops
k13gomez Dec 28, 2023
67e0e57
feat: accum simplify using persistent strucs for results
k13gomez Dec 28, 2023
37eaa07
feat: memory tweaks to initialize nodes as persistent colls
k13gomez Dec 28, 2023
ac03512
feat: enhance compilter with hamf collections and update serialization
k13gomez Dec 29, 2023
1b3db0d
feat: add parallel processing support to various node types
k13gomez Dec 30, 2023
77f3174
feat: initial pass at poppping multiple activations for parallelization
k13gomez Dec 30, 2023
8abaa78
feat: first cut of parallel compute rhs activations
k13gomez Dec 30, 2023
c03f6f0
feat: add support for async rhs productions
k13gomez Dec 31, 2023
05fa0c0
chore: lint produce-try macro
k13gomez Dec 31, 2023
34f643d
feat: update pom
k13gomez Dec 31, 2023
cf3f864
feat: update docs
k13gomez Dec 31, 2023
f1b6ab6
fix: use correct build status url
k13gomez Jan 1, 2024
2090874
feat: update platform async patterns using futurama, prepare for release
k13gomez Jan 2, 2024
823512f
chore: update pom deps
k13gomez Jan 2, 2024
3ab4626
feat: async enhancements, add async tests
k13gomez Jan 2, 2024
3df55fe
chore: cleanup comments
k13gomez Jan 2, 2024
8463bd4
feat: add explicit async methods
k13gomez Jan 6, 2024
7f9e53b
feat: refactor fire rules into two separate async and sync handlers
k13gomez Jan 7, 2024
d74b0b9
WIP
k13gomez Jan 7, 2024
40b3e88
feat: refactor bound session and fire rules fns
k13gomez Jan 7, 2024
8f6770b
feat: disable rule order tests when doing parallel testing
k13gomez Jan 7, 2024
ac03bde
feat: update changelog and pom file for release
k13gomez Jan 7, 2024
a36ae24
feat: bump futurama to 0.3.8
k13gomez Jan 7, 2024
7be6dea
feat: add YourKit to README
k13gomez Jan 7, 2024
7645ec8
feat: bump futurama to 0.3.9
k13gomez Jan 7, 2024
302f99c
feat: add a built-in way to interrupt running session during rules
k13gomez Jan 8, 2024
c1faf0c
chore: release 0.9.3
k13gomez Jan 8, 2024
9afec6a
feat: update futurama lib version
k13gomez Jan 8, 2024
9cf439c
feat: update docs, explicit use async future for fire-rules-async
k13gomez Jan 9, 2024
1db3fbb
feat: async enhancements and add engine test with 10k async rules fired
k13gomez Jan 10, 2024
191c7b8
feat: update to latest async lib
k13gomez Jan 11, 2024
0db7edc
feat: update deps and format clj-kondo
k13gomez Jan 25, 2024
b45a54f
chore: update deps
k13gomez Jan 25, 2024
30f4c6c
feat: update pom deps
k13gomez Jan 25, 2024
f7db812
chore: fix futurama version in pom
k13gomez Jan 25, 2024
af3d083
feat: bump deps for release 1.0.0
k13gomez Feb 12, 2024
810b894
feat: initial major release 1.0.0
k13gomez Feb 12, 2024
98c7628
feat: bump futurama to latest version 1.0.1
k13gomez Feb 12, 2024
f3166f2
feat: release 1.0.2 with latest futurama version
k13gomez Feb 17, 2024
4bcea93
feat: implement custom caching support using core CacheProtocol
k13gomez Feb 19, 2024
58dc178
feat: update deps and change log and prepare for release 1.1.0
k13gomez Feb 19, 2024
d859436
feat: add support for a compiler cache which can cache exprs eval'd
k13gomez Feb 19, 2024
1e0c118
feat: don't include node id on compiled fns
k13gomez Feb 19, 2024
77f32e0
chore: set next dev version 1.2.0-SNAPSHOT
k13gomez Feb 19, 2024
2ea09cc
chore: update changelog and docs
k13gomez Feb 19, 2024
7b14dd7
feat: release 1.2.0
k13gomez Feb 19, 2024
eab3bcc
feat: enhancements of compiler hotspots using ham fisted mutables
k13gomez Feb 21, 2024
dfeee4e
feat: compiler class loading enhancements
k13gomez Feb 21, 2024
4ea3e1c
feat: prepare for release 1.3.0
k13gomez Feb 21, 2024
cf05a64
feat: update user ns
k13gomez Feb 21, 2024
0b40d90
feat: enhance caching performance by more predictable md5 caching and…
k13gomez Feb 22, 2024
e01304f
feat: enhance memory add-activations implementation by replacing get/…
k13gomez Feb 26, 2024
7506c0e
chore: remove deprecated either schema usage from compiler ns
k13gomez Feb 27, 2024
741707f
feat: extract way to build the production without adding to beta network
k13gomez Feb 27, 2024
a106cf1
feat: some more progress to support functions that are rules
k13gomez Feb 27, 2024
587a360
feat: change handler declaration to var, add var serde, add load sources
k13gomez Feb 27, 2024
271fa4f
chore: add useful user ns scribles
k13gomez Feb 27, 2024
a01d840
chore: run CI on all branches
k13gomez Feb 27, 2024
f897ad2
chore: minor enhancements for rule loading
k13gomez Feb 27, 2024
8d6ee9a
chore: prepare for snapshot release 1.4.0-SNAPSHOT
k13gomez Feb 27, 2024
07e0892
feat: store handler as symbol instead of var
k13gomez Feb 27, 2024
b27dc9e
feat: add coverage test to verify coverage works
k13gomez Feb 29, 2024
4b77d75
feat: upgrade clojure to 1.11.2 and fix clj-kondo linter
k13gomez Mar 8, 2024
bee29ff
feat: linter fixes
k13gomez Mar 10, 2024
b92f791
feat: implement hierarchy loaders, fact loaders, update deps (#2)
k13gomez Mar 10, 2024
364fa0f
chore: update deps
k13gomez Mar 11, 2024
0854469
Merge branch '1.3.3-hotfix'
k13gomez Mar 11, 2024
0d6a8ae
Merge remote-tracking branch 'upstream/main'
k13gomez May 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
feat: progress converting memory to mutable map/coll, add kaocha script
k13gomez committed Dec 28, 2023
commit 32192b08744ef0afafd443c22917e0c50accf43b
2 changes: 2 additions & 0 deletions bin/kaocha
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
#!/usr/bin/env bash
clojure -M:dev:test:runner $@
7 changes: 6 additions & 1 deletion dev/user.clj
Original file line number Diff line number Diff line change
@@ -1 +1,6 @@
(ns user)
(ns user
(:require [criterium.core :refer [report-result
quick-benchmark] :as crit]
[ham-fisted.api :as hf]
[ham-fisted.mut-map :as hm]
[clara.rules.platform :as platform]))
104 changes: 64 additions & 40 deletions src/main/clojure/clara/rules/memory.clj
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
(ns clara.rules.memory
"This namespace is for internal use and may move in the future.
Specification and default implementation of working memory"
(:require [ham-fisted.api :as hf])
(:import [java.util
Map
Collections
LinkedList
NavigableMap
PriorityQueue
TreeMap]))
TreeMap]
[ham_fisted MutableMap]))

(defprotocol IPersistentMemory
(to-transient [memory]))
@@ -154,15 +157,36 @@
(.add dest x))
dest)

(defn- linked-list?
[coll]
(instance? LinkedList coll))

(defn- ->linked-list
"Creates a new java.util.LinkedList from the coll, but avoids using
Collection.addAll(Collection) since there is unnecessary overhead
in this of calling Collection.toArray() on coll."
^java.util.List [coll]
(if (instance? LinkedList coll)
(if (linked-list? coll)
coll
(add-all! (LinkedList.) coll)))

(defn- mutable-map?
[m]
(instance? MutableMap m))

(defn- ->mutable-map
"Creates a new ham_fisted.MutableMap from the map, but only if necessary."
[m]
(if (mutable-map? m) m
(hf/mut-map m)))

(defn- ->persistent-coll
"Creates a persistent collection from the input collection, but only if necessary"
[coll]
(if (coll? coll)
coll
(seq coll)))

(defn- remove-first-of-each!
"Remove the first instance of each item in the given remove-seq that
appears in the collection coll. coll is updated in place for
@@ -417,11 +441,11 @@
activation-group-sort-fn
activation-group-fn
alphas-fn
^:unsynchronized-mutable alpha-memory
^:unsynchronized-mutable beta-memory
^:unsynchronized-mutable accum-memory
^:unsynchronized-mutable production-memory
^:unsynchronized-mutable ^NavigableMap activation-map]
^Map ^:unsynchronized-mutable alpha-memory
^Map ^:unsynchronized-mutable beta-memory
^Map ^:unsynchronized-mutable accum-memory
^Map ^:unsynchronized-mutable production-memory
^NavigableMap ^:unsynchronized-mutable activation-map]

IMemoryReader
(get-rulebase [memory] rulebase)
@@ -483,7 +507,8 @@

ITransientMemory
(add-elements! [memory node join-bindings elements]
(let [binding-element-map (get alpha-memory (:id node) {})
(let [node-id (:id node)
binding-element-map (get alpha-memory node-id {})
previous-elements (get binding-element-map join-bindings)]

(cond
@@ -492,7 +517,7 @@
(coll? previous-elements)
(set! alpha-memory
(assoc! alpha-memory
(:id node)
node-id
(assoc binding-element-map
join-bindings
(into previous-elements elements))))
@@ -507,15 +532,16 @@
elements
(set! alpha-memory
(assoc! alpha-memory
(:id node)
node-id
(assoc binding-element-map
join-bindings
elements))))))

(remove-elements! [memory node join-bindings elements]
;; Do nothing when no elements to remove.
(when-not (coll-empty? elements)
(let [binding-element-map (get alpha-memory (:id node) {})
(let [node-id (:id node)
binding-element-map (get alpha-memory node-id {})
previous-elements (get binding-element-map join-bindings)]
(cond
;; Do nothing when no previous elements to remove from.
@@ -537,7 +563,7 @@
remaining-elements))]
(set! alpha-memory
(assoc! alpha-memory
(:id node)
node-id
new-bindings-map))
removed-elements))

@@ -547,19 +573,20 @@
(when (.isEmpty ^java.util.List previous-elements)
(set! alpha-memory
(assoc! alpha-memory
(:id node)
node-id
(dissoc binding-element-map join-bindings))))
removed-elements)))))

(add-tokens! [memory node join-bindings tokens]
(let [binding-token-map (get beta-memory (:id node) {})
(let [node-id (:id node)
binding-token-map (get beta-memory node-id {})
previous-tokens (get binding-token-map join-bindings)]
;; The reasoning here is the same as in add-elements! impl above.
(cond
(coll? previous-tokens)
(set! beta-memory
(assoc! beta-memory
(:id node)
node-id
(assoc binding-token-map
join-bindings
(into previous-tokens tokens))))
@@ -570,15 +597,16 @@
tokens
(set! beta-memory
(assoc! beta-memory
(:id node)
node-id
(assoc binding-token-map
join-bindings
tokens))))))

(remove-tokens! [memory node join-bindings tokens]
;; The reasoning here is the same as remove-elements!
(when-not (coll-empty? tokens)
(let [binding-token-map (get beta-memory (:id node) {})
(let [node-id (:id node)
binding-token-map (get beta-memory node-id {})
previous-tokens (get binding-token-map join-bindings)]
(if (coll-empty? previous-tokens)
[]
@@ -589,18 +617,17 @@
;; equality though since those semantics are supported within the engine. This
;; slower path should be rare for any heavy retraction flows - such as those that come
;; via truth maintenance.
two-pass-remove! (fn [remaining-tokens tokens]
two-pass-remove! (fn do-remove-tokens
[remaining-tokens tokens]
(let [[removed-tokens not-removed-tokens]
(remove-first-of-each! tokens
remaining-tokens
(fn [t1 t2]
(fast-token-compare identical? t1 t2)))]
(partial fast-token-compare identical?))]

(if-let [other-removed (and (seq not-removed-tokens)
(-> not-removed-tokens
(remove-first-of-each! remaining-tokens
(fn [t1 t2]
(fast-token-compare = t1 t2)))
(partial fast-token-compare =))
first
seq))]
(into removed-tokens other-removed)
@@ -614,7 +641,7 @@
(assoc binding-token-map join-bindings remaining-tokens))]
(set! beta-memory
(assoc! beta-memory
(:id node)
node-id
new-tokens-map))
removed-tokens)

@@ -623,7 +650,7 @@
(when (.isEmpty ^java.util.List previous-tokens)
(set! beta-memory
(assoc! beta-memory
(:id node)
node-id
(dissoc binding-token-map join-bindings))))

removed-tokens)))))))
@@ -785,22 +812,19 @@
(.clear activation-map))

(to-persistent! [memory]
(let [->persistent-coll #(if (coll? %)
%
(seq %))
update-vals (fn [m update-fn]
(let [update-vals (fn do-update-vals [update-fn m]
(->> m
(reduce-kv (fn [m k v]
(assoc! m k (update-fn v)))
(transient m))
persistent!))
persistent-vals #(update-vals % ->persistent-coll)]
persistent-vals (partial update-vals ->persistent-coll)]
(->PersistentLocalMemory rulebase
activation-group-sort-fn
activation-group-fn
alphas-fn
(update-vals (persistent! alpha-memory) persistent-vals)
(update-vals (persistent! beta-memory) persistent-vals)
(update-vals persistent-vals (persistent! alpha-memory))
(update-vals persistent-vals (persistent! beta-memory))
(persistent! accum-memory)
(persistent! production-memory)
(into {}
@@ -882,10 +906,10 @@
activation-group-sort-fn
activation-group-fn
alphas-fn
(transient alpha-memory)
(transient beta-memory)
(transient accum-memory)
(transient production-memory)
(->mutable-map alpha-memory)
(->mutable-map beta-memory)
(->mutable-map accum-memory)
(->mutable-map production-memory)
(let [treemap (TreeMap. ^java.util.Comparator activation-group-sort-fn)]
(doseq [[activation-group activations] activation-map]
(.put treemap
@@ -901,8 +925,8 @@
activation-group-sort-fn
activation-group-fn
alphas-fn
{}
{}
{}
{}
{}))
(hf/hash-map)
(hf/hash-map)
(hf/hash-map)
(hf/hash-map)
(hf/hash-map)))
54 changes: 27 additions & 27 deletions src/test/clojure/clara/test_durability.clj
Original file line number Diff line number Diff line change
@@ -216,7 +216,7 @@
r-temp-his-res (query restored dr/temp-his)
r-temps-under-thresh-res (query restored dr/temps-under-thresh)

facts @(:holder mem-serializer)]
facts (sort-by hash @(:holder mem-serializer))]

(testing "Ensure the queries return same before and after serialization"
(is (= (frequencies [{:?ws (dr/->UnpairedWindSpeed ws10)}])
@@ -268,32 +268,32 @@

;; All of these facts must have an identical? relationship (same object references)
;; as the actual facts being tested against.
expected-facts [temp50
temp40
temp30
temp20
[temp50 temp40 temp30 temp20]
mci
lax
san
chi
twenty
cold20
unpaired-ws10
temp-his
ws50
ws40
ws10
irk
fifty
forty
thirty
thresh50
temps-under-thresh
hot40
hot30
hot50
[temp40 temp30 temp20]]]
expected-facts (sort-by hash [temp50
temp40
temp30
temp20
[temp50 temp40 temp30 temp20]
mci
lax
san
chi
twenty
cold20
unpaired-ws10
temp-his
ws50
ws40
ws10
irk
fifty
forty
thirty
thresh50
temps-under-thresh
hot40
hot30
hot50
[temp40 temp30 temp20]])]

(is (= (count expected-facts)
(count facts))