Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Starlark #341

Open
wants to merge 14 commits into
base: master
Choose a base branch
from
1 change: 1 addition & 0 deletions beehive.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ import (
"github.com/muesli/beehive/app"
"github.com/muesli/beehive/cfg"
_ "github.com/muesli/beehive/filters"
_ "github.com/muesli/beehive/filters/starlark"
_ "github.com/muesli/beehive/filters/template"

"github.com/muesli/beehive/bees"
Expand Down
3 changes: 3 additions & 0 deletions bees/actions.go
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,9 @@ func execAction(action Action, opts map[string]interface{}) bool {
}

bee := GetBee(a.Bee)
if bee == nil {
log.Fatal("cannot find bee ", a.Bee)
}
if (*bee).IsRunning() {
(*bee).LogAction()

Expand Down
21 changes: 16 additions & 5 deletions bees/filters.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@
package bees

import (
"strings"

log "github.com/sirupsen/logrus"

"github.com/muesli/beehive/filters"
Expand All @@ -35,15 +37,24 @@ type Filter struct {
}

// execFilter executes a filter. Returns whether the filter passed or not.
func execFilter(filter string, opts map[string]interface{}) bool {
f := *filters.GetFilter("template")
log.Println("\tExecuting filter:", filter)
func execFilter(source string, opts map[string]interface{}) bool {
name := "template"
if strings.Contains(source, "def main(") {
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not too happy with this "string magic". Maybe we could introduce proper filter types? I was also pondering having simple bash-scripts as filters, which either return 0 or 1 as a result.

Copy link
Contributor Author

@orsinium orsinium Oct 8, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm also not sure if it is a good solution but it requires a selector on the UI side and I'm not ready to dive into Ember. If you're ready to do such adjustments, please, do.

If to talk about a better backend-only solution, I can propose shebangs. In that case, this line will be if strings.HasPrefix(source, "#!/usr/bin/env starlark") , for example.

Bash scripts is a cool and fun idea. I'm sure some will be happy to build grep/sed/awk/perl pipelines.

name = "starlark"
}

filter := filters.GetFilter(name)
if filter == nil {
log.Error("cannot find filter", name)
return false
}
log.Println("\tExecuting filter:", source)

defer func() {
if e := recover(); e != nil {
log.Println("Fatal filter event:", e)
log.Error("Fatal filter event: ", e)
}
}()

return f.Passes(opts, filter)
return (*filter).Passes(opts, source)
}
41 changes: 41 additions & 0 deletions docs/filters.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Filters

Whenever an Event occurs, all of the Filters you defined in a Chain get executed with the data the Event provided. Only if the Event passes all of the Filters, the configured Actions in this Chain get then executed.

At the moment, there are 2 different types of filters:

+ temmplate
+ starlark

## Template

"Template" filter type uses [text/template](https://golang.org/pkg/text/template/) Go package to execute filters. Beehive exposes in the template a few helper functions and most of the [strings](https://golang.org/pkg/strings/) package functions to make templates more powerful. Also an important thing is that the template for filters should go inside `{{test ...}}`. That's pretty much it.

For example, let's check if `text` contains word "beehive":

```clojure
{{test Contains (ToLower .text) "beehive"}}
```

See [Beehive wiki](https://github.com/muesli/beehive/wiki/Filters) for more information.

## Starlark

[Starlark](https://github.com/bazelbuild/starlark) is a dialect of Python created for [Bazel](https://bazel.build/) configuration files. If you know Python, you already know Starlark. To make sure if a syntax feature is supported check [the specification](https://github.com/google/starlark-go/blob/master/doc/spec.md).

Beehive executes `main` function from the filter, passing all variables inside as keyword arguments. The function must return a boolean result.

For example, let's check if title or description of an RSS feed contains any of the given terms:

```python
def main(title, description, **kwargs):
text = title + " " + description
text = text.lower()
terms = ["beehive", "muesli"]
for term in terms:
if term in text:
return True
return False
```

"Starlark" filters are more verbose that "template" but much more powerful and turing-complete.
2 changes: 1 addition & 1 deletion filters/filters.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ type FilterInterface interface {
Description() string

// Execute the filter
Passes(data interface{}, value interface{}) bool
Passes(data map[string]interface{}, value string) bool
}

var (
Expand Down
50 changes: 50 additions & 0 deletions filters/starlark/dedent.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
// https://github.com/lithammer/dedent
package starlarkfilter

import (
"regexp"
"strings"
)

var (
whitespaceOnly = regexp.MustCompile("(?m)^[ \t]+$")
leadingWhitespace = regexp.MustCompile("(?m)(^[ \t]*)(?:[^ \t\n])")
)

// Dedent removes any common leading whitespace from every line in text.
//
// This can be used to make multiline strings to line up with the left edge of
// the display, while still presenting them in the source code in indented
// form.
func dedent(text string) string {
var margin string

text = whitespaceOnly.ReplaceAllString(text, "")
indents := leadingWhitespace.FindAllStringSubmatch(text, -1)

// Look for the longest leading string of spaces and tabs common to all
// lines.
for i, indent := range indents {
if i == 0 {
margin = indent[1]
} else if strings.HasPrefix(indent[1], margin) {
// Current line more deeply indented than previous winner:
// no change (previous winner is still on top).
continue
} else if strings.HasPrefix(margin, indent[1]) {
// Current line consistent with and no deeper than previous winner:
// it's the new winner.
margin = indent[1]
} else {
// Current line and previous winner have no common whitespace:
// there is no margin.
margin = ""
break
}
}

if margin != "" {
text = regexp.MustCompile("(?m)^"+margin).ReplaceAllString(text, "")
}
return text
}
179 changes: 179 additions & 0 deletions filters/starlark/dedent_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
// https://github.com/lithammer/dedent/blob/master/dedent_test.go
package starlarkfilter

import (
"fmt"
"testing"
)

const errorMsg = "\nexpected %q\ngot %q"

type dedentTest struct {
text, expect string
}

func TestDedentNoMargin(t *testing.T) {
texts := []string{
// No lines indented
"Hello there.\nHow are you?\nOh good, I'm glad.",
// Similar with a blank line
"Hello there.\n\nBoo!",
// Some lines indented, but overall margin is still zero
"Hello there.\n This is indented.",
// Again, add a blank line.
"Hello there.\n\n Boo!\n",
}

for _, text := range texts {
if text != dedent(text) {
t.Errorf(errorMsg, text, dedent(text))
}
}
}

func TestDedentEven(t *testing.T) {
texts := []dedentTest{
{
// All lines indented by two spaces
text: " Hello there.\n How are ya?\n Oh good.",
expect: "Hello there.\nHow are ya?\nOh good.",
},
{
// Same, with blank lines
text: " Hello there.\n\n How are ya?\n Oh good.\n",
expect: "Hello there.\n\nHow are ya?\nOh good.\n",
},
{
// Now indent one of the blank lines
text: " Hello there.\n \n How are ya?\n Oh good.\n",
expect: "Hello there.\n\nHow are ya?\nOh good.\n",
},
}

for _, text := range texts {
if text.expect != dedent(text.text) {
t.Errorf(errorMsg, text.expect, dedent(text.text))
}
}
}

func TestDedentUneven(t *testing.T) {
texts := []dedentTest{
{
// Lines indented unevenly
text: `
def foo():
while 1:
return foo
`,
expect: `
def foo():
while 1:
return foo
`,
},
{
// Uneven indentation with a blank line
text: " Foo\n Bar\n\n Baz\n",
expect: "Foo\n Bar\n\n Baz\n",
},
{
// Uneven indentation with a whitespace-only line
text: " Foo\n Bar\n \n Baz\n",
expect: "Foo\n Bar\n\n Baz\n",
},
}

for _, text := range texts {
if text.expect != dedent(text.text) {
t.Errorf(errorMsg, text.expect, dedent(text.text))
}
}
}

// dedent() should not mangle internal tabs.
func TestDedentPreserveInternalTabs(t *testing.T) {
text := " hello\tthere\n how are\tyou?"
expect := "hello\tthere\nhow are\tyou?"
if expect != dedent(text) {
t.Errorf(errorMsg, expect, dedent(text))
}

// Make sure that it preserves tabs when it's not making any changes at all
if expect != dedent(expect) {
t.Errorf(errorMsg, expect, dedent(expect))
}
}

// dedent() should not mangle tabs in the margin (i.e. tabs and spaces both
// count as margin, but are *not* considered equivalent).
func TestDedentPreserveMarginTabs(t *testing.T) {
texts := []string{
" hello there\n\thow are you?",
// Same effect even if we have 8 spaces
" hello there\n\thow are you?",
}

for _, text := range texts {
d := dedent(text)
if text != d {
t.Errorf(errorMsg, text, d)
}
}

texts2 := []dedentTest{
{
// dedent() only removes whitespace that can be uniformly removed!
text: "\thello there\n\thow are you?",
expect: "hello there\nhow are you?",
},
{
text: " \thello there\n \thow are you?",
expect: "hello there\nhow are you?",
},
{
text: " \t hello there\n \t how are you?",
expect: "hello there\nhow are you?",
},
{
text: " \thello there\n \t how are you?",
expect: "hello there\n how are you?",
},
}

for _, text := range texts2 {
if text.expect != dedent(text.text) {
t.Errorf(errorMsg, text.expect, dedent(text.text))
}
}
}

func Examplededent() {
s := `
Lorem ipsum dolor sit amet,
consectetur adipiscing elit.
Curabitur justo tellus, facilisis nec efficitur dictum,
fermentum vitae ligula. Sed eu convallis sapien.`
fmt.Println(dedent(s))
fmt.Println("-------------")
fmt.Println(s)
// Output:
// Lorem ipsum dolor sit amet,
// consectetur adipiscing elit.
// Curabitur justo tellus, facilisis nec efficitur dictum,
// fermentum vitae ligula. Sed eu convallis sapien.
// -------------
//
// Lorem ipsum dolor sit amet,
// consectetur adipiscing elit.
// Curabitur justo tellus, facilisis nec efficitur dictum,
// fermentum vitae ligula. Sed eu convallis sapien.
}

func BenchmarkDedent(b *testing.B) {
for i := 0; i < b.N; i++ {
dedent(`Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Curabitur justo tellus, facilisis nec efficitur dictum,
fermentum vitae ligula. Sed eu convallis sapien.`)
}
}
Loading