Skip to content
/ cmdlet Public

Cmdlet is a python module provides pipe-like mechanism to cascade functions and generators.

License

Notifications You must be signed in to change notification settings

GaryLee/cmdlet

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

What is cmdlet?

Cmdlet provides pipe-like mechanism to cascade functions and generators. It uses symbol(|) to convert function to Pipe object and cascade them. This sequence of commands can be executed and evaluated later. Just like pipe mechanism in Unix shell. For example:

from cmdlet.cmds import *

# Create piped commands.
cmds = range(10) | pipe.filter(lambda x: x > 5) | fmt('item#%d')

# Execute commands and return the last processed data.
run(cmds)
# >>> 'item#9'

# Execute commands and return processed data in a list.
result(cmds)
# >>> ['item#6', 'item#7', 'item#8', 'item#9']

# Execute commands and return iterator for processed data.
for data in cmds:
    print data
# >>> item#6
# >>> item#7
# >>> item#8
# >>> item#9

First, we created commands and used | to cascade them. Then, we can execute commands by run(), result() or iterator.

cmdlet can convert corresponding types to Pipe object automatically. In above example, range(10) is an iterator not a Pipe object. Because second item is a Pipe object(made by pipe.filter), it turns out first item to be converted to a Pipe object automatically.

There are many useful utilities in cmdlet.cmds modules. They can provide a great convenience to build up useful pipes. Here is an example:

from cmdlet.cmds import *

query_topic =
    'find ./mydoc -name "*.txt" -print' |
    readline(end=10) |
    match(r'^[tT]opic:\s*(?P<topic>.+)\s*', to=dict) |
    values('topic')

for topic in query_topic:
    print topic

In above example, the goal is to query topic from article files. To achieve the goal, we have to:

  1. Search text files in a given folder.
  2. Read first 10 lines from each file.
  3. Find the line that matched 'topic: foo bar' pattern.
  4. Extract the topic string.

With the utilities provided by cmdlet.cmds, we only need to write a few of code. The first string which starts with 'find' is a normal shell script. It is converted to sh pipe automatically and executed with system shell. The readline pipe can open files whose name passed from sh pipe. match pipe and values pipe work together to extract topic from file content.

Above example shows not only small code but also readability. It's really easy to understand the purpose of source code.

NOTE:

When using cmdlet's pipe mechanism, make sure one of your first two pipe items is a valid Pipe object.

There is another advantage to use cmdlet. The pipe object is evaluated when calling result, run or iter. It implies you can reuse them. Let's modify previous example.

from cmdlet.cmds import *

# Separate from query_topic command.
extract_topic =
    readline(end=10) |
    match(r'^[tT]opic:\s*(?P<topic>.+)\s*', to=dict) |
    values('topic')

for topic in ('find ./mydoc1 -name "*.txt" -print' | extract_topic):
    print topic

for topic in ('find ../mydoc2 -name "*.md" -print' | extract_topic):
    print topic

How to install

Just like other packages on PyPI. You can use pip to download and install automatically.

$ pip install cmdlet

Alternatively, you can install from github directly.

$ pip install git+https://github.com/GaryLee/cmdlet.git

Run piped commands and get result

There are 3 ways to execute piped commands and get the result.

  1. Use run(cmds) or cmds.run() to execute cmds and get the last processed data. Use this if you don't need all processed data. Or, the tasks you need to do have been done by cascaded Pipe objects.

  2. Use result(cmds) or cmds.result() to get the processed data in a list. Use this method when you need to take all processed data to other mechanisms.

  3. Use cmds as a iterator to handle the processed data one by one. It treats cascaded Pipe objects as a pre-processing function. Use it to process data and invoke it by a for loop to do the last processing by yourself.

Function wrapper

Function should not be used in pipes directly, unless using auto-type conversion. Cmdlet provides a set of basic wrappers to wrap function to Pipe object.

pipe.func(generator_function)

The most basic wrapper. In Python, generator function is a function with yield statement in it. The generator_function defined here is a Python generator function with at least one argument. The first argument is a generator object passed by previous Pipe object. generator_function can take it as input or just leave it. It looks like:

# Generator function which use prev as input.
@pipe.func
def my_generator(prev):
    for data in prev:
        # ... Put some code to process data ...
        yield new_data
# Generator function which ignore input.
@pipe.func
def my_generator_ignore_prev(prev):
    while True:
        # ... Generate data and break loop in some conditions. ...
        yield data

For example:

@pipe.func
def randint_generator(prev, num):
    for i in range(num):
        yield random.randint(0, 1000)

@pipe.func
def power(prev, th):
    for n in prev:
        yield n ** th

cmds = randint_generator(10) | power
ans = result(cmds)
# Equals to:
# ans = []
# for i in range(10):
#     ans.append(random.randint(0, 1000)

pipe.map(function)

Wrap function to a mapper. The input is a normal function with at least one argument for data input. The returned value will be passed to next Pipe object. It looks like:

@pipe.map
def my_mapper(data):
    # ... Put some code to process data ...            
    return new_data

For example:

@pipe.func
def randint_generator(prev, num):
    for i in range(num):
        yield random.randint(0, 1000)

@pipe.map
def power(n, th):
    return n ** th

cmds = randint_generator(10) | power
ans = result(cmds)
# Equals to:
# ans = []
# for i in range(10):
#     ans.append(random.randint(0, 1000)

The power pipe can also be written in this way:

power = pipe.map(lambda n, th: n ** th)

Anything returned by mapper will be sent to next Pipe object. If mapper return None, next Pipe object will receive None. That is, you can't use mapper to filter data out. That's why we have pipe.filter.

pipe.filter(function)

Wrap function to a filter. Filter is a function with at least one argument as data input. Filter should return Boolean value, True or False. If True, data from previous Pipe object is allowed to pass through. If False, data is dropped. It looks like:

@pipe.filter
def my_filter(data):
    # Handle data and check conditions.
    if you_should_not_pass:
        return False
    else:
        return True

For example:

@pipe.filter
def less_than(data, thrd):
    return data < thrd

cmds = range(10) | less_than(3)
ans = result(cmds)
# Equals to:
# ans = []
# thrd = 3
# for n in range(10):
#     if n < thrd:
#          ans.append()

You can write filter pipe in this way:

less_than = pipe.filter(lambda data, thrd: data < thrd)

pipe.reduce(function)

Wrap function as a reducer. A reducer is a function which has at least two arguments. The first one is the data to be processed. The second one is used as accumulated result. A optional keyword argument init can be used to specify initial value to accumulated result. It looks like:

@pipe.reduce
def my_reducer(data, accum_result):
    # Calculate new accum_result according to data.
    return accum_result + len(data)

For example:

@pipe.reduce
def count_mod(data, accum_result, mod_by):
    if (data % mod_by) == 0:
        return accum_result
    else:
        return accum_result + 1

cmds = range(1000) | count_mod(10, init=0)

pipe.stopper(function)

Wrap function as a stopper. Stopper is used to stop the pipe execution. It returns true to stop the pipe execution. Return false to pass data to next. It looks like:

@pipe.stopper
def my_stopper(data):
    if check_stop_criteria(data):
        return True
    return False

The usage of wrapper

Here is a example to show how to use function wrapper.

from random import randint
from cmdlet.cmds import *

@pipe.func
def random_number(prev, amount):
    for i in range(amount):
        yield randint(0, 100000)

@pipe.filter
def in_range(data, lower_bound, upper_bound):
    return data >= lower_bound and data <= upper_bound

@pipe.reduce
def count(data, accum_result):
    return accum_result + 1

@pipe.map
def format_output(data, format):
    return format % data

# Generate 1000 random number and count how many of them between 100 and 500.
# Then, format the result to 'ans=%d'.
cmds = random_number(1000) | in_range(100, 500) | count(init=0) | format_output('ans=%d')

print cmds.run()
# >>> ans=40

If wrapped code is just a expression, following code shows another way to make them:

in_range = pipe.filter(lambda data: data >= lower_bound and data <= upper_bound)
count = pipe.reduce(lambda accum_result, data: accum_result + 1)
format_output = pipe.reduce(lambda data, format: format % data)

NOTE:

As you might already noticed, the number of argument using in piped commands is different from the definition of wrapped function. You should know your function is wrapped to a Pipe object. The function is not invoked when cascading pipes. It is called when using run(), result() or iteration. The arguments will be stored in Pipe object and append to the argument list of wrapped function when it is invoked.

Auto-type conversion

If the operand of | operator is not a Pipe object, cmdlet will call proper creator to convert and wrap it to a Pipe object. The data type of operand must be registered in cmdlet. Otherwise, exception UnregisteredPipeType will be raised.

cmdlet.cmds has registered some basic types by default. You can use them directly.

Type wrapper Description
type pipe.map Convert processed data to specified type
function pipe.map Wrap function as a mapper.
method pipe.map Wrap method as a mapper.
tuple seq Wrap tuple to generator.
list seq Wrap list to generator.
str sh Wrap string to command line and execute it.
unicode sh Wrap string to command line and execute it.
file fileobj Wrap file object for read/write operation.

cmdlet.cmds utilities.

cmdlet.cmds has predefined some commands. Here are brief descriptions.

Pipe commnds for iterable object.

Command Description
pack Take N elements from pipe and group them into one element.
enum Generate (index, value) pair from previous pipe.
counter Count the number of data from previous pipe.
flatten Flatten the data passed from previous pipe.
items Extract (key, value) pair from a dict-like object.
seq Extract any iterable object.
attr Extract the value of given attribute from previous pipe.
attrs Extract the value of given attributes from previous pipe.
attrdict Extract the value of given attributes from previous pipe.

Pipe commands for file

Command Description
stdout Output data from previous pipe to stdout.
stderr Output data from previous pipe to stderr.
readline Read data from file line by line.
fileobj Read/write file with pipe data.

Pipe commands for shell

Command Description
sh Execute system shell script to handle the stdin/stdout. The data from previous pipe will be the standard input of specified shell command.
execmd Execute system shell script to handle the stdin/stdout. The data from previous pipe will be the command line to be executed.

Pipe commands for strings

Alias of string method

Command Description
upper alias of string.upper
lower alias of string.lower
capwords alias of string.capwords
capitalize alias of string.capitalize
lstrip alias of string.lstrip
rstrip alias of string.rstrip
strip alias of string.strip
expandtabs alias of string.expandtabs
strip alias of string.strip
find alias of string.find
fmt alias of % operator of string (not string.format)
rfind alias of string.rfind
count alias of string.count
split alias of string.split
rsplit alias of string.rsplit
swapcase alias of string.swapcase
translate alias of string.translate
ljust alias of string.ljust
rjust alias of string.rjust
center alias of string.center
zfill alias of string.zfill
replace alias of string.replace
join alias of string.join
substitute alias of string.Template.substitute
safe_substitute alias of string.Template.safe_substitute

String split, search and match

Command Description
grep Grep strings with regular expression.
match Grep strings with regular expression and generate MatchObject.
wildcard Grep strings with wildcard character.
resplit Split strings with regular expression.
sub Substitute strings with regular expression.
subn Substitute strings with regular expression.

About

Cmdlet is a python module provides pipe-like mechanism to cascade functions and generators.

Resources

License

Stars

Watchers

Forks

Packages

No packages published