Mview Readme

What is Mview?

mview (Mask viewer) is a program in the style of Unix coreutils that lets you view raw data though a mask. The mask is configurable and supports the common data types. Of course mview is written in Rust and therefore 🚀🚀🚀 blazingly fast 🚀🚀🚀.

One of mview’s original applications was viewing messages that come from network. E.g. when you have an embedded device that sends it status via network over and over, Mview can read those messages and present them in a human readable way. But mview is very versatile and has many other applications, see examples below.

Examples

Watching network messages in a human readable way

Let’s say we have a socket that listens on port 3000. This can be simulated with netcat. We can read from that socket and pipe the output to mview:

nc -l -p 3000 | mview -c ./src/example_config --pause 100

Where --pause 100 just tells mview to wait with updating its output for 100ms. And the example_config passed to mview is this file:

MyData1:String:2
SomeOtherData:u8
SomeMoreData:bool8
EvenMoreData:String:3

(See below for the different datatypes.) The first String before the first : is an arbitrary identifier. You can name that field however you want, for the sake of your own confusion you can also give different fields the same name. Now we connect to that socket using netcat again: (in another terminal)

nc localhost 3000

We can now send data to that socket in stdin:

~ > nc localhost 3000
ab3456 # I typed this in and pressed return

mview will output

MyData1: ab
SomeOtherData: 51
SomeMoreData: true
EvenMoreData: 56

Mind that mview is trying to output the third byte it finds in the data it received as a u8. But since we typed characters into nc it will output the ASCII-value of the character 3, which is 51. The type bool8 will interpret a whole byte like a boolean. Like in C anything other than 0 is interpreted as true.

If we typed more data in that socket, mview would start over and display the next few bytes through that mask.

Reading captured data from tcpdump or wireshark

Data from tcpdump and such is usually captured in a file format called PCAP. Such files can be read by mview and the output of tcpdump can also be piped into mview. To do so, mview has the flag --pcap.

tcpdump -i eth0 -w - -U | mview -c ./myconfig --pcap

Or reading from a file

mview -c ./myconfig --pcap -i mycapture.pcap

The arguments --rawhex, --stats and --bitpos can then be used to get an idea of the length of a chunk, where it starts end ends, how long jumps over eventual IP headers should be etc.

Using --pcap and the argument --timestamp the timestamps that are extracted from the PCAP input file or the PCAP input stream are used. (Without --pcap the argument --timestamp prints the current time the chunk is written to the output as timestamp. --timestamp does not make much sense when reading from a raw file rather than a raw stream or PCAP file.)

Decoding a binary file

Lets say you have a super secret header in a binary file and you are tired of reading raw hexadecimal values. Lets make such a file first:

echo hello345test0000000 > ~/myFile

Here we are only interested at the data before the 0s start. The 0s represent the rest of the file that comes after the header. Here is the according config to that:

Shouldbehello:String:5
SomeOtherData:u8
SomeMoreData:bool8
SomeMoreData:u8
Shouldbetest:String:4

We know the header is 12 bytes long, so lets pipe that into mview:

~ > head -c 12 ~/myFile | mview -c ./src/example_config
Shouldbehello: hello
SomeOtherData: 51
SomeMoreData: true
SomeMoreData: 53
Shouldbetest: test

Instead of using the head command to get the first 12 bytes of the file, we could use the build in --head argument with the --infile to read directly from a file instead of stdin as well:

~ > mview -c /home/pascal/dev/mview/src/example_config --infile ~/myFile --head 12 # head makes mview read 12 bytes, then exit
Shouldbehello: hello
SomeOtherData: 51
SomeMoreData: true
SomeMoreData: 53
Shouldbetest: test

Configuration file

The tell mview what the mask is, we need to pass a configuration file to it with the -c / --config argument. The following points are the possible contents of that file.

Comments

Lines that start with a # are not evaluated by mview and can be used as comments. Also everything that comes after an # in a line is not evaluated. Example:

Myfieldname:String:3 # this line is evaluated, this comment not
# A comment that is not evaluated

Supported data types

bool1:

Evaluates a single bit in the chunk as true or false.

Example line in config

Myfieldname:bool1

bool8

Evaluates a whole byte to true or false in C style: Everything except 0 is true.

Example line in config

Myfieldname:bool8

u8, u16, u32, u64, u128, i8, i16 etc.

Evaluates 8, 16, 32 bits into an integer. I think the type names are self-explanatory, if not look them up here.

Example line in config

Myfieldname:i16
Myfieldname:i32:h # can also be displayed in hexadecimal
Myfieldname:i32:hex # works as well
Myfieldname:i32:hexadecimal # works as well
Myfieldname:u8:b # can also be displayed in binary
Myfieldname:u8:binary # works as well

f32 / f64

Evaluates a 32 bits / 64 bits into a floating point number.

Example line in config

Myfieldname:f32

iarb / uarb

In order to save a few bits of space sometimes integers don’t use full bytes in network messages. Therefore there is this type. The length operator (the number at the end in the config line below) represents the length of that field in bits, not bytes!

Example line in config

Myfieldname:iarb:7

String

Read a few bytes of the chunk and display them as ASCII characters. The length operator represents the length in bytes, not bits! Only one-byte characters are supported like standard strings in C.

Example line in config

Myfieldname:String:4

bytegap, bitgap

Sometimes you want to skip a few bytes and don’t display them in the output. You could just fill those with bool8 and bool1, but for tidiness sake there are those types. Mview will jump the bytes and bits ahead in a chunk and continue evaluation with the next config line. Bytegap takes a length operator in bytes and bitgap takes a length operator in bits.

Example line in config

Myfieldname:bytegap:2 # a 2 byte wide gap
Myfieldname:bitgap:4 # a 4 bit wide gap

Byte order

Because mview is primarily used for decoding network messages, integers that consist of several bytes are evaluated in network byte order (big-endian/ motorola order) by default. However, the expected byte order can be changed to little-endian (intel) order with the --le flag.

Messages, chunks and fields

mview receives messages from stdin or a file. It then divides a received message into chunks, where the size of a chunk is determined by the config. (The length of the datatypes added up.) If a datagram socket is read, usually the chunksize is the same like the messages size:

+--------------+   +--------------+        +------------------+
|   Message 1  |   |    Chunk 1   |------->| Data field 1     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |-->|              |------->| Data field 2     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |   |              |------->| Data field 3     |
+--------------+   +--------------+        +------------------+
+--------------+   +--------------+        +------------------+
|   Message 2  |   |    Chunk 1   |------->| Data field 1     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |-->|              |------->| Data field 2     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |   |              |------->| Data field 3     |
+--------------+   +--------------+        +------------------+

But if a stream socket is read, or a socket was recorded and that record is fed into mview, mview has no possible way to determine where one message ends and another starts. In this case (when the message is longer than chunk size) mview will take the length of a chunk in bytes from the message and display it fields, then continue to take the next length of a chunk from the message and display that until the whole message is processed.

+--------------+   +--------------+        +------------------+
|   Message 1  |   |    Chunk 1   |------->| Data field 1     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |-->|              |------->| Data field 2     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |   |              |------->| Data field 3     |
|              |   +--------------+        +------------------+
|              |   +--------------+        +------------------+
|              |   |    Chunk 2   |------->| Data field 1     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |-->|              |------->| Data field 2     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |   |              |------->| Data field 3     |
+--------------+   +--------------+        +------------------+
+--------------+   +--------------+        +------------------+
|   Message 2  |   |    Chunk 1   |------->| Data field 1     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |-->|              |------->| Data field 2     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |   |              |------->| Data field 3     |
|              |   +--------------+        +------------------+
|              |   +--------------+        +------------------+
|              |   |    Chunk 2   |------->| Data field 1     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |-->|              |------->| Data field 2     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |   |              |------->| Data field 3     |
+--------------+   +--------------+        +------------------+

In case the message size is not a whole multiple of chunk size, the data fields of the last message will be cut off: (this will be printed in mviews output as “values size is bigger than what is left of that data chunk”)

+--------------+   +--------------+        +------------------+
|   Message 1  |   |    Chunk 1   |------->| Data field 1     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |-->|              |------->| Data field 2     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |   |              |------->| Data field 3     |
|              |   +--------------+        +------------------+
|              |   +--------------+        +------------------+
|              |   |    Chunk 2   |------->| Data field 1     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |-->|              |------->| Data field 2     |
|______________|   |              |        +------------------+
                   |              |
                   |              |
                   +--------------+
+--------------+   +--------------+        +------------------+
|   Message 2  |   |    Chunk 1   |------->| Data field 1     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |-->|              |------->| Data field 2     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |   |              |------->| Data field 3     |
|              |   +--------------+        +------------------+
|              |   +--------------+        +------------------+
|              |   |    Chunk 2   |------->| Data field 1     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |-->|              |------->| Data field 2     |
|______________|   |              |        +------------------+
                   |              |
                   |              |
                   +--------------+

In the same way the data fields will be cut off and a message displayed for the fields that have no data will be displayed if the chunksize is bigger than message size:

+--------------+   +--------------+        +------------------+
|   Message 1  |   |    Chunk 1   |------->| Data field 1     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |-->|              |------->| Data field 2     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |   |              |------->| Data field 3     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |   |              |------->| Data field 4     |
|______________|   |              |        +------------------+
                   |              |
                   |              |
                   |              |
                   |              |
                   |              |
                   +--------------+
+--------------+   +--------------+        +------------------+
|   Message 2  |   |    Chunk 1   |------->| Data field 1     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |-->|              |------->| Data field 2     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |   |              |------->| Data field 3     |
|              |   |              |        +------------------+
|              |   |              |        +------------------+
|              |   |              |------->| Data field 4     |
|______________|   |              |        +------------------+
                   |              |
                   |              |
                   |              |
                   |              |
                   |              |
                   +--------------+

Usually the chunk size is calculated from the config file, but it can be set manually with the argument --chunksize. (In bytes) In the case that the sum of the length of the fields in config is bigger than the size from the argument, the message “values size is bigger than what is left of that data chunk” will be displayed for the fields that have no data. If the chunk size from the argument is bigger, the remaining bytes from that chunk will not be evaluated.

Record to --outfile

Please note that writing to a file is different that piping stdout of mview into a file like this:

mview -c ./src/example_config --stats > loggingfile.txt

Mview resets the cursor position for every chunk, therefore the output file would only have the last chunk printed in it like this:

Message no: 3
Message length: 30 bytes
Current chunk in this message: 2

Shouldbehello: oehus
SomeOtherData: 111
SomeMoreData: true
SomeMoreData: 115
Shouldbetest: hutn
EvenMoreData: eu

When using the ---outfile argument, mview will not reset the cursor position and the outfile will look like this:

Message no: 1
Message length: 37 bytes
Current chunk in this message: 1

Shouldbehello: otnes
SomeOtherData: 117
SomeMoreData: true
SomeMoreData: 115
Shouldbetest: oent
EvenMoreData: hsh

Message no: 1
Message length: 37 bytes
Current chunk in this message: 2

Shouldbehello: usneu
SomeOtherData: 116
SomeMoreData: true
SomeMoreData: 111
Shouldbetest: euho
EvenMoreData: euh

Message no: 1
Message length: 37 bytes
Current chunk in this message: 3

Shouldbehello: sotne
SomeOtherData: 117
SomeMoreData: true
SomeMoreData: values size is bigger than what is left of that data chunk
Shouldbetest: values size is bigger than what is left of that data chunk
EvenMoreData: values size is bigger than what is left of that data chunk

However, it is also possible to use the --nojump argument instead. This makes mview print to stdout like it does to files.

Printing chunks ‘on top of each other’

Normally mview resets the cursor position for every chunk. It therefore print chunks ‘on top of each other’, so to speak. This behavior is hard to implement because different terminal emulators have different features and react differently to the same control messages. Therefore it can happen, that mview deletes to many or to few lines, or it does not delete the lines at all. For this mview can also clear the terminal instead with the --clear flag. With this flag an output always starts at top of the terminal and the terminal is cleared before a chunk is printed.