Skip to content

Make header size functions public - #21

Open
mullr wants to merge 1 commit into
esrlabs:masterfrom
auxoncorp:expose-header-size-functions
Open

Make header size functions public#21
mullr wants to merge 1 commit into
esrlabs:masterfrom
auxoncorp:expose-header-size-functions

Conversation

@mullr

@mullr mullr commented May 23, 2024

Copy link
Copy Markdown

I'm using this library in a streaming context, where I don't necessarily have the whole buffer available when reading data. By exposing these functions, I can interpret the framing protocol directly, allowing me to stream in a single message at a time.

For reference, my usage looks something like this:

/// Read a single, complete DLT message from `stream`, and parse it.
pub async fn read_dlt_message<S>(
    stream: &mut S,
) -> Result<dlt_core::parse::ParsedMessage, anyhow::Error>
where
    S: AsyncRead + Unpin,
{
    // dlt_core only works on a byte buffer. So here we interpret the
    // framing part of the protocol just enough to determine the
    // message size, then read it into a buffer, and pass that down to
    // dlt_core.

    // Read the first byte, from which we can calculate the header size
    let header_type_byte = stream.read_u8().await?;
    let headers_len = dlt_core::dlt::calculate_all_headers_length(header_type_byte) as usize;

    // Read the whole header
    let mut header_buf = vec![0u8; headers_len];
    header_buf[0] = header_type_byte;
    stream.read_exact(&mut header_buf[1..]).await?;
    let (_, header) = dlt_core::parse::dlt_standard_header(&header_buf)?;

    // Read the payload. Put it in a single buffer together with the header
    let total_message_size = headers_len + header.payload_length as usize;
    let mut msg_buf = vec![0u8; total_message_size];
    (&mut msg_buf[0..headers_len]).copy_from_slice(&header_buf);
    stream.read_exact(&mut msg_buf[headers_len..]).await?;

    let (remaining_data, dlt_msg) = dlt_core::parse::dlt_message(&msg_buf, None, false)?;
    if !remaining_data.is_empty() {
        return Err(anyhow!(
            "Remaining data after loading DLT message: {remaining_data:?}"
        ));
    }

    Ok(dlt_msg)
}

@marcmo

marcmo commented Feb 3, 2025

Copy link
Copy Markdown
Member

Hello @mullr , your usecase is actually very familiar to us ;) I thought about it and I think it makes sense to also offer a stream based API for the dlt parser.
your implementation already looks quite good, maybe we can further improve on the memory allocation performance and also the error handling. we have a lot of test data, so I suggest starting with your version we implement this and create a PR, then you can also take a look at it.
alternatively you are welcome to create a PR. The issue for this is here: #34
that way, we also do not have to expose the private API

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants