Review IMAP SDC decompression work

[alastairtree]

Maxine says that the "container is up and running" for the decompression processing logic. I assume this is the thing which actions API calls to the SDC data integration endpoints.

• code review
• Give them more test data
• Test it works
• compare the science data to what we decoded
• check the compression flags appear correctly in the CDF

[alastairtree]

PR for this is at IMAP-Science-Operations-Center/imap_processing#801

Hi all,

I just wanted to let you know that I have completed the MAG compression algorithm and it is up for code review now. I would really appreciate any thoughts or comments on that if you’re willing to take a look.

I was looking to pull data from webPODA to test it, and Alastair gave me some time ranges to check – but I wasn’t sure which packet they were in. Have you all been able to retrieve science data from webPODA successfully? Could you send me the URL you’ve been using to do that? It doesn’t look like the science packets are where I would expect (https://lasp.colorado.edu/ops/imap/poda/dap2/packets/SID2/), but I wanted to double check with you to see if I was missing something.

I asked the POC and Sasha Forsyth said that you need to add a new tab in the MAG_TLM spreadsheet for each science packet in order to have those packets show up in webPODA where I can access them. You would need to add a new tab for each science packet that defines the CCSDS header only on that tab in their spreadsheet and provide the POC with a new delivery. This is the only way that webPODA will know those packets will exist, even though we are never decomming them in real-time.

If you have any questions, you can reach out to Sasha ([email protected], also CC’ed here).

I have put the compression algorithm up in AWS, so if you have Level 0 data you would like to test, you can upload that to our API and the latest version of the code is running there. No guarantees about that working until I’m able to run through some real test data.

Please let me know if you have any questions!

Best,

Maxine

[tech3371]

Thank you Alastair for reviewing this PR! Maxine and we appreciate your feedbacks

[alastairtree]

Code review done, lots of minor stuff, nothing major. Maxine has flagged some issues she found in some packets - "that the end of the secondary vectors is cut off.... data I sent is from SHCOARSE time 461974160. "

Have asked @mfacchinelli to plot it

More info on the bad data

 formatted for input to the code you sent:

input = np.array([72, 240, 251, 245, 206, 219, 247, 184, 255, 247, 231, 255, 239, 126, 103, 254, 223, 158, 111, 255, 247, 254, 92, 78, 176, 29, 127, 166, 159, 239, 255, 223, 255, 249, 255, 251, 255, 103, 254, 127], dtype=np.uint8)

[alastairtree]

We have sent the data back to maxine for review, seems like a couple of issues in the logic.

[alastairtree]

Maxine has fixed bug and science now matches our data results except for a know issue with a 5s offset. Done.