feat: track package descriptions when loading (#2639)

# Goals

- Track descriptions during package loading such that partial
descriptions can be provided even if an error is returned
- Provide a `describe` cli subcommand for high-level descriptions of a
package, primarily for debug purposes
#2650. Since we want a description even
if some loading stage fails (e.g. extension resolution), tracking
partial descriptions is a requirement.

## Secondary goals
- Unify loading approaches across json/model, centralise more validation
logic
- Reduce burden on EnvelopeError - more local errors
- Replace existing debug extraction (generator, used extensions,
envelopeconfig) with single description

# Approach

In order to make this PR non-breaking I have deprecated existing reading
functions, and introduced new functions that return a full description
on successful read, and a partial when error. The deprecated functions
are most of the missing test coverage.

Reading naturally falls in to two stages: header and payload. If the
header fails no description is valid, so that error does not come with a
description. Payload errors can come with descriptions.

In order to build descriptions while loading I introduce the `Reader`
struct to track data across the reading stages. This is handed off to
the model `Context` when that stage takes over.

## Partial descriptions

Descriptions are defined in description.rs. They are intended to be
small amounts of simple data, so all fields are pub. All fields except
the envelope header are optional, a value of None indicates the field
has not been set yet.


## Appendix: JSON
Since the JSON format is effectively legacy I did not focus much on
incremental description tracking, it is more difficult since serde likes
to read all at once.


I found a surprise performance regression in JSON loading. In commit
[af01043](af01043)
I attempted to address this by removing an unnecessary two stage serde
deser, which has swung things in the other direction, improving
performance from prior.

## Appendix: EnvelopeError
EnvelopeError is used as a catch all for all reading and writing errors.
This PR goes some way to introducing new more specific errors for
reaeding. EnvelopeError should really in future have the reading errors
be taken out of it and renamed to `WriteError` or similar, then all the
conversions from the reading errors can be removed. Removing the
deprecated read functions is a pre-requisite for this. The merging of
this PR should be accompanied by issues for these follow ups.

Author: ss2165

hugr-cli/src/convert.rs

@@ -49,7 +49,7 @@ pub struct ConvertArgs {
 impl ConvertArgs {
     /// Convert a HUGR between different envelope formats
     pub fn run_convert(&mut self) -> Result<()> {
-        let (env_config, package) = self.input_args.get_envelope()?;
+        let (env_config, package) = self.input_args.get_described_package()?;
 
         // Handle text and binary format flags, which override the format option
         let mut config = if self.text {
@@ -67,7 +67,7 @@ impl ConvertArgs {
                     "model-text-exts" => EnvelopeFormat::ModelTextWithExtensions,
                     _ => Err(CliError::InvalidFormat(fmt.clone()))?,
                 },
-                None => env_config.format, // Use input format if not specified
+                None => env_config.header.config().format, // Use input format if not specified
             };
             EnvelopeConfig::new(format)
         };

hugr-cli/src/hugr_io.rs

@@ -1,7 +1,8 @@
 //! Input/output arguments for the HUGR CLI.
 
 use clio::Input;
-use hugr::envelope::{EnvelopeConfig, EnvelopeError, read_envelope};
+use hugr::envelope::description::PackageDesc;
+use hugr::envelope::{EnvelopeConfig, read_described_envelope};
 use hugr::extension::ExtensionRegistry;
 use hugr::package::Package;
 use hugr::{Extension, Hugr};
@@ -29,7 +30,7 @@ pub struct HugrInputArgs {
         short,
         long,
         help_heading = "Input",
-        help = "Paths to serialised extensions to validate against."
+        help = "Paths to additional serialised extensions needed to load the Hugr."
     )]
     pub extensions: Vec<PathBuf>,
     /// Read the input as a HUGR JSON file instead of an envelope.
@@ -48,7 +49,7 @@ impl HugrInputArgs {
     /// If [`HugrInputArgs::hugr_json`] is `true`, [`HugrInputArgs::get_hugr`] should be called instead as
     /// reading the input as a package will fail.
     pub fn get_package(&mut self) -> Result<Package, CliError> {
-        self.get_envelope().map(|(_, package)| package)
+        self.get_described_package().map(|(_, package)| package)
     }
 
     /// Read a hugr envelope from the input and return the envelope
@@ -58,13 +59,24 @@ impl HugrInputArgs {
     ///
     /// If [`HugrInputArgs::hugr_json`] is `true`, [`HugrInputArgs::get_hugr`] should be called instead as
     /// reading the input as a package will fail.
+    #[deprecated(since = "0.24.1", note = "Use get_described_envelope instead")]
     pub fn get_envelope(&mut self) -> Result<(EnvelopeConfig, Package), CliError> {
+        let (desc, package) = self.get_described_package()?;
+        Ok((desc.header.config(), package))
+    }
+
+    /// Read a hugr envelope from the input and return the envelope
+    /// description and the decoded package.
+    ///
+    /// # Errors
+    ///
+    /// If [`HugrInputArgs::hugr_json`] is `true`, [`HugrInputArgs::get_hugr`] should be called instead as
+    /// reading the input as a package will fail.
+    pub fn get_described_package(&mut self) -> Result<(PackageDesc, Package), CliError> {
         let extensions = self.load_extensions()?;
         let buffer = BufReader::new(&mut self.input);
-        read_envelope(buffer, &extensions).map_err(|e| match e {
-            EnvelopeError::MagicNumber { .. } => CliError::NotAnEnvelope,
-            _ => CliError::Envelope(e),
-        })
+
+        Ok(read_described_envelope(buffer, &extensions)?)
     }
     /// Read a hugr JSON file from the input.
     ///

hugr-cli/src/lib.rs

@@ -98,6 +98,9 @@ pub enum CliError {
         /// The generator of the HUGR.
         generator: Box<String>,
     },
+    #[error("Error reading envelope.")]
+    /// Errors produced when reading an envelope.
+    ReadEnvelope(#[from] hugr::envelope::ReadError),
 }
 
 impl CliError {

hugr-cli/src/mermaid.rs

@@ -43,9 +43,8 @@ impl MermaidArgs {
 
     /// Write the mermaid diagram for a HUGR envelope.
     pub fn run_print_envelope(&mut self) -> Result<()> {
-        let package = self.input_args.get_package()?;
-        let generator = hugr::envelope::get_generator(&package.modules);
-
+        let (desc, package) = self.input_args.get_described_package()?;
+        let generator = desc.generator();
         if self.validate {
             package
                 .validate()

hugr-cli/src/validate.rs

@@ -30,14 +30,15 @@ impl ValArgs {
         if self.input_args.hugr_json {
             #[allow(deprecated)]
             let hugr = self.input_args.get_hugr()?;
+            #[allow(deprecated)]
             let generator = hugr::envelope::get_generator(&[&hugr]);
 
             hugr.validate()
                 .map_err(PackageValidationError::Validation)
                 .map_err(|val_err| CliError::validation(generator, val_err))?;
         } else {
-            let package = self.input_args.get_package()?;
-            let generator = hugr::envelope::get_generator(&package.modules);
+            let (desc, package) = self.input_args.get_described_package()?;
+            let generator = desc.generator();
             package
                 .validate()
                 .map_err(|val_err| CliError::validation(generator, val_err))?;

hugr-cli/tests/convert.rs

@@ -7,7 +7,7 @@
 use assert_cmd::Command;
 use assert_fs::{NamedTempFile, fixture::FileWriteStr};
 use hugr::builder::{DataflowSubContainer, ModuleBuilder};
-use hugr::envelope::{EnvelopeConfig, EnvelopeFormat, read_envelope};
+use hugr::envelope::{EnvelopeConfig, EnvelopeFormat, read_described_envelope};
 use hugr::package::Package;
 use hugr::types::Type;
 use hugr::{
@@ -77,7 +77,9 @@ fn test_convert_to_json(test_envelope_file: NamedTempFile, mut convert_cmd: Comm
     let output_content = std::fs::read(output_file.path()).expect("Failed to read output file");
     let reader = BufReader::new(output_content.as_slice());
     let registry = ExtensionRegistry::default();
-    let (config, _) = read_envelope(reader, &registry).expect("Failed to read output envelope");
+    let (desc, _) =
+        read_described_envelope(reader, &registry).expect("Failed to read output envelope");
+    let config = desc.header.config();
 
     // Verify the format is correct
     assert_eq!(config.format, EnvelopeFormat::PackageJson);
@@ -102,8 +104,9 @@ fn test_convert_to_model(test_envelope_file: NamedTempFile, mut convert_cmd: Com
     let output_content = std::fs::read(output_file.path()).expect("Failed to read output file");
     let reader = BufReader::new(output_content.as_slice());
     let registry = ExtensionRegistry::default();
-    let (config, _) = read_envelope(reader, &registry).expect("Failed to read output envelope");
-
+    let (desc, _) =
+        read_described_envelope(reader, &registry).expect("Failed to read output envelope");
+    let config = desc.header.config();
     // Verify the format is correct
     assert_eq!(config.format, EnvelopeFormat::Model);
 }
@@ -174,7 +177,9 @@ fn test_convert_model_text_format(test_envelope_file: NamedTempFile, mut convert
     let output_content = std::fs::read(output_file.path()).expect("Failed to read output file");
     let reader = BufReader::new(output_content.as_slice());
     let registry = ExtensionRegistry::default();
-    let (config, _) = read_envelope(reader, &registry).expect("Failed to read output envelope");
+    let (desc, _) =
+        read_described_envelope(reader, &registry).expect("Failed to read output envelope");
+    let config = desc.header.config();
 
     // Verify the format is correct
     assert_eq!(config.format, EnvelopeFormat::ModelText);
@@ -192,14 +197,14 @@ fn test_format_roundtrip(test_package: Package) {
     let config_model = EnvelopeConfig::new(EnvelopeFormat::Model);
     let reader = BufReader::new(json_data.as_slice());
     let registry = ExtensionRegistry::default();
-    let (_, package) = read_envelope(reader, &registry).unwrap();
+    let (_, package) = read_described_envelope(reader, &registry).unwrap();
 
     let mut model_data = Vec::new();
     hugr::envelope::write_envelope(&mut model_data, &package, config_model).unwrap();
 
     // Convert back to JSON
     let reader = BufReader::new(model_data.as_slice());
-    let (_, package_back) = read_envelope(reader, &registry).unwrap();
+    let (_, package_back) = read_described_envelope(reader, &registry).unwrap();
 
     // Package should be the same after roundtrip conversion
     assert_eq!(test_package, package_back);
@@ -215,7 +220,9 @@ fn test_convert_text_flag(test_envelope_text: (String, Package), mut convert_cmd
 
     let reader = BufReader::new(stdout.as_slice());
     let registry = ExtensionRegistry::default();
-    let (config, _) = read_envelope(reader, &registry).expect("Failed to read output envelope");
+    let (desc, _) =
+        read_described_envelope(reader, &registry).expect("Failed to read output envelope");
+    let config = desc.header.config();
 
     // Verify it's a text-based format
     assert!(config.format.ascii_printable());
@@ -231,7 +238,9 @@ fn test_convert_binary_flag(test_envelope_text: (String, Package), mut convert_c
 
     let reader = BufReader::new(stdout.as_slice());
     let registry = ExtensionRegistry::default();
-    let (config, _) = read_envelope(reader, &registry).expect("Failed to read output envelope");
+    let (desc, _) =
+        read_described_envelope(reader, &registry).expect("Failed to read output envelope");
+    let config = desc.header.config();
 
     // Verify it's a binary format (not ASCII printable)
     assert!(!config.format.ascii_printable());

hugr-cli/tests/validate.rs

@@ -151,7 +151,7 @@ fn test_bad_json(mut val_cmd: Command) {
     val_cmd
         .assert()
         .failure()
-        .stderr(contains("Error decoding HUGR envelope"))
+        .stderr(contains("Error reading package payload in envelope"))
         .stderr(contains("missing field"));
 }