HF dataset integration

[marconaguib]

Description

Introduces :

• from_huggingface_hub() : a data connector to stream/load HuggingFace datasets into edsnlp Streams (supports split/config auto-detection, shuffle, looping, and converter kwargs).
• to_huggingface_hub() : export Stream or Docs to a datasets.IterableDataset, with optional materialize-and-push (push_to_hub=True).
• hf_ner : converter to read/write token-level NER HF datasets.
• hf_text : converter to read/write text-only HF datasets.

N.B. sorry for the messy last PR, I closed it and opened a cleaner one

Checklist

• If this PR is a bug fix, the bug is documented in the test suite.
• Changes were documented in the changelog (pending section).
• If necessary, changes were made to the documentation (eg new pipeline).

[github-actions]

Docs preview URL

https://edsnlp-hfdatasets.vercel.app/

[github-actions]

Coverage Report

Name	Stmts	Miss	∆ Miss	Cover
edsnlp/data/converters.py Was already missing at line 427 elif key == "XPOS": - word.tag_ = value elif key == "FEATS": Was already missing at line 835 if self.keep_raw_attribute_values: - return value try: Was already missing at line 897 if not attr: - continue if "=" in attr: Was already missing at line 928 if span is None: - continue for k, v in attrs.items(): Was already missing at line 998 if isinstance(value, (bool, int, float)): - return repr(value) s = str(value) New missing coverage at line 1307 ! if current_type is not None: - entities.append((start_idx, i, current_type)) start_idx = i New missing coverage at line 1401 ! if start < 0 or start >= len(tags): - continue tags[start] = f"B-{label}" Was already missing at line 1445 if isinstance(converter, type): - return converter(**kwargs), {} return converter, validate_kwargs(converter, kwargs)	500	8	2	98.40%
edsnlp/data/huggingface_dataset.py New missing coverage at line 259 ! if isinstance(item, DatasetEndSentinel): - continue else:	99	1	1	98.99%
TOTAL	12793	252	3	98.03%

Files without new missing coverage

Name	Stmts	Miss	∆ Miss	Cover
edsnlp/utils/torch.py Was already missing at line 102 def load_pruned_obj(obj, _): - return obj Was already missing at line 118 def save_align_devices_hook(pickler, obj): - pickler.save_reduce(load_align_devices_hook, (obj.__dict__,), obj=obj) Was already missing at lines 121-128 def load_align_devices_hook(state): - state["execution_device"] = MAP_LOCATION ... - AlignDevicesHook = None Was already missing at line 143 if torch.Tensor in copyreg.dispatch_table: - old_dispatch[torch.Tensor] = copyreg.dispatch_table[torch.Tensor] copyreg.pickle(torch.Tensor, reduce_empty)	81	9	0	88.89%
edsnlp/utils/span_getters.py Was already missing at lines 73-75 if span_getter is None: - yield doclike[:], None - return if callable(span_getter): Was already missing at lines 76-78 if callable(span_getter): - yield from span_getter(doclike) - return for key, span_filter in span_getter.items(): Was already missing at lines 99-102 else: - for span, group in candidates: - if span.label_ in span_filter: - yield span, group Was already missing at line 107 if callable(span_setter): - span_setter(doc, matches) else: Was already missing at line 138 if callable(value): - return value if isinstance(value, str): Was already missing at line 187 elif isinstance(v, str): - new_value[k] = [v] elif isinstance(v, list) and all(isinstance(i, str) for i in v):	241	10	0	95.85%
edsnlp/utils/resources.py Was already missing at line 33 if not verbs: - return conjugated_verbs	25	1	0	96.00%
edsnlp/utils/numbers.py Was already missing at line 34 else: - string = s string = string.lower().strip() Was already missing at lines 38-41 return int(string) - except ValueError: - parsed = DIGITS_MAPPINGS.get(string, None) - return parsed	16	4	0	75.00%
edsnlp/utils/fuzzy_alignment.py Was already missing at line 70 if len(other.begins) == 0: - return self begins = self.unapply(other.begins, side="left")	191	1	0	99.48%
edsnlp/utils/filter.py Was already missing at line 206 if isinstance(label, int): - return [span for span in spans if span.label == label] else:	74	1	0	98.65%
edsnlp/utils/file_system.py Was already missing at line 39 if isinstance(filesystem, str): - filesystem = fsspec.filesystem(filesystem) Was already missing at line 50 if not isinstance(inferred_protocols, (list, tuple, set)): - inferred_protocols = [inferred_fs.protocol] if not isinstance(filesystem_protocols, (list, tuple, set)): Was already missing at line 52 if not isinstance(filesystem_protocols, (list, tuple, set)): - filesystem_protocols = [filesystem.protocol] assert set(filesystem_protocols) & set(inferred_protocols), (	31	3	0	90.32%
edsnlp/tune.py Was already missing at line 165 ) - except RuntimeError as e: if "zero total variance" in str(e): # pragma: no cover Was already missing at line 680 else: - n_trials = compute_n_trials( gpu_hours, compute_time_per_trial(study, ema=True)	284	2	0	99.30%
edsnlp/training/trainer.py Was already missing at line 57 if result is None: - result = {} if isinstance(x, dict): Was already missing at lines 365-371 if self.sub_batch_size and self.sub_batch_size[1] == "splits": - data = data.batchify( ... - data = data.map(lambda b: [nlp.collate(sb, device=device) for sb in b]) elif self.sub_batch_size: Was already missing at lines 897-904 raise - except Exception: ... - raise Was already missing at lines 931-933 ) > grad_max_dev * math.sqrt(grad_var): - spike = True - spikes += 1 else: Was already missing at line 940 if spike and grad_dev_policy == "clip_mean": - torch.nn.utils.clip_grad_norm_( grad_params, grad_mean, norm_type=2 Was already missing at line 944 elif spike and grad_dev_policy == "clip_threshold": - torch.nn.utils.clip_grad_norm_( grad_params,	338	12	0	96.45%
edsnlp/training/loggers.py Was already missing at line 51 if self._file is not None: - return os.makedirs(self.logging_dir, exist_ok=True) Was already missing at line 88 if col not in values and col != "step": - row.append("") else: Was already missing at line 211 def tracker(self): - return self.printer Was already missing at line 278 ) - logging_dir = env_logging_dir assert logging_dir is not None, (	152	4	0	97.37%
edsnlp/reducers.py Was already missing at line 115 if not hasattr(module, "__file__"): - return True if module.__file__ is None: Was already missing at line 117 if module.__file__ is None: - return False # Hack to avoid copying the full module dict	68	2	0	97.06%
edsnlp/processing/spark.py Was already missing at line 50 getActiveSession = SparkSession.getActiveSession - except AttributeError:	47	1	0	97.87%
edsnlp/processing/multiprocessing.py Was already missing at lines 222-230 return re.findall(r"/[^\s]+\.so[^\s]*", f.read()) - except Exception: ... - return [] Was already missing at lines 233-235 loaded = loaded_libs() - except Exception: - return False return any(any(k in os.path.basename(p).lower() for k in libs) for p in loaded) Was already missing at line 254 ) - method = "spawn" Was already missing at lines 258-264 if has_hdfs and method == "fork": - safe = "forkserver" if "forkserver" in methods else "spawn" ... - method = safe Was already missing at lines 453-455 pass - except StopSignal: - pass for name, queue in self.consumer_queues(stage): Was already missing at lines 668-670 if isinstance(docs, StreamSentinel): - self.active_batches[stage].append([None, None, None, docs]) - continue batch_id = str(hash(tuple(id(x) for x in docs)))[-8:] + "-" + self.uid Was already missing at lines 1193-1199 if out[0].kind == requires_sentinel: - missing_sentinels -= 1 ... - missing_sentinels = len(self.cpu_worker_names) continue	663	22	0	96.68%
edsnlp/processing/deprecated_pipe.py Was already missing at lines 207-209 def converter(doc): - res = results_extractor(doc) - return ( [{"note_id": doc._.note_id, **row} for row in res]	53	2	0	96.23%
edsnlp/pipes/trainable/span_linker/span_linker.py Was already missing at lines 402-404 if self.reference_mode == "synonym": - embeds = embeds.to(new_lin.weight) - new_lin.weight.data = embeds else:	175	2	0	98.86%
edsnlp/pipes/trainable/span_classifier/span_classifier.py Was already missing at line 380 if not all(keep_bindings): - logger.warning( "Some attributes have no labels or values and have been removed:"	171	1	0	99.42%
edsnlp/pipes/trainable/ner_crf/ner_crf.py Was already missing at line 302 if self.labels is not None and not self.infer_span_setter: - return Was already missing at lines 310-312 if callable(self.target_span_getter): - for span in get_spans(doc, self.target_span_getter): - inferred_labels.add(span.label_) else: Was already missing at line 447 ) - self._has_warned = True	177	4	0	97.74%
edsnlp/pipes/trainable/layers/crf.py Was already missing at line 21 # out: 2 * N * O - return (log_A.unsqueeze(-1) + log_B.unsqueeze(-3)).logsumexp(-2) Was already missing at line 29 # out: 2 * N * O - return (log_A.unsqueeze(-1) + log_B.unsqueeze(-3)).max(-2) Was already missing at line 98 if learnable_transitions: - self.transitions = torch.nn.Parameter( torch.zeros_like(forbidden_transitions, dtype=torch.float) Was already missing at line 108 if learnable_transitions and with_start_end_transitions: - self.start_transitions = torch.nn.Parameter( torch.zeros(num_tags, dtype=torch.float) Was already missing at line 117 if learnable_transitions and with_start_end_transitions: - self.end_transitions = torch.nn.Parameter( torch.zeros(num_tags, dtype=torch.float)	140	5	0	96.43%
edsnlp/pipes/trainable/embeddings/transformer/transformer.py Was already missing at line 166 if quantization is not None: - kwargs["quantization_config"] = quantization Was already missing at line 189 if self.cls_token_id is None: - [self.cls_token_id] = self.tokenizer.convert_tokens_to_ids( [self.tokenizer.special_tokens_map["bos_token"]] Was already missing at line 193 if self.sep_token_id is None: - [self.sep_token_id] = self.tokenizer.convert_tokens_to_ids( [self.tokenizer.special_tokens_map["eos_token"]]	166	3	0	98.19%
edsnlp/pipes/qualifiers/reported_speech/reported_speech.py Was already missing at lines 24-28 return "REPORTED" - elif token._.rspeech is False: - return "DIRECT" - else: - return None	100	3	0	97.00%
edsnlp/pipes/qualifiers/negation/negation.py Was already missing at line 28 else: - return None	101	1	0	99.01%
edsnlp/pipes/qualifiers/hypothesis/hypothesis.py Was already missing at line 27 else: - return None	98	1	0	98.98%
edsnlp/pipes/qualifiers/history/history.py Was already missing at lines 26-32 def history_getter(token: Union[Token, Span]) -> Optional[str]: - if token._.history is True: - return "ATCD" - elif token._.history is False: - return "CURRENT" - else: - return None Was already missing at lines 353-359 ) - except ValueError: ... - note_datetime = None Was already missing at lines 368-374 ) - except ValueError: ... - birth_datetime = None Was already missing at lines 440-443 ) - except ValueError as e: - absolute_date = None - logger.warning( "In doc {}, the following date {} raises this error: {}. "	180	14	0	92.22%
edsnlp/pipes/qualifiers/family/family.py Was already missing at line 27 else: - return None	83	1	0	98.80%
edsnlp/pipes/ner/tnm/model.py Was already missing at line 143 def __str__(self): - return self.norm() Was already missing at line 167 ) - exclude_unset = skip_defaults	109	2	0	98.17%
edsnlp/pipes/ner/scores/sofa/sofa.py Was already missing at line 32 if not assigned: - continue if assigned.get("method_max") is not None: Was already missing at line 40 else: - method = "Non précisée"	25	2	0	92.00%
edsnlp/pipes/ner/scores/elston_ellis/patterns.py Was already missing at line 26 if x <= 5: - return 1 Was already missing at lines 32-36 else: - return 3 - - except ValueError: - return None	21	4	0	80.95%
edsnlp/pipes/ner/scores/charlson/patterns.py Was already missing at lines 21-23 return int(extracted_score) - except ValueError: - return None	13	2	0	84.62%
edsnlp/pipes/ner/disorders/solid_tumor/solid_tumor.py Was already missing at lines 131-137 for span in spans: - span.label_ = "solid_tumor" ... - yield span	38	6	0	84.21%
edsnlp/pipes/ner/disorders/peripheral_vascular_disease/peripheral_vascular_disease.py Was already missing at line 108 if "peripheral" not in span._.assigned.keys(): - continue	16	1	0	93.75%
edsnlp/pipes/ner/disorders/diabetes/diabetes.py Was already missing at line 131 # Mostly FP - continue Was already missing at line 134 elif self.has_far_complications(span): - span._.status = 2 Was already missing at line 145 if next(iter(self.complication_matcher(context)), None) is not None: - return True return False	30	3	0	90.00%
edsnlp/pipes/ner/disorders/connective_tissue_disease/connective_tissue_disease.py Was already missing at line 104 # Huge change of FP / Title section - continue	15	1	0	93.33%
edsnlp/pipes/ner/disorders/ckd/ckd.py Was already missing at lines 121-124 dfg_value = float(dfg_span.text.replace(",", ".").strip()) - except ValueError: - logger.trace(f"DFG value couldn't be extracted from {dfg_span.text}") - return False	30	3	0	90.00%
edsnlp/pipes/ner/disorders/cerebrovascular_accident/cerebrovascular_accident.py Was already missing at lines 112-114 if span._.source == "ischemia": - if "brain" not in span._.assigned.keys(): - continue	18	2	0	88.89%
edsnlp/pipes/ner/adicap/models.py Was already missing at line 15 def norm(self) -> str: - return self.code Was already missing at line 18 def __str__(self): - return self.norm()	14	2	0	85.71%
edsnlp/pipes/misc/split/split.py Was already missing at lines 186-188 if max_length <= 0 and self.regex is None: - yield doc - return	74	2	0	97.30%
edsnlp/pipes/misc/sections/sections.py Was already missing at line 126 if sections is None: - sections = patterns.sections sections = dict(sections)	46	1	0	97.83%
edsnlp/pipes/misc/quantities/quantities.py Was already missing at lines 147-149 def __getitem__(self, item: int): - assert isinstance(item, int) - return [self][item] Was already missing at lines 160-163 def __eq__(self, other: Any): - if isinstance(other, SimpleQuantity): - return self.convert_to(other.unit) == other.value - return False Was already missing at line 166 if other.unit == self.unit: - return SimpleQuantity(self.value + other.value, self.unit, self.registry) return SimpleQuantity( Was already missing at line 193 return self.convert_to(other_unit) - except KeyError: raise AttributeError(f"Unit {other_unit} not found") Was already missing at line 198 def verify(cls, ent): - return True Was already missing at line 264 def __lt__(self, other: Union[SimpleQuantity, "RangeQuantity"]): - return max(self.convert_to(other.unit)) < min((part.value for part in other)) Was already missing at line 275 return self.convert_to(other.unit) == other.value - return False Was already missing at line 289 def verify(cls, ent): - return True Was already missing at line 888 if snippet.end != last and doclike.doc[last: snippet.end].text.strip() == "": - pseudo.append("w") pseudo = "".join(pseudo) Was already missing at line 1069 if start_line is None: - continue Was already missing at lines 1100-1102 unit_norm = self.unit_followers[unit_before.label_] - except (KeyError, AttributeError, IndexError): - pass Was already missing at line 1145 ): - ent = doc[unit_text.start: number.end] else: Was already missing at lines 1152-1154 dims = self.unit_registry.parse_unit(unit_norm)[0] - except KeyError: - continue Was already missing at lines 1260-1262 last._.set(last.label_, new_value) - except (AttributeError, TypeError): - merged.append(ent) else:	442	20	0	95.48%
edsnlp/pipes/misc/dates/models.py Was already missing at line 157 else: - d["month"] = note_datetime.month if self.day is None: Was already missing at lines 161-167 else: - if self.year is None: ... - d["day"] = default_day Was already missing at lines 175-177 return dt - except ValueError: - return None Was already missing at line 193 else: - return None Was already missing at line 209 if self.second: - norm += f"{self.second:02}s"	203	11	0	94.58%
edsnlp/pipes/misc/dates/dates.py Was already missing at line 249 if isinstance(absolute, str): - absolute = [absolute] if isinstance(relative, str): Was already missing at line 251 if isinstance(relative, str): - relative = [relative] if isinstance(duration, str): Was already missing at line 253 if isinstance(duration, str): - relative = [duration] if isinstance(false_positive, str): Was already missing at lines 357-366 if self.merge_mode == "align": - alignments = align_spans(matches, spans, sort_by_overlap=True) ... - matches.append(span) Was already missing at lines 462-464 if v1.mode == Mode.DURATION: - m1 = Bound.FROM if v2.bound == Bound.UNTIL else Bound.UNTIL - m2 = v2.mode or Bound.FROM elif v2.mode == Mode.DURATION:	153	14	0	90.85%
edsnlp/pipes/misc/consultation_dates/consultation_dates.py Was already missing at line 131 else: - self.date_matcher = None Was already missing at line 134 if not consultation_mention: - consultation_mention = [] elif consultation_mention is True:	48	2	0	95.83%
edsnlp/pipes/llm/llm_span_qualifier/llm_span_qualifier.py Was already missing at line 579 if isinstance(formatted_context, Doc): - context_text = formatted_context.text else: Was already missing at line 657 if start == -1 or end <= 0 or end <= start: - return None try: Was already missing at line 750 if next_yield >= len(doc_states): - return for state in doc_states[next_yield:]:	249	3	0	98.80%
edsnlp/pipes/llm/llm_markup_extractor/llm_markup_extractor.py Was already missing at line 309 if seed is not None: - api_kwargs["seed"] = seed self.retriever = None Was already missing at line 355 if span is None: - continue spans.append(span) Was already missing at lines 467-469 if not contexts: - remaining_ctx_counts[doc_idx] = 0 - buffer[doc_idx] = doc else: Was already missing at line 490 if result is None: - pass else:	157	5	0	96.82%
edsnlp/pipes/core/normalizer/__init__.py Was already missing at line 7 def excluded_or_space_getter(t): - return t.is_space or t.tag_ == "EXCLUDED"	5	1	0	80.00%
edsnlp/pipes/core/endlines/endlines.py Was already missing at lines 160-164 if end_lines_model is None: - path = build_path(__file__, "base_model.pkl") - - with open(path, "rb") as inp: - self.model = pickle.load(inp) elif isinstance(end_lines_model, str): Was already missing at lines 167-169 self.model = pickle.load(inp) - elif isinstance(end_lines_model, EndLinesModel): - self.model = end_lines_model else: Was already missing at line 200 ): - return "ENUMERATION" Was already missing at line 287 if np.isnan(sigma): - sigma = 1	89	7	0	92.13%
edsnlp/pipes/core/contextual_matcher/contextual_matcher.py Was already missing at lines 241-243 ): - to_keep = False - break	130	2	0	98.46%
edsnlp/patch_spacy.py Was already missing at lines 67-69 # if module is reloaded. - existing_func = registry.factories.get(internal_name) - if not util.is_same_func(factory_func, existing_func): raise ValueError(	31	2	0	93.55%
edsnlp/package.py Was already missing at lines 474-476 version = version or pyproject["project"]["version"] - except (KeyError, TypeError): - version = "0.1.0" name = name or pyproject["project"]["name"] Was already missing at line 480 else: - main_package = None model_package = snake_case(name.lower())	214	3	0	98.60%
edsnlp/metrics/span_attribute.py Was already missing at lines 67-69 ) - assert attributes is None - attributes = kwargs.pop("qualifiers") if attributes is None:	74	2	0	97.30%
edsnlp/matchers/simstring.py Was already missing at line 280 if custom: - attr = attr[1:].lower() Was already missing at line 295 if custom: - token_text = getattr(token._, attr) else:	146	2	0	98.63%
edsnlp/language.py Was already missing at line 103 if last != begin: - logger.warning( "Missed some characters during"	52	1	0	98.08%
edsnlp/data/standoff.py Was already missing at line 38 def __init__(self, ann_file, line): - super().__init__(f"File {ann_file}, unrecognized Brat line {line}") Was already missing at line 192 ) - except Exception: raise Exception(	186	2	0	98.92%
edsnlp/data/polars.py Was already missing at line 36 if hasattr(data, "collect"): - data = data.collect() assert isinstance(data, pl.DataFrame)	55	1	0	98.18%
edsnlp/data/json.py Was already missing at line 81 return records - except Exception as e: raise Exception(f"Cannot read {file}: {e}")	112	1	0	99.11%
edsnlp/data/conll.py Was already missing at lines 81-83 ) - except StopIteration: - cols = DEFAULT_COLUMNS warnings.warn( Was already missing at lines 92-96 if not line: - if doc["words"]: - yield doc - doc = {"words": []} - continue if line.startswith("#"):	76	6	0	92.11%
edsnlp/core/torch_component.py Was already missing at line 407 if hasattr(self, "compiled"): - res = self.compiled(batch) else: Was already missing at line 453 """ - return self.preprocess(doc)	190	2	0	98.95%
edsnlp/core/stream.py Was already missing at line 155 else: - yield res return Was already missing at lines 203-205 if isinstance(batch, StreamSentinel): - yield batch - continue results = [] Was already missing at lines 1030-1032 elif op.batch_fn is None: - batch_size = op.size - batch_fn = batchify else:	383	5	0	98.69%
edsnlp/core/registries.py Was already missing at line 135 if isinstance(obj, DraftPipe): - return obj elif isinstance(obj, dict): Was already missing at line 140 if result is not None: - return result elif isinstance(obj, (tuple, list, set)): Was already missing at line 145 if result is not None: - return result return None	196	3	0	98.47%
edsnlp/core/pipeline.py Was already missing at line 605 if name in exclude: - continue if name not in components: Was already missing at lines 713-716 """ - res = Stream.ensure_stream(docs) - res = res.map(functools.partial(self.preprocess, supervision=supervision)) - return res	454	4	0	99.12%
edsnlp/connectors/omop.py Was already missing at line 69 if not isinstance(row.ents, list): - continue Was already missing at line 87 else: - doc.spans[span.label_].append(span) Was already missing at line 127 if df.note_id.isna().any(): - df["note_id"] = range(len(df)) Was already missing at line 171 if i > 0: - df.term_modifiers += ";" df.term_modifiers += ext + "=" + df[ext].astype(str)	84	4	0	95.24%

281 files skipped due to complete coverage.

Coverage success: total of 98.03% is above 98.02% 🎉

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
1 Accepted issue

Measures
0 Security Hotspots
0.0% Coverage on New Code
3.7% Duplication on New Code

See analysis details on SonarQube Cloud