Getting a uft-8 Error

[MilesCranmer]

Discussed in #944

^{Originally posted by AaronRoseDA May 28, 2025}

@AaronRoseDA

Hi Miles,

I'm really struggling to deal with a particular utf-8 encoding issue. I have no problems with any other dataset so I think it might be the data frame but I can't find anything that stands out in the data:

C:\Users\arose\.conda\envs\test-env\Lib\site-packages\pysr\sr.py:2774: UserWarning: Note: it looks like you are running in Jupyter. The progress bar will be turned off.
  warnings.warn(
[ Info: Started!
[ Info: Final population:
[ Info: Results saved to:
Elapsed time: 64.96 seconds
Error in callback _flush_stdio (for post_execute), with arguments args (),kwargs {}:
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
File ~\.julia\packages\PythonCall\WMWY0\src\JlWrap\any.jl:262, in __call__(self, *args, **kwargs)
    260     return ValueBase.__dir__(self) + self._jl_callmethod($(pyjl_methodnum(pyjlany_dir)))
    261 def __call__(self, *args, **kwargs):
--> 262     return self._jl_callmethod($(pyjl_methodnum(pyjlany_call)), args, kwargs)
    263 def __bool__(self):
    264     return True

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe2 in position 4095: unexpected end of data

I've narrowed it down to a few things but I can't put my finger on the exact reason. It could be something wrong with my data frame but I can't find anything that stands out.

I've tried pairing down the model as much as possible and here's my model setup:

# Define input (X) and target (y)
X = data[feature_names].values
# y = data["entropy"].values
y = data["entropy_kde_XY"].values
# y = data["entropy_theoretical_XY"].values


model = PySRRegressor(
    model_selection="best",
    run_id=f'Bi-Skew-Normal - {datetime.datetime.now().strftime("%Y-%m-%d-%H-%M-%S.%f")[:-4]}',
    niterations=10000000,
    binary_operators=["+", "-", "*", "/"],
    unary_operators=["log", "sqrt", "square","exp","cube","cbrt"
                     # , "sin", "cos", "tan"
                     ],
    nested_constraints = {
    "log": {"log": 0},
    "sqrt": {"sqrt": 0},
    "square": {"square": 0},
    "exp": {"exp": 0},
    "cube": {"cube": 0},
    "cbrt": {"cbrt": 0}
    },
    populations=48,
    ncycles_per_iteration=5000,
    # batching=True,
    # weight_optimize=.005,
    # turbo=True,
    
    maxsize=26,
    warm_start=True,
    temp_equation_file=False,
    output_directory=r"C:/Users/arose/OneDrive/Desktop/output/BivariateSkewNormal",
    verbosity=1,
    timeout_in_seconds=60,
    complexity_of_constants=1,
    # logger_spec=logger_spec
    # early_stop_condition="f(loss, complexity) = (loss <= 5.526e-15) && (complexity < 6)",print_precision=24
)

# model.weight_optimize = .005
# model.maxsize = 26
# model.ncycles_per_iteration = 5000
# model.timeout_in_seconds = 120
# model.print_precision = 16
# model.model_selection = 'accuracy'
# model.unary_operators = ["log", "square"]
# model.populations = 96



st = time.time()
model.fit(X, y, variable_names=feature_names)
runtime = round(time.time() - st, 2)
print(f"Elapsed time: {runtime} seconds")

Any expertise you can provide would be immensely helpful. I'm going to take a deeper dive into my data and see if I can find anything that stands out in my data.

[MilesCranmer]

@AaronRoseDA
Thanks for the report. Is it possible at all to share the dataset, or maybe an example dataset that generates the same bug?

I guess I'm wondering if there is some variable name with a special character that could be causing it.

[MilesCranmer]

Also does it help if you disable the print_precision parameter setting?

[AaronRoseDA]

Hey @MilesCranmer,

I played around trying to reproduce the error and I can confirm it's not the data frame and probably not the print_precision parameter.

I tested a few different cases today and I believe the issue lies somewhere with how verbosity and logger_spec interact.

The easiest way to get the error was running the fit call below three times while warm_start = True. Usually the error happened between runs but that's not always the case:

model.fit(X, y, variable_names=feature_names)

In case 1, verbosity = 1 and logger_spec = None the code executes as expected. The three runs work with no issues.

In case 2, verbosity = 0 and logger_spec = logger_spec the code also runs great. I can monitor the progress on Tensorboard with no issues.

In case 3, verbosity = 1 and logger_spec = logger_spec and I don't launch TensorBoard the model will run twice or start the third run and produce the utf-8 error. When I get this error my whole python kernel gets screwed up and even running something like a = "hello" in the console causes the same error to repeat and sometimes the python kernel dies:

Error in callback _flush_stdio (for post_execute), with arguments args (),kwargs {}:
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
File ~\.julia\packages\PythonCall\L4cjh\src\JlWrap\any.jl:262, in __call__(self, *args, **kwargs)
    260     return ValueBase.__dir__(self) + self._jl_callmethod($(pyjl_methodnum(pyjlany_dir)))
    261 def __call__(self, *args, **kwargs):
--> 262     return self._jl_callmethod($(pyjl_methodnum(pyjlany_call)), args, kwargs)
    263 def __bool__(self):
    264     return True

UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 4094-4095: unexpected end of data

a = "test"
Error in callback _flush_stdio (for post_execute), with arguments args (),kwargs {}:
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
File ~\.julia\packages\PythonCall\L4cjh\src\JlWrap\any.jl:262, in __call__(self, *args, **kwargs)
    260     return ValueBase.__dir__(self) + self._jl_callmethod($(pyjl_methodnum(pyjlany_dir)))
    261 def __call__(self, *args, **kwargs):
--> 262     return self._jl_callmethod($(pyjl_methodnum(pyjlany_call)), args, kwargs)
    263 def __bool__(self):
    264     return True

UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 4094-4095: unexpected end of data

In case 4, where verbosity = 1 and logger_spec = logger_spec and I am watching progress in TensorBoard I don't get the same utf-8 error, however, the model stalls somewhere during or after the second run. My CPU usage goes to 0% and the code just spins at this point:

Press 'q' and then <enter> to stop execution early.

Expressions evaluated per second: 1.020e+05
Progress: 297 / 480000000 total iterations (0.000%)
═══════════════
Elapsed time: 97.07 seconds
[ Info: Final population:
[ Info: Results saved to:
C:\Users\arose\.conda\envs\sr-env\Lib\site-packages\pysr\sr.py:2811: UserWarning: Note: it looks like you are running in Jupyter. The progress bar will be turned off.
  warnings.warn(

Here's my full model def incase anything stands out:

logger_spec = TensorBoardLoggerSpec(
    # log_dir=r"C:/Users/arose/OneDrive/Desktop/output/BivariateSkewNormal_testCase/logger",
    log_dir=r"C:/Users/arose/output/BivariateSkewNormal_testCase/logger",

    log_interval=10
)

model = PySRRegressor(
    model_selection="best",
    run_id=f'Bi-Skew-Normal-{datetime.datetime.now().strftime("%Y-%m-%d-%H-%M-%S.%f")[:-4]}',
    niterations=10000000,
    binary_operators=["+", "-", "*", "/"],
    unary_operators=["log", "sqrt", "square","exp","cube","cbrt"],
    nested_constraints = {
      "log": {"log": 0},
      "sqrt": {"sqrt": 0},
      "square": {"square": 0},
      "exp": {"exp": 0},
      "cube": {"cube": 0},
      "cbrt": {"cbrt": 0}
    },
    populations=48,
    ncycles_per_iteration=5000,    
    maxsize=26,
    warm_start=True,
    temp_equation_file=False,
    output_directory=r"C:/Users/arose/output/BivariateSkewNormal_testCase",
    timeout_in_seconds=90,
    complexity_of_constants=1,

    # 1:
    #verbosity=1,
    #logger_spec=None
    # 2:
    #verbosity=0,
    #logger_spec=logger_spec
    # 3 & 4 - Error Case:
    verbosity=1,
    logger_spec=logger_spec
    #
)

In conclusion I'm not sure whether warm_start has anything to do with the issue OR if I ran the model in a single trial for long enough I would get the same issue.

I have noticed that the output from verbosity lags far behind the updates observed on TensorBoard though which I don't know if thats by design or a limitation.

My environment is running python 3.12.7, pySR 1.5.8, and Tensorboard 2.19.0.

Hopefully, you can reproduce the same error and it's not just something on my end! In the meantime I'll turn verbosity off and hopefully that solves the issue.

Thanks!

[MilesCranmer]

Thanks. Very weird indeed. But good to know it is related to the tensorboard logging. Do you know if any other things are correlated with the error? e.g., if you remove all the parameters except for verbosity and logger_spec, does it still hit the error? (i.e., a "minimal reproducible example": https://stackoverflow.com/help/minimal-reproducible-example)

[DorofeevKirillDev]

@MilesCranmer Also struggle with this utf-8 error in some cases. Tried to alter an example and got this error quite consistently on some functions. Here is an example (running it in colab if it is relevant):

import pysr
import sympy
import numpy as np
from matplotlib import pyplot as plt
from pysr import PySRRegressor
from sklearn.model_selection import train_test_split

# Dataset
np.random.seed(30)
X = 0.01 + 1*np.random.random((200, 2))
y = 1.49 * ((X[:, 0] / X[:, 1]) ** 0.5) + 19*(X[:, 0]**0.25 *  X[:, 1]) + ((1 - X[:, 1] / X[:, 0]) ** 2)

default_pysr_params = dict(
    populations=50,
    model_selection="best",
)

# Learn equations
model = PySRRegressor(
    niterations=50,
    ncycles_per_iteration=1000,
    binary_operators=["+", "-", "*", "/"],
    unary_operators=["square", "sqrt"],
    **default_pysr_params,
)

model.fit(X, y)

[MilesCranmer]

@AaronRoseDA @DorofeevKirillDev are you both running in Jupyter? Does this happen outside of Jupyter too?

I wonder if it has something to do with TensorBoard automatically trying to talk to Jupyter directly, but is expecting to do so through Python rather than Julia(??).

Also, what happens if you do

import os
os.environ["PYTHON_JULIACALL_AUTOLOAD_IPYTHON_EXTENSION"] = "no"

before importing PySR for the first time?

[DorofeevKirillDev]

@MilesCranmer it looks like adding these two lines solved the problem, I've tried several functions with different parameters and didn't notice any issues (using Jupyter). Also tried running the original code in IPython and the problem was not present. I guess the problem is in Jupyter indeed.

[GongJr0]

Hi @MilesCranmer, @AaronRoseDA, and @DorofeevKirillDev,

Wanted to chip-in as I've been experiencing this for a few months now. I'm not very well informed about the backend of PySRRegressor objects, please excuse my reasoning if I seem to be going off the track.

I'm seeing the exact same traceback shared in the thread (accurate to the byte position of the unexpected EOF), and I was never able to reproduce this outside of jupyter notebooks. Unfortunately setting os.environ["PYTHON_JULIACALL_AUTOLOAD_IPYTHON_EXTENSION"] = "no" does not reliably solve the issue on my end. Using the same parameters, the error can either happen in the first fit, or the following fits.

Specific to ipython evironments, I don't believe TensorBoard has a part in this error as I've been consistently reproducing this error with logger_spec=None and verbosity=1. The error does not affect functionality in my case, but breaks the cell outputs of the whole jupyter notebook. Once the fit is executed, all cells will reproduce this error, and some specific functionality such as automatically rendering formatted DataFrames in the output will not work. However, the ipython kernel will not be disrupted and I can continue to use the notebook without having to restart the interpreter.

I think running the fit through subprocess.Popen(..., stdout=PIPE) and pipingstdout to the jupyter cell might be a good test case to pinpoint the problem. I would expect this to work with no errors as the output will be produced outside of ipython. If this is not the case consistently, I think even some character or formatting in the output string could be the culprit.

Some environment info:

OS: Win11 (24H2)
Python version: 3.12.11
PySR version: 1.5.9
juliacall version: 0.9.26
juliapkg version: 0.1.17

[MilesCranmer]

@GongJr0 just to confirm, you are saying that with both that environment variable set to disable the IPython hook, and also with logger=None, you still see the UTF-8 error?

[GongJr0]

Hi @MilesCranmer, yes. In my case, I can fix the error when I set PYTHON_JULIACALL_AUTOLOAD_IPYTHON_EXTENSION as a system level environment variable instead of calling os at the process level. Maybe some scope or access problems are happening on my end.

[MilesCranmer]

Weird, that shouldn’t have any effect if you set it with os.environ before importing pysr. pysr is reading the env variables using os.environ after all

[MilesCranmer]

Maybe you already imported pysr in the same Jupyter session

[GongJr0]

I tried reproducing it with a fresh env and kernel and couldn't do so, it seems to be a problem on my end. Thanks for the advices! setting it through os is just fine.