Repository navigation
Remove the bias column from the credit dataset - #3
Merged
Merged
Conversation
`StrategicClassification` shifts every column of the feature matrix, so the constant-1 column that `CreditDataset` appended was treated as a manipulable feature and moved along with the real ones. With epsilon=10 and seed 10, the intercept feature went from 1 to -2.408 under logistic regression; under the NN it spread over [-1.13, 3.32]. That turns the intercept's contribution to the score into w_b * (1 - epsilon * w_b), which is concave in w_b and flips sign past w_b > 1/epsilon, distorting the performative loss landscape for every optimizer. An intercept is a model parameter, not a feature, so drop the column and let the models carry the intercept themselves: logistic regression keeps a flat 11-entry parameter vector whose last entry is the intercept, and the haiku MLP already has its own bias, which made the column redundant there anyway. On logistic regression with RGD (100 iterations, n=20000), accuracy improves from 52.21% to 61.80%. This also exposed a latent orientation bug in `PerfGDReinforce`: `delta_f_theta` is the Jacobian df/dtheta of shape (dim(f), dim(theta)) and has to be transposed before contracting over f. The missing transpose was invisible while dim(f) == dim(theta), and it is numerically a no-op wherever that Jacobian is diagonal (the pricing example gives bit-identical results at d=1 and d=3), but it is required now that the credit example has 10 features and 11 parameters. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dm-haiku has to move to 0.0.17 along with it, because 0.0.14 does not even import against jax 0.10.2 (`jax.lib.xla_extension` is gone). The only code change needed is in the credit example: `jnp.clip()` no longer accepts the `a_min`/`a_max` keywords. jax is capped below 0.11 because that version is not supported yet. Since 0.11 requires Python 3.12, an uncapped requirement resolves to two different jax versions across the supported Python range: 0.10.2 on 3.11 and 0.11.1 on 3.12. All examples produce the same numbers as they did on jax 0.6.1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`delta_f_theta` stacked the parameter iterates with `jnp.array()`, which only works when the parameters are a flat array. With the haiku MLP of the credit example, whose parameters are a nested dict, this failed outright. Ravel each iterate with `ravel_pytree` before the finite-difference estimate, and unravel the resulting flat gradient back into the structure of the parameters, so that it can be added to the first gradient term. For flat parameters this is a no-op: the credit, pricing, linear, nonlinear, mixture and cosine examples all give bit-identical results. Note that the estimate is heavily underdetermined for the MLP, since it fits a Jacobian of shape (10, 1201) from H=4 finite differences. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tmke8
commented
Aug 26, 2026
Comment on lines
+93
to
+95
| mlp = hk.Sequential( | ||
| [hk.Linear(100, with_bias=True), jax.nn.relu, hk.Linear(1)] | ||
| ) |
Member
Author
There was a problem hiding this comment.
with_bias actually defaults to True, so with the NN we had two biases
tmke8
commented
Aug 26, 2026
Comment on lines
+107
to
+110
| # The last entry of `params` is the bias term; it is a parameter | ||
| # rather than a feature, so the distribution map cannot shift it. | ||
| self.h = lambda params, x: x @ params[:-1] + params[-1] | ||
| return initialize_params((self.dataset.num_features + 1,), self.seed) |
Member
Author
There was a problem hiding this comment.
this is the important change to the logistic regression code
tmke8
commented
Aug 26, 2026
Comment on lines
+167
to
+174
| # Estimating the second part of the performative gradient. | ||
| # The parameters may be an arbitrary pytree (such as the parameter dict of | ||
| # a haiku model), so the iterates are raveled into vectors first. | ||
| flat_params = [ | ||
| ravel_pytree(p)[0] | ||
| for p in self.params_history[self.i - self.H: self.i + 1] | ||
| ] | ||
| delta_theta = (jnp.stack(flat_params[:-1]) - flat_params[-1]).T |
Member
Author
There was a problem hiding this comment.
this fixes an unrelated bug in PerfGDReinforce which had the effect that training a NN didn't work with PerfGDReinforce (only logistic regression did)
There was a problem hiding this comment.
Pull request overview
Removes the credit dataset’s synthetic bias feature and moves intercept handling into models.
Changes:
- Removes bias-column augmentation and updates documentation.
- Adds explicit logistic-regression intercept and pytree support to PerfGDReinforce.
- Upgrades JAX and Haiku dependencies.
Reviewed changes
Copilot reviewed 5 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
uv.lock |
Locks upgraded JAX and Haiku packages. |
requirements.txt |
Updates pinned dependencies. |
pyproject.toml |
Raises JAX and Haiku requirements. |
PerfGDReinforce.py |
Supports pytree parameters and corrected gradient dimensions. |
strategic_classification.py |
Documents intercept handling. |
datasets.py |
Removes bias-feature augmentation. |
examples/credit.py |
Adds model-owned intercepts and dynamic feature dimensions. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Member
Author
|
copilot approves, so I'll merge |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With this change, the bias term for the linear model is not added to the features anymore. This only required a small change to the code of the linear model.
I also updated the jax and haiku dependencies because I noticed that when I was running with newer jax in a different problem, the code in this library was giving errors.