Skip to content

Conversation

@JaroslavTulach
Copy link
Member

@JaroslavTulach JaroslavTulach commented Dec 11, 2025

Pull Request Description

  • Removing snowflake from enso native image executable.
  • Fixing code to deal with dual JVM objects

Important Notes

Execute in mock dual JVM mode via:

sbt:enso> runEngineDistribution 
  --vm.D=polyglot.enso.classLoading=Standard.Snowflake:guest,hosted 
  --env ENSO_SNOWFLAKE_USER=xxx
  --env ENSO_SNOWFLAKE_PASSWORD=yyy
  --env ENSO_SNOWFLAKE_ACCOUNT=zzz
  --env ENSO_SNOWFLAKE_DATABASE=bbb
  --run test/Snowflake_Tests/

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

@JaroslavTulach JaroslavTulach self-assigned this Dec 11, 2025
@JaroslavTulach JaroslavTulach added the CI: No changelog needed Do not require a changelog entry for this PR. label Dec 11, 2025
@JaroslavTulach
Copy link
Member Author

JaroslavTulach commented Dec 11, 2025

There is currently 40 failures:

[FAILED] [Snowflake] Info: [4/5, 15346ms]                             
  | => enso / runEngineDistribution 191s
    - [FAILED] correctly handles Decimal and Float types [5812ms]
        Reason: [1.234568] did not equal [(Decimal.Value 1.234568)]; first difference at index 0  (at test/Snowflake_Tests/src/Snowflake_Spec.enso:260:13-74).

[FAILED] [Snowflake] Edge Cases: [3/4, 24526ms]                       
  | => enso / runEngineDistribution 220s
    - [FAILED] should be able to round-trip a BigInteger column [2185ms]
        Reason: (1.1805916207174113E21): Expected a value of type Standard.Base.Data.Numbers.Integer but got a value [1.1805916207174113E21] of type Standard.Base.Data.Numbers.Float instead (at test/Snowflake_Tests/src/Snowflake_Spec.enso:371:64-84).

[FAILED] [Snowflake] Table.aggregate should correctly select result types: [2/4, 11500ms]
  | => enso / runEngineDistribution 370s
    - [FAILED] widening to decimals on Average [3637ms]
        Reason: 0.666667 did not equal 0.6666666666666666 (at test/Table_Tests/src/Common_Table_Operations/Aggregate_Spec.enso:1221:13-70).


    - [FAILED] widening to decimals on Percentile [2629ms]
        Reason: [2.50000] did not equal [2.5]; first difference at index 0  (at test/Table_Tests/src/Common_Table_Operations/Aggregate_Spec.enso:1244:13-59).


    - [FAILED] should work when stacked one after another [2421ms]
        Reason: [7.000000] did not equal [7]; first difference at index 0  (at test/Table_Tests/src/Common_Table_Operations/Aggregate_Spec.enso:1480:13-64).


    - [FAILED] round should work correctly on Decimals [70ms]
        Reason: An unexpected panic was thrown: (Unsupported_Argument_Types.Error [] 'Cannot convert \'1.3\'(language: Java, type: com.oracle.truffle.polyglot.PolyglotMap) to Java type \'java.math.BigDecimal\': Unsupported target type.')
        at <enso> case_branch(distribution/lib/Standard/Database/0.0.0-dev/src/Internal/Statement_Setter.enso:38:30-73)
        at <enso> Statement_Setter.type.create_fill_hole.fill_hole_jdbc(/distribution/lib/Standard/Database/0.0.0-dev/src/Internal/Statement_Setter.enso:34-58)

[FAILED] [Snowflake] (Derived_Columns_Spec) Table.set with Simple_Expression: [9/10, 18911ms]
  | => enso / runEngineDistribution 1082s
    - [FAILED] arithmetics [3589ms]
        Reason: [10.000000, 20.000000] did not equal [10, 20]; first difference at index 0  (at enso/test/Table_Tests/src/Common_Table_Operations/Derived_Columns_Spec.enso:53:13-170).

[FAILED] [Snowflake] (Upload_Spec) Uploading an in-memory Table: [10/11, 80795ms]
  | => enso / runEngineDistribution 
    - [FAILED] should include the created table in the tables directory [3206ms]
        Reason: An unexpected panic was thrown: java.lang.IllegalArgumentException: Unsupported StorageType for `is_in`: org.enso.table.data.column.storage.type.TextType@896
        at <Unknown Location> related to com.oracle.truffle.host.HostObject.doInvoke
        at <enso> case_branch.java_result(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Table/0.0.0-dev/src/Internal/In_Memory_Column_Implementation.enso:586:25-102)
        at <enso> Java_Problems.with_map_operation_problem_aggregator.Java_Problems.with_map_operation_problem_aggregator(/home/devel/NetBeansProjects/enso/enso/distribution/lib/Standard/Table/0.0.0-dev/src/Internal/Java_Problems.enso:128:9-25)
        at <enso> Java_Problems.with_problem_aggregator(distribution/lib/Standard/Table/0.0.0-dev/src/Internal/Java_Problems.enso:118:14-25)
        at <enso> Java_Problems.with_map_operation_problem_aggregator(distribution/lib/Standard/Table/0.0.0-dev/src/Internal/Java_Problems.enso:126-128)

need to be addressed.

@github-actions github-actions bot added the -libs-API-change-Base Marks a PR that changes the public API of Standard.Base label Dec 11, 2025
@JaroslavTulach
Copy link
Member Author

@JaroslavTulach
Copy link
Member Author

JaroslavTulach commented Dec 11, 2025

14 failures remaining:

- [FAILED] a numeric column (Decimals) supports abs operation [3201ms]
         Reason: [10.500000000000, 0E-12, Nothing, 1.200000000000, 12.340000000000, 123454567890.123456789000, 12345678901890.123456789000] did not equal [(Decimal.Value 10.5), (Decimal.Value 0.0), Nothing, (Decimal.Value 1.2), (Decimal.Value 12.34), (Decimal.Value 123454567890.1234567890), (Decimal.Value 12345678901890.1234567890)]; first difference at index 0  (at /home/runner/work/enso/enso/test/Table_Tests/src/Common_Table_Operations/Numeric_Column_Spec.enso:64:9-98).
Error: An unexpected panic was thrown: org.enso.jvm.interop.impl.OtherJvmException: org.enso.base.enso_cloud.HideableValue is a sealed interface
          at <Unknown Location> related to case_branch
          at <enso> Enso_Secret.as_hideable_value(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Base/2025.3.1-dev/src/Enso_Cloud/Enso_Secret.enso:334-344)
          at <enso> if_then_else(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Database/2025.3.1-dev/src/Internal/JDBC_Connection.enso:332:17-76)
          at <enso> JDBC_Connection.create.properties_as_java_props.case properties_as_java_props(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Database/2025.3.1-dev/src/Internal/JDBC_Connection.enso:331-332)
          at <enso> Array_Like_Helpers.map.Array_Like_Helpers.map(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Base/2025.3.1-dev/src/Internal/Array_Like_Helpers.enso:277:56-77)
          at <enso> Array_Like_Helpers.vector_from_function(Internal)
          at <enso> JDBC_Connection.create(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Database/2025.3.1-dev/src/Internal/JDBC_Connection.enso:336:18-86)
          at <enso> Snowflake_Connection.type.create(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Snowflake/2025.3.1-dev/src/Snowflake_Connection.enso:79:27-96)
          at <enso> Snowflake_Details.connect<arg-1>(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Snowflake/2025.3.1-dev/src/Connection/Snowflake_Details.enso:59:13-106)
          at <enso> Snowflake_Details._enhance_connection_errors(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Snowflake/2025.3.1-dev/src/Connection/Snowflake_Details.enso:102:14-19)
          at <enso> Snowflake_Details.connect(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Snowflake/2025.3.1-dev/src/Connection/Snowflake_Details.enso:58-59)
          at <enso> case_branch(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Database/2025.3.1-dev/src/Connection/Database.enso:35:14-36)
          at <enso> Database.connect(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Database/2025.3.1-dev/src/Connection/Database.enso:33-35)
          at <enso> Auth_Spec.add_specs.Auth_Spec.add_specs.table(/home/runner/work/enso/enso/test/Snowflake_Tests/src/Auth_Spec.enso:77:34-108)
          at <enso> Auth_Spec.with_temp_user<arg-1>(/home/runner/work/enso/enso/test/Snowflake_Tests/src/Auth_Spec.enso:25:10-27)
- [FAILED] division should be aligned with the Enso arithmetic [2965ms]
         Reason: [0.500000, 2.500000, 2.500000, 20.000000] did not equal [0.5, 2.5, 2.5, 20.0]; first difference at index 0  (at /home/runner/work/enso/enso/test/Table_Tests/src/Common_Table_Operations/Column_Operations_Spec.enso:996:13-43).
- [FAILED] should work when stacked one after another
Error: [7.000000] did not equal [7];
 Reason: An unexpected panic was thrown: java.lang.UnsupportedOperationException: executeLargeUpdate not implemented
        at <enso> case_branch<arg-2>(/home/runner/work/enso/enso/built-distribution/enso-engine-2025.3.1-dev-linux-amd64/enso-2025.3.1-dev/lib/Standard/Database/2025.3.1-dev/src/Connection/Connection.enso:425:71-93)

@JaroslavTulach JaroslavTulach changed the title Removing snowflake from native image Running Standard.Snowflake in dual JVM mode Dec 11, 2025
@JaroslavTulach JaroslavTulach changed the title Running Standard.Snowflake in dual JVM mode Run Standard.Snowflake in dual JVM mode Dec 11, 2025
@JaroslavTulach JaroslavTulach added -libs Libraries: New libraries to be implemented -compiler labels Dec 11, 2025
@JaroslavTulach JaroslavTulach moved this to 🔧 Implementation in Issues Board Dec 11, 2025
@jdunkerley jdunkerley added this to the 2026.1 Release milestone Dec 11, 2025
@JaroslavTulach
Copy link
Member Author

With e59ef9c we are down to 10 failures:

sbt:enso> runEngineDistribution --vm.D=polyglot.enso.classLoading=Standard.Snowflake:guest,hosted --env ENSO_SNOWFLAKE_USER=aaa --env ENSO_SNOWFLAKE_PASSWORD=222 --env ENSO_SNOWFLAKE_ACCOUNT=555 --env ENSO_SNOWFLAKE_DATABASE=dddd --run test/Snowflake_Tests/

1179 tests succeeded.
10 tests failed.
37 tests skipped.
11 groups skipped.

Failed tests: 'correctly.handles.Decimal.and.Float.types|should.be.able.to.round-trip.a.BigInteger.column|widening.to.decimals.on.Average|widening.to.decimals.on.Percentile|should.work.when.stacked.one.after.another|should.work.with.mixed.types|division.should.be.aligned.with.the.Enso.arithmetic|should.allow.round.on.a.\(Decimal.Nothing.Nothing\).column.\(to.>0.decimal.places\)|arithmetics|a.numeric.column.\(Decimals\).supports.abs.operation'


    correctly handles Decimal and Float types
        Reason: [1.234568] did not equal [(Decimal.Value 1.234568)]; first difference at index 0  (at /home/devel/NetBeansProjects/enso/enso/test/Snowflake_Tests/src/Snowflake_Spec.enso:260:13-74).


    should be able to round-trip a BigInteger column [2533ms]
            Reason: (1.1805916207174113E21): Expected a value of type Standard.Base.Data.Numbers.Integer but got a value [1.1805916207174113E21] of type Standard.Base.Data.Numbers.Float instead (at /home/devel/NetBeansProjects/enso/enso/test/Snowflake_Tests/src/Snowflake_Spec.enso:371:64-84).

  - [FAILED] widening to decimals on Average [2572ms]
        Reason: 0.666667 did not equal 0.6666666666666666 (at /home/devel/NetBeansProjects/enso/enso/test/Table_Tests/src/Common_Table_Operations/Aggregate_Spec.enso:1221:13-70).


    - [FAILED] widening to decimals on Percentile [2416ms]
        Reason: [2.50000] did not equal [2.5]; first difference at index 0  (at /home/devel/NetBeansProjects/enso/enso/test/Table_Tests/src/Common_Table_Operations/Aggregate_Spec.enso:1244:13-59).
       
    - [FAILED] should work when stacked one after another [2778ms]
        Reason: [7.000000] did not equal [7]; first difference at index 0  (at /home/devel/NetBeansProjects/enso/enso/test/Table_Tests/src/Common_Table_Operations/Aggregate_Spec.enso:1480:13-64).
       
   - [FAILED] should work with mixed types [3392ms]
        Reason: {Table: X->Decimal (precision=38, scale=0), Y->Decimal (precision=38, scale=0)} [50.000000, 5.000000] did not equal [50, 5]; first difference at index 0  (at /home/devel/NetBeansProjects/enso/enso/test/Table_Tests/src/Common_Table_Operations/Column_Operations_Spec.enso:872:17-56).


    - [FAILED] division should be aligned with the Enso arithmetic [1702ms]
        Reason: [0.500000, 2.500000, 2.500000, 20.000000] did not equal [0.5, 2.5, 2.5, 20.0]; first difference at index 0  (at /home/devel/NetBeansProjects/enso/enso/test/Table_Tests/src/Common_Table_Operations/Column_Operations_Spec.enso:996:13-43).


   - [FAILED] should allow round on a (Decimal Nothing Nothing) column (to >0 decimal places) [396ms]
        Reason: [0.5, 0.6, 3.5, 3.6, -0.5, -0.6, -3.5, -3.6] did not equal [0.5, 0.6, 3.5, 3.6, -0.5, -0.6, -3.5, -3.6]; first difference at index 0  (at /home/devel/NetBeansProjects/enso/enso/test/Table_Tests/src/Common_Table_Operations/Column_Operations_Spec.enso:1099:17-90).

 - [FAILED] arithmetics [3517ms]
        Reason: [10.000000, 20.000000] did not equal [10, 20]; first difference at index 0  (at /home/devel/NetBeansProjects/enso/enso/test/Table_Tests/src/Common_Table_Operations/Derived_Columns_Spec.enso:53:13-170).
        
    - [FAILED] a numeric column (Decimals) supports abs operation [2419ms]
        Reason: [10.500000000000, 0E-12, Nothing, 1.200000000000, 12.340000000000, 123454567890.123456789000, 12345678901890.123456789000] did not equal [(Decimal.Value 10.5), (Decimal.Value 0.0), Nothing, (Decimal.Value 1.2), (Decimal.Value 12.34), (Decimal.Value 123454567890.1234567890), (Decimal.Value 12345678901890.1234567890)]; first difference at index 0  (at /home/devel/NetBeansProjects/enso/enso/test/Table_Tests/src/Common_Table_Operations/Numeric_Column_Spec.enso:64:9-98).        

@JaroslavTulach
Copy link
Member Author

JaroslavTulach commented Dec 19, 2025

With 3907ebf we seems to be down at a single failure when executed locally:

1 tests failed.
Failed tests: 'should.be.able.to.round-trip.a.BigInteger.column'

but then the extra run reports:

[Snowflake] Audit Logs

❌ should see Database operations performed on a table through the standard workflow
Found relevant events are:
❌ should see Database operations performed manually
An unexpected dataflow error (Not_Found) has been matched (at /home/runner/work/enso/enso/test/Table_Tests/src/Database/Common/Audit_Spec.enso:81:17-45).
✅ should still be able to open a local data link if not logged in
❌ should know the asset id of the data link used for the connection
Found relevant events are:

[Snowflake] Edge Cases

✅ materialize should respect the overridden type
❌ should be able to round-trip a BigInteger column
(1.1805916207174113E21): Expected a value of type Standard.Base.Data.Numbers.Integer but got a value [1.1805916207174113E21] of type Standard.Base.Data.Numbers.Float instead (at /home/runner/work/enso/enso/test/Snowflake_Tests/src/Snowflake_Spec.enso:371:64-84).
✅ should round-trip timestamptz column, preserving instant but converting to UTC
✅ will round-trip timestamp column without timezone by converting it to UTC

[Snowflake] Key-pair authentication

✅ using local private key
✅ using local private key encrypted with passphrase
❌ via the Cloud flow using secrets
An unexpected panic was thrown: org.enso.jvm.interop.impl.OtherJvmException: org.enso.base.enso_cloud.HideableValue is a sealed interface
✅ using local private key encrypted with passphrase stored in secret
✅ correctly handle existing key files
✅ fail if existing key is broken

suite_builder.group prefix+"Table.Offset with default fill strategy" group_builder->
colA = ["A", [1, 2, 3]]
t1 = build_sorted_table [colA]
sorted = Sorted_Table.Ordered (build_sorted_table [colA])
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • running single snowflake test takes ages
sbt:enso> runEngineDistribution --run test/Snowflake_Tests/ Decimal.and.Float.types

BuilderUtil.LOG.trace(
"makeLocal unsuccessful for {}:{} size {}", t.typeChar(), t.size(), storage.getSize());
switch (localType) {
case BigIntegerType _ -> {
Copy link
Member Author

@JaroslavTulach JaroslavTulach Dec 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sbt:enso> runEngineDistribution 
  --vm.D=polyglot.enso.classLoading=Standard.Snowflake:guest,hosted 
  --env ENSO_SNOWFLAKE_USER=aaa 
  --env ENSO_SNOWFLAKE_PASSWORD=bbb
  --env ENSO_SNOWFLAKE_ACCOUNT=ccc
  --env ENSO_SNOWFLAKE_DATABASE=CI_TEST_DB
  --run test/Snowflake_Tests should.be.able.to.round-trip.a.BigInteger.column

[FAILED] [Snowflake] Edge Cases: [0/1, 2127ms]                        

    - [FAILED] should be able to round-trip a BigInteger column [2127ms]
        Reason: (1.1805916207174113E21): Expected a value of type Standard.Base.Data.Numbers.Integer 
        but got a value [1.1805916207174113E21] of type Standard.Base.Data.Numbers.Float instead
       (at enso/enso/test/Snowflake_Tests/src/Snowflake_Spec.enso:375:17-37).
  • the problem is that with Proxy Storage there is Truffle value conversion
  • when obtaining values from the storage - e.g. when calling from Enso to Java
    • and BigInteger may be returned as double or float
    • when it can be exactly represented by double or float
  • by eagerly copying the content of the storage to local JVM we convert everything back to BigInteger
  • conversions to big integer had to be improved

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • it would be faster to use off-heap memory transfer
  • but I don't know what's appropriate Arrow format for java.math.BigInteger!?
  • there is Int+bit width, but the width can be 64 bits at max...

try {
return switch (o) {
case BigInteger big -> big;
case Double d -> new BigDecimal(d).toBigIntegerExact();
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

var isProxy = Proxy.isProxyClass(storage.getClass());
this.storage = isProxy ? Builder.makeLocal(storage) : storage;
var type = this.storage.getType();
LOG.debug(
Copy link
Member Author

@JaroslavTulach JaroslavTulach Dec 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Run any tests with org.enso.table.Logger.level=trace to see Column instances being created
  • for example running:
sbt:enso> runEngineDistribution 
  --vm.D=org.enso.table.Logger.level=trace
  --env ENSO_SNOWFLAKE_USER=aaa 
  --env ENSO_SNOWFLAKE_PASSWORD=bbb 
  --env ENSO_SNOWFLAKE_ACCOUNT=ccc 
  --env ENSO_SNOWFLAKE_DATABASE=CI_TEST_DB 
  --run test/Snowflake_Tests nicnicnic
  

8 tests succeeded.
0 tests failed.
0 tests skipped.
0 groups skipped.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • a25e91c delays the initialization a lot, but not fully

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

-compiler -libs Libraries: New libraries to be implemented -libs-API-change-Base Marks a PR that changes the public API of Standard.Base CI: No changelog needed Do not require a changelog entry for this PR.

Projects

Status: 🔧 Implementation

Development

Successfully merging this pull request may close these issues.

3 participants