Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pulumi up results in: error reading from server: read tcp 127.0.0.1:63025->127.0.0.1:63022: wsarecv: An existing connection was forcibly closed by the remote host. #658

Closed
douglasg14b opened this issue Mar 25, 2024 · 9 comments
Assignees
Labels
impact/panic This bug represents a panic or unexpected crash kind/bug Some behavior is incorrect or out of spec resolution/fixed This issue was fixed

Comments

@douglasg14b
Copy link

What happened?

When I try an run puluumi up from a rpeviously working configuration. I get the following exception:

Diagnostics:
digitalocean:index:App (hiveAppService):
error: error reading from server: read tcp 127.0.0.1:63025->127.0.0.1:63022: wsarecv: An existing connection was forcibly closed by the remote host.

pulumi:pulumi:Stack (hive-npr):
panic: runtime error: index out of range [0] with length 0
goroutine 48 [running]:
github.com/digitalocean/terraform-provider-digitalocean/digitalocean/app.expandAppCORSPolicy({0xc000c7d100?, 0xc000f03170?, 0x1f8a416?})
/home/runner/work/pulumi-digitalocean/pulumi-digitalocean/upstream/digitalocean/app/app_spec.go:2034 +0xb45
github.com/digitalocean/terraform-provider-digitalocean/digitalocean/app.expandAppIngress({0xc000c7d0f0, 0xc000d44b70?, 0x1f8eea9?})
/home/runner/work/pulumi-digitalocean/pulumi-digitalocean/upstream/digitalocean/app/app_spec.go:2122 +0x72c
github.com/digitalocean/terraform-provider-digitalocean/digitalocean/app.expandAppSpec({0xc0010872a0?, 0x1f899ea?, 0xc0002b7650?})
/home/runner/work/pulumi-digitalocean/pulumi-digitalocean/upstream/digitalocean/app/app_spec.go:1027 +0xc05
github.com/digitalocean/terraform-provider-digitalocean/digitalocean/app.resourceDigitalOceanAppUpdate({0x24999e8, 0xc0002b7650}, 0xc000d3b180, {0x1d91d60?, 0xc0006263f0?})
/home/runner/work/pulumi-digitalocean/pulumi-digitalocean/upstream/digitalocean/app/resource_app.go:149 +0xdd
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).update(0xc0003121c0, {0x2499940, 0xc000889470}, 0xd?, {0x1d91d60, 0xc0006263f0})
/home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:817 +0x11b
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).Apply(0xc0003121c0, {0x2499940, 0xc000889470}, 0xc0007c1930, 0xc0005a6c80, {0x1d91d60, 0xc0006263f0})
/home/runner/go/pkg/mod/github.com/pulumi/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:924 +0x83a
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfshim/sdk-v2.v2Provider.Apply({0xc000620420?, {0x0?, 0x1d?, 0xffffffffffffffff?}}, {0x2499940, 0xc000889470}, {0x1f9c8d1, 0x10}, {0x2499fa8, 0xc00072ede0}, ...)
/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfshim/sdk-v2/provider.go:122 +0x165
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tfbridge.(*Provider).Update(0xc000533980, {0x2499940?, 0xc000888c30?}, 0xc0004f2090)
/home/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tfbridge/provider.go:1268 +0xbee
github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Update_Handler.func1({0x2499940, 0xc000888c30}, {0x1eecb20?, 0xc0004f2090})
/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider_grpc.pb.go:629 +0x75
github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x2499940, 0xc00051ca50}, {0x1eecb20, 0xc0004f2090}, 0xc000710aa0, 0xc000f882a0)
/home/runner/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/go/otgrpc/server.go:57 +0x3d0
github.com/pulumi/pulumi/sdk/v3/proto/go._ResourceProvider_Update_Handler({0x1f526c0?, 0xc000533980}, {0x2499940, 0xc00051ca50}, 0xc000d3a080, 0xc000a7b5c0)
/home/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/provider_grpc.pb.go:631 +0x135
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000608a00, {0x2499940, 0xc00051c8d0}, {0x24a34c0, 0xc00070a4e0}, 0xc0008a4120, 0xc0001cdaa0, 0x30ef930, 0x0)
/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1383 +0xe03
google.golang.org/grpc.(*Server).handleStream(0xc000608a00, {0x24a34c0, 0xc00070a4e0}, 0xc0008a4120)
/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1794 +0x100c
google.golang.org/grpc.(*Server).serveStreams.func2.1()
/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1027 +0x8b
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 55
/home/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1038 +0x135

error: update failed

Example

Unknown, this started happening out of the blue. I don't know how to repro it, or where I would even start.

Output of pulumi about

CLI
Version      3.109.0
Go Version   go1.22.0
Go Compiler  gc

Plugins
NAME          VERSION
digitalocean  4.25.2
nodejs        unknown

Host
OS       Microsoft Windows 10 Pro
Version  10.0.19045 Build 19045
Arch     x86_64

This project is written in nodejs: executable='C:\Program Files\nodejs\node.exe' version='v20.10.0'

Current Stack: douglasg14b/hive/npr

TYPE                                                                            URN
pulumi:pulumi:Stack                                                             urn:pulumi:npr::hive::pulumi:pulumi:Stack::hive-npr
pulumi:providers:digitalocean                                                   urn:pulumi:npr::hive::pulumi:providers:digitalocean::default_4_25_2
digitalocean:index/spacesBucket:SpacesBucket                                    urn:pulumi:npr::hive::digitalocean:index/spacesBucket:SpacesBucket::hiveThumbnailsBucket
digitalocean:index/vpc:Vpc                                                      urn:pulumi:npr::hive::digitalocean:index/vpc:Vpc::hiveServiceVpc
digitalocean:index/spacesBucket:SpacesBucket                                    urn:pulumi:npr::hive::digitalocean:index/spacesBucket:SpacesBucket::hiveCreativesBucket
digitalocean:index/spacesBucketCorsConfiguration:SpacesBucketCorsConfiguration  urn:pulumi:npr::hive::digitalocean:index/spacesBucketCorsConfiguration:SpacesBucketCorsConfiguration::hiveCreativesBucketCors
digitalocean:index/spacesBucketCorsConfiguration:SpacesBucketCorsConfiguration  urn:pulumi:npr::hive::digitalocean:index/spacesBucketCorsConfiguration:SpacesBucketCorsConfiguration::hiveThumbnailsBucketCors
digitalocean:index/databaseCluster:DatabaseCluster                              urn:pulumi:npr::hive::digitalocean:index/databaseCluster:DatabaseCluster::hiveDatabaseCluster
digitalocean:index/databaseUser:DatabaseUser                                    urn:pulumi:npr::hive::digitalocean:index/databaseUser:DatabaseUser::hideDatabaseUser
digitalocean:index/databaseDb:DatabaseDb                                        urn:pulumi:npr::hive::digitalocean:index/databaseDb:DatabaseDb::hiveDatabase
digitalocean:index/app:App                                                      urn:pulumi:npr::hive::digitalocean:index/app:App::hiveAppService


Found no pending operations associated with npr

Backend
Name           pulumi.com
URL            https://app.pulumi.com/douglasg14b
User           douglasg14b
Organizations  douglasg14b
Token type     personal

Pulumi locates its logs in C:\Users\Douglas\AppData\Local\Temp by default

Additional context

My deployments where working, but not they do not. Unsure if this is a local machine problem, or a problem with pulimi? Either way, there's probably a location here for error handling.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

@douglasg14b douglasg14b added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team labels Mar 25, 2024
@Frassle Frassle transferred this issue from pulumi/pulumi Mar 26, 2024
@iwahbe iwahbe added needs-repro Needs repro steps before it can be triaged or fixed impact/panic This bug represents a panic or unexpected crash and removed needs-triage Needs attention from the triage team labels Mar 26, 2024
@iwahbe
Copy link
Member

iwahbe commented Mar 26, 2024

Hey @douglasg14b. I'm sorry you're hitting that, it looks like something has gone wrong here. If you could provide us a program that reproduces the panic, we would be happy to take a look.

Unfortunately, it's very hard for us to fix the problem if we can't reproduce it.

@douglasg14b
Copy link
Author

I'm having trouble reproing this anywhere else. Is there more information I might be able to provide that may lead to this being more approachable by you or your team?

@iwahbe
Copy link
Member

iwahbe commented Apr 9, 2024

Sorry for the long wait time. If you run PULUMI_DEBUG_GRPC=logs.json pulumi up, then send us the resulting logs.json file that could help.

That json file might contain secrets, depending on how you have configured your provider. Make sure to scrub those.

@mdegel
Copy link

mdegel commented Apr 13, 2024

Might be unrelated, but I had a similar issue with pulumi-artifactory (Python):

With pulumi_artifactory==6.5.1 I get similar errors:

$ pulumi up
# ...
  artifactory:index:LocalGenericRepository (demo-staging-generic-local):
    error: error reading from server: read tcp 127.0.0.1:60632->127.0.0.1:60629: wsarecv: An existing connection was forcibly closed by the remote host.

Reverting to an older version pulumi_artifactory==6.4.5 yields actually more information about then real issue:

$ pulumi up
# ...
  artifactory:index:LocalGenericRepository (demo-staging-generic-local):
    error: 1 error occurred:
        *
    400 PUT https://my.url.com/artifactory/api/repositories/demo-staging-generic-local
    {
      "errors" : [ {
        "status" : 400,
        "message" : "This REST API is available only in Artifactory Pro (see: jfrog.com/artifactory/features). If you are already running Artifactory Pro please make sure your server is activated with a valid license key.\n"
      } ]
    }

In my case the newer versions of the generated pulumi libraries started hiding the actual internal errors (which themselves are unrelated to pulumi). Might be a similar issue here with unexpected responses from digitalocean?

@douglasg14b
Copy link
Author

douglasg14b commented Jun 27, 2024

@iwahbe

This happens when i have an app spec with CORS rules (Which even has a placeholder example header 😅):

When the CORS rules are removed, this issue no longer occurs.

new digitalocean.App('[REDACTED]', {
        spec: {
            name: '[REDACTED]',
            region: digitalocean.Region.NYC3,
            databases: [{
                name: '[REDACTED]',
                clusterName: '[REDACTED]',
                dbName: '[REDACTED]',
                dbUser: '[REDACTED]',
                production: true,
            }],
            services: [{
                name: '[REDACTED]',
                instanceSizeSlug: 'professional-xs',
                github: {
                    branch: '[REDACTED]',
                    repo: '[REDACTED]',
                    deployOnPush: false,
                },
                sourceDir: '[REDACTED]',
                dockerfilePath: '[REDACTED]',
                httpPort: 8080,
            }],
            staticSites: [{
                name: 'client-app',
                dockerfilePath: 'Dockerfile.client',
                outputDir: '/app',
                github: {
                    branch: '[REDACTED]',
                    repo: '[REDACTED]',
                    deployOnPush: false,
                },
                catchallDocument: 'index.html',
            }],
            ingress: {
                rules: [
                    {
                        component: {
                            name: '[REDACTED]',
                            preservePathPrefix: true,
                        },
                        match: {
                            path: {
                                prefix: '/api',
                            },
                        },
                        cors: {
                            allowMethods: ['GET', 'POST', 'PUT', 'DELETE'],
                            allowHeaders: ['*'],
                            exposeHeaders: ['X-My-Custom-Header'],
                            allowCredentials: false,
                        },
                    },
                    {
                        component: {
                            name: 'client-app',
                        },
                        match: {
                            path: {
                                prefix: '/',
                            },
                        },
                    },
                ],
            },
        },
    });

@pulumi-bot pulumi-bot added needs-triage Needs attention from the triage team and removed awaiting-feedback Blocked on input from the author labels Jun 27, 2024
@iwahbe iwahbe removed needs-repro Needs repro steps before it can be triaged or fixed needs-triage Needs attention from the triage team labels Jun 27, 2024
@iwahbe
Copy link
Member

iwahbe commented Jun 27, 2024

Hey @douglasg14b. Thanks for the repro. Amazingly, the code you gave actually works as given (except the reference to the free variable envs, which can be removed).

For a simplified repro:

import * as digitalocean from "@pulumi/digitalocean";

new digitalocean.App('[REDACTED]', {
  spec: {
    name: '[REDACTED]',
    region: digitalocean.Region.NYC3,
    ingress: {
      rules: [
        {
          cors: {
            allowMethods: ['GET', 'POST', 'PUT', 'DELETE'],
          },
        },
      ],
    },
  },
});

@iwahbe
Copy link
Member

iwahbe commented Jun 27, 2024

I have confirmed that this is a bug upstream and have opened an issue there: digitalocean/terraform-provider-digitalocean#1176. We will take the fix as soon as they do.

@iwahbe iwahbe added blocked The issue cannot be resolved without 3rd party action. awaiting-upstream The issue cannot be resolved without action in another repository (may be owned by Pulumi). labels Jun 27, 2024
@douglasg14b
Copy link
Author

Interesting, thank you for finding & confirming the issue.

@mikhailshilkov mikhailshilkov added resolution/fixed This issue was fixed and removed blocked The issue cannot be resolved without 3rd party action. awaiting-upstream The issue cannot be resolved without action in another repository (may be owned by Pulumi). labels Sep 20, 2024
@mikhailshilkov mikhailshilkov self-assigned this Sep 20, 2024
@mikhailshilkov
Copy link
Member

The upstream issue digitalocean/terraform-provider-digitalocean#1176 has been fixed in digitalocean/terraform-provider-digitalocean#1184, so closing this one too

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
impact/panic This bug represents a panic or unexpected crash kind/bug Some behavior is incorrect or out of spec resolution/fixed This issue was fixed
Projects
None yet
Development

No branches or pull requests

5 participants