Test Failures despite appropriate response

[mkhraisha]

We have ran into a couple of failing tests where when reconstructed manually via postman they return the correct response however in this test suite they are failing.
This applies to:

1. credential.credentialSubject should not be empty
2. "credential.issuer" MUST be a string or an object

Our implementation returns a 400, with a body that outlines what is missing, and the test suite marks that as a failure.

[msporny]

Hey @aljones15 thoughts on the above?

[aljones15]

I will look into this, but I checked here: https://w3c-ccg.github.io/vc-api-issuer-test-suite/

And for both test it says mavenet is returning a response and not an error. It might be the case that it is returning a response with a status 400 and the issue is on our side. I won't be able to check this today though as I am not at home and I keep the oauth2 stuff on my computer at home. @mkhraisha thanks for filing the issue and I should be able to look at this soon.

[aljones15]

@mkhraisha ok so I don't have the latest oauth2 token so I can't actually test the credentialSubject locally to debug, but some of our tests are in fact compound tests where multiple invalid states are tried that all sum up to a single normative statement. This PR changes that for the credentialSubject tests, by breaking them out into single negative tests: #34

[aljones15]

@mkhraisha the issuer tests have also been broken out into several new negative and positive tests here: #35

[msporny]

@mkhraisha the issuer tests have also been broken out into several new negative and positive tests here: #35

We can't do this as it makes it seem as if normative requirements are being asserted that are not in any spec. This will cause people to get confused about the normative statements in the spec. I left some suggestions in the PR.

[aljones15]

@mkhraisha ok so have some actual results:

• "credential.credentialSubject" MUST NOT be null - passing (Mavenet returns 400 HTTPError with a really good message)
• "credential.credentialSubject" MUST NOT be a boolean - failing mavenent returns 201
• "credential.credentialSubject" MUST NOT be a number - failing mavenent returns 201
• "credential.credentialSubject" MUST NOT be an empty Array - failing mavennet returns 201
• "credential.credentialSubject" MUST NOT be a string - failing mavennet returns 500

I don't have them for the credential.issuer tests yet. Thank you for raising this issue again as it has clarified the test results quite a bit.

[mkhraisha]

Thank you!

This helps quite a bit. Will dig into these on my end.