https://services.nvd.nist.gov/ endopint is giving 503 Service Unavailable #6758

hpriya19 · 2024-07-01T08:24:12Z

OWASP Client version: 9.0.8

We are seeing the services endpoint timeout from last couple of days

RUN /tmp/dependency-check/bin/dependency-check.sh --updateonly  --nvdApiKey ${NVD_API_KEY} --nvdApiDelay 3000
Picked up JAVA_TOOL_OPTIONS: -Dhttp.proxyHost=*** -Dhttp.proxyPort=80 -Dhttps.proxyHost=*** -Dhttps.proxyPort=80
[INFO] Checking for updates
[WARN] NVD API request failures are occurring; retrying request for the 5 time
[WARN] NVD API request failures are occurring; retrying request for the 6 time
[WARN] NVD API request failures are occurring; retrying request for the 7 time
[WARN] NVD API request failures are occurring; retrying request for the 5 time
[WARN] NVD API request failures are occurring; retrying request for the 6 time
[WARN] NVD API request failures are occurring; retrying request for the 7 time

We have verified the nvdApiKey is valid.
The curl response to the endpoint gives the response below

curl -H "Accept: application/json" -H "apiKey:******" -v https://services.nvd.nist.gov/rest/json/cves/2.0\?cpeName\=cpe:2.3:o:microsoft:windows_10:1607:\*:\*:\*:\*:\*:\*:\*
* Uses proxy env variable https_proxy ***
*   Trying 10.68.69.53...
* TCP_NODELAY set
* Connected to <proxy> (10.68.69.53) port 80 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to services.nvd.nist.gov:443
> CONNECT services.nvd.nist.gov:443 HTTP/1.1
> Host: services.nvd.nist.gov:443
> User-Agent: curl/7.61.1
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.0 200 Connection established
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=*.nvd.nist.gov
*  start date: May 31 01:13:03 2024 GMT
*  expire date: Aug 29 01:13:02 2024 GMT
*  subjectAltName: host "services.nvd.nist.gov" matched cert's "*.nvd.nist.gov"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET /rest/json/cves/2.0?cpeName=cpe:2.3:o:microsoft:windows_10:1607:*:*:*:*:*:*:* HTTP/1.1
> Host: services.nvd.nist.gov
> User-Agent: curl/7.61.1
> Accept: application/json
> apiKey:****
> 
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/1.1 503 Service Unavailable
< content-length: 107
< cache-control: no-cache
< content-type: text/html
< 
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
* Connection #0 to host *** left intact

Is there a change in Service endpoint? Can we get some assistance with this.

The text was updated successfully, but these errors were encountered:

hpriya19 · 2024-07-01T08:28:27Z

From the Web;
The endpoint intermittently shows

And also TBD with initial public draft

richardzaat · 2024-07-01T08:39:22Z

We are facing the same issue, cannot update the OWASP DC database. Might it be related to NVD moving to CVSS 4.0 on June 27th ? https://nvd.nist.gov/general/news/cvss-v4-0-official-support

fc-skaviedes · 2024-07-01T08:57:29Z

We also have myriads of jobs/pipelines failing due to this issue. Though this should teach us a lesson not to rely on its availability.

himanshukumar4642 · 2024-07-01T09:02:57Z

@jeremylong Can you please help us with this issue ?

jruhe-adesso · 2024-07-01T09:15:01Z

Same here 9.2.0

kheyang · 2024-07-01T09:19:44Z

The NVD API (https://nvd.nist.gov/developers/vulnerabilities) seems on and off at the moment

1 attempt shows 200 response with expected results

[root@blabla ~]# curl -H "Accept: application/json" -v https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2019-1010218
* About to connect() to services.nvd.nist.gov port 443 (#0)
*   Trying 18.235.227.114...
* Connected to services.nvd.nist.gov (18.235.227.114) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=*.nvd.nist.gov
*       start date: May 31 01:13:03 2024 GMT
*       expire date: Aug 29 01:13:02 2024 GMT
*       common name: *.nvd.nist.gov
*       issuer: CN=R3,O=Let's Encrypt,C=US
> GET /rest/json/cves/2.0?cveId=CVE-2019-1010218 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: services.nvd.nist.gov
> Accept: application/json
> 
< HTTP/1.1 200 OK
< content-type: application/json
< x-frame-options: SAMEORIGIN
< access-control-allow-origin: *
< access-control-allow-headers: accept, apiKey, content-type, origin, x-requested-with
< access-control-allow-methods: GET, HEAD, OPTIONS
< access-control-allow-credentials: false
< date: Mon, 01 Jul 2024 09:13:29 GMT
< content-length: 2639
< apikey: No
< strict-transport-security: max-age=31536000
< 
{"resultsPerPage":1,"startIndex":0,"totalResults":1,"format":"NVD_CVE","version":"2.0","timestamp":"2024-07-01T09:13:30.227","vulnerabilities":[{"cve":{"id":"CVE-2019-1010218","sourceIdentifier":"[email protected]","published":"2019-07-22T18:15:10.917","lastModified":"2020-09-30T13:40:18.163","vulnStatus":"Analyzed","cveTags":[],"descriptions":[{"lang":"en","value":"Cherokee Webserver Latest Cherokee Web server Upto Version 1.2.103 (Current stable) is affected by: Buffer Overflow - CWE-120. The impact is: Crash. The component is: Main cherokee command. The attack vector is: Overwrite argv[0] to an insane length with execl. The fixed version is: There's no fix yet."},{"lang":"es","value":"El servidor web de Cherokee más reciente de Cherokee Webserver Hasta Versión 1.2.103 (estable actual) está afectado por: Desbordamiento de Búfer - CWE-120. El impacto es: Bloqueo. El componente es: Comando cherokee principal. El vector de ataque es: Sobrescribir argv[0] en una longitud no sana con execl. La versión corregida es: no hay ninguna solución aún."}],"metrics":{"cvssMetricV31":[{"source":"[email protected]","type":"Primary","cvssData":{"version":"3.1","vectorString":"CVSS:3.1\/AV:N\/AC:L\/PR:N\/UI:N\/S:U\/C:N\/I:N\/A:H","attackVector":"NETWORK","attackComplexity":"LOW","privilegesRequired":"NONE","userInteraction":"NONE","scope":"UNCHANGED","confidentialityImpact":"NONE","integrityImpact":"NONE","availabilityImpact":"HIGH","baseScore":7.5,"baseSeverity":"HIGH"},"exploitabilityScore":3.9,"impactScore":3.6}],"cvssMetricV2":[{"source":"[email protected]","type":"Primary","cvssData":{"version":"2.0","vectorString":"AV:N\/AC:L\/Au:N\/C:N\/I:N\/A:P","accessVector":"NETWORK","accessComplexity":"LOW","authentication":"NONE","confidentialityImpact":"NONE","integrityImpact":"NONE","availabilityImpact":"PARTIAL","baseScore":5.0},"baseSeverity":"MEDIUM","exploitabilityScore":10.0,"impactScore":2.9,"acInsufInfo":false,"obtainAllPrivilege":false,"obtainUserPrivilege":false,"obtainOtherPrivilege":false,"userInteractionRequired":false}]},"we* Connection #0 to host services.nvd.nist.gov left intact
aknesses":[{"source":"[email protected]","type":"Primary","description":[{"lang":"en","value":"CWE-787"}]},{"source":"[email protected]","type":"Secondary","description":[{"lang":"en","value":"CWE-120"}]}],"configurations":[{"nodes":[{"operator":"OR","negate":false,"cpeMatch":[{"vulnerable":true,"criteria":"cpe:2.3:a:cherokee-project:cherokee_web_server:*:*:*:*:*:*:*:*","versionEndIncluding":"1.2.103","matchCriteriaId":"DCE1E311-F9E5-4752-9F51-D5DA78B7BBFA"}]}]}],"references":[{"url":"https:\/\/i.imgur.com\/PWCCyir.png","source":"[email protected]","tags":["Exploit","Third Party Advisory"]}]}}]}

On another attempt right after shows 503 response:

[root@blabla ~]# curl -H "Accept: application/json" -v https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=CVE-2019-1010218
* About to connect() to services.nvd.nist.gov port 443 (#0)
*   Trying 18.235.227.114...
* Connected to services.nvd.nist.gov (18.235.227.114) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=*.nvd.nist.gov
*       start date: May 31 01:13:03 2024 GMT
*       expire date: Aug 29 01:13:02 2024 GMT
*       common name: *.nvd.nist.gov
*       issuer: CN=R3,O=Let's Encrypt,C=US
> GET /rest/json/cves/2.0?cveId=CVE-2019-1010218 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: services.nvd.nist.gov
> Accept: application/json
> 
< HTTP/1.1 503 Service Unavailable
< content-length: 107
< cache-control: no-cache
< content-type: text/html
< 
<html><body><h1>503 Service Unavailable</h1>
No server is available to handle this request.
</body></html>
* Connection #0 to host services.nvd.nist.gov left intact

No luck on the builds though, all of them are failing.

schilli91 · 2024-07-01T09:34:02Z

Also affected with Dependency-Check Core version 9.0.9.
Is there perhaps an official page by nvd that indicates availability? I couldn't find anything.

jruhe-adesso · 2024-07-01T09:41:10Z

Workaround is autoUpdate=false

ftiercelin · 2024-07-01T09:50:24Z

another workaround:

download meta and json.gz files from https://nvd.nist.gov/vuln/data-feeds#JSON_FEED
put them on a local http server
configure this location as data feed:
<nvdDatafeedUrl>http://internal-mirror.mycorp.com/nvdcve-{0}.json.gz</nvdDatafeedUrl>

richardzaat · 2024-07-01T11:26:03Z

Workaround in our case: We have private build servers on which we have OWASP DC installed. Each team running their build runs with --noupdate, and every 4 hours we run a separate job to update all OWASP DC databases. In this case at least the builds from the teams keep running, and they never have to wait for updates. In case updating fails, and it has failed before, only we, the system team, are impacted.

mebigfatguy · 2024-07-01T13:54:44Z

probably not much this project can do anything about, altho it would be nice if it would fail fast. rather than hanging for hours.

chadlwilson · 2024-07-01T15:48:29Z

IMHO, it's probably at root due to #6746 and the NVD APIs getting a DDoS from many thousands of ODC clients constantly retrying requests that will never succeed (until fixed within ODC, and every user around the world upgrading from v9 to v10).

himanshukumar4642 · 2024-07-02T04:57:12Z

Any updates on this issue ? looks like 10.0.0 version has some bug while updating CVEs.

LukaszRacon · 2024-07-02T05:13:58Z

Try with setting api key --nvdApiKey that resolved issues for us with fetching data.

https://nvd.nist.gov/general/news/API-Key-Announcement
https://nvd.nist.gov/developers/request-an-api-key

chadlwilson · 2024-07-02T05:25:15Z

Needing to use an API key to avoid certain errors/rate limiting has been true since v9 of ODC (especially if starting without an existing cache/database), but if you are recently upgrading from v8 -> v9 or v8 -> v10 you may have this as a root cause for your issue.

But there also appear to be some other load problems, at least yesterday.

As for bugs and compatibility problems with v10 and certain databases/setups, you probably need to follow #6746 . Right now I don't think there is any working version of ODC v9 or v10 that is fine across all setups. Might be OK with an H2 Database but definitely some issues with Postgres and MySQL? Not sure about v8.

jeremylong · 2024-07-02T12:07:13Z

@ftiercelin the datafeeds are no longer supported. To use them you would have to be using an unsupported version of ODC - prior to 9.0.0.

jeremylong · 2024-07-02T12:09:32Z

Yes, the NVD API is having capacity issues right now. I have seen successful calls into the API - generally using an API key. There is not much the project (ODC) can do about this.

I am releasing 10.0.1 - which contains a few minor fixes if you are using postgres or mssql as a centralized database. The release also removes a bit of debug logging that makes it look like errors occurred when they did not.

zg2pro · 2024-07-02T15:17:48Z

Hi @jeremylong
Where can you follow status of that capacity issue? Do they have a page with a news feed?
I've also had issues for 2 days now and being not able to update our CVE database is becoming critical.

chadlwilson · 2024-07-02T15:28:41Z

@zg2pro No, they don't.

They have widespread and well documented funding and governance problems which you can google and read about. They have had persistent periodic technical problems since the launch of the API especially.

In any case, if you are relying upon OSS and a free service from the US government's NVD for your critical workflows rather than a commercial vendor, it's all best efforts.

Personally (just a hunch), I have a feeling that the NVD capacity issues will abate as people update to ODC 10.0.1+ and gradually get their caches back in sync - or another bunch of folks get forced by the issues to relook at their caching strategy.

In any case, my H2-database ODC cache updated slowly but successfully on 10.0.1 after 3 days of nothing flowing through, so it does seem generally to be working, especially if you are not trying to populate a database from scratch.

ftiercelin · 2024-07-02T15:50:52Z

Hi @jeremylong , if the datafeed option is not relevant anymore, you might want to update the example page which is showcasing this option: http://jeremylong.github.io/DependencyCheck/dependency-check-maven/index.html#Example_5:
(and btw thanks for the fantastic job, this project is really neat!)

seancorfield · 2024-07-03T00:29:34Z

In any case, my H2-database ODC cache updated slowly but successfully on 10.0.1 after 3 days of nothing flowing through, so it does seem generally to be working, especially if you are not trying to populate a database from scratch.

Sounds like I just picked a bad time to blow away my local database (it seemed to have gotten corrupted) but this was also a good opportunity to update to 10.0.1 from 9.x!

Update: it took a while, with lots of retry messages in the output, but it did succeed in rebuilding/updating the database using 10.0.1.

jeremylong · 2024-07-03T00:52:24Z

The NVD is aware of the availability issues and they are working on it; see https://groups.google.com/a/list.nist.gov/g/nvd-news/c/sJmF-2XIA80

juanmanuelromeraferrio · 2024-07-03T08:29:37Z

Hi @jeremylong,
I'm currently using Dependency Check v9.0.9 with a Postgres database and have been encountering 503 errors for the NVD services since last Friday. I noticed that there are several related issues reported.

My doubt is, given my setup (v9.0.9 of Dependency Check with a Postgres database), is upgrading to v10.0.1 the only solution, or are there alternative fixes available?

Thanks!

jeremylong · 2024-07-03T10:14:52Z

Version 9 will not be able to parse the current results from the NVD. 10+ is the only currently supported version.

RobSHK · 2024-07-03T10:21:08Z

Does the 10+ version also support Oracle databases? Our NVD mirroring is hosted on Oracle DB.

juanmanuelromeraferrio · 2024-07-03T17:19:30Z

Version 9 will not be able to parse the current results from the NVD. 10+ is the only currently supported version.

Thanks for the response. Regarding v9, are there any plans to release an update that, while it won't support CVSS4.0, ensures compatibility without breaking?

chadlwilson · 2024-07-03T17:28:50Z

Version 9 will not be able to parse the current results from the NVD. 10+ is the only currently supported version.

Thanks for the response. Regarding v9, are there any plans to release an update that, while it won't support CVSS4.0, ensures compatibility without breaking?

@juanmanuelromeraferrio you can read some discussion about this at #6756 but it seems unlikely at this stage. However there shouldn't be any remaining bugs preventing you from using v10 to my knowledge and it's not really a breaking change except for folks with certain DB configurations. It will also make no difference to whether the NVD API responds with timeout related errors to have a new V9.

As to whether this needed to be done with a v10, I think it would have been possible to first release a v9 with my original compatibility patch, but then use a different approach to deal with Jeremy's concerns about lost CVSSv4 data (release v10 later after more time for testing, with CVSSv4 support and a data validity epoch "snap back* that forces re-retrieval of all CVEs since the CVSSv4 NVD go-live date when v10 is installed, which would override local last updated records as a one-off), but it's all work and like all OSS needs time and effort, sometimes more than is available for free 😅

marcelstoer · 2024-07-04T05:38:02Z

<OT>

IMHO might not be a good idea to have multiple agents potentially writing to the same network folder.

there is a lock file appearing there during the update

We've had this setup for years with potentially ~10 agents going against the same data directory simultaneously. Haven't had any issues so far. If an agent sees there's a lock file in place (i.e. another agent running an update currently), it waits until the lock file disappears. It prints a message to that effect to the console. Of course, should the agent running the update die, the lock file won't be removed automatically thereby causing issues for the waiting agents.
</OT>

chadlwilson · 2024-07-04T05:48:58Z

We've had this setup for years with potentially ~10 agents going against the same data directory simultaneously. Haven't had any issues so far. If an agent sees there's a lock file in place (i.e. another agent running an update currently), it waits until the lock file disappears. It prints a message to that effect to the console. Of course, should the agent running the update die, the lock file won't be removed automatically thereby causing issues for the waiting agents.

Thanks @marcelstoer ! TIL!

Might be a good idea for us to update the documentation to note this as a valid configuration for the multiple-writers option. Maybe I'll have a go at that (and try and fix the formatting at the same time!)

ankurga · 2024-07-04T10:31:23Z

@chadlwilson I can also vouch for the setup that @marcelstoer is having because we are also using a similar config for more than 2 years and likewise, we also get the warning if an update is already in place and another agent tries to update the same data.

jeremylong · 2024-07-04T12:45:53Z

@ftiercelin I missed your comment about the data feed option - it is used by many organizations that create a datafeed using the vulnz-cli.

chadlwilson · 2024-07-04T14:26:51Z

@marcelstoer @ankurga If you'd like to review my suggested update to the docs at https://github.com/jeremylong/DependencyCheck/pull/6804/files?short_path=2fc6754#diff-2fc67544c882e9fcef577dbcac2c1d6285dbb9bc8d890a9637fa35d09f88e72c it may be useful :-)

ankurga · 2024-07-05T05:07:49Z

@chadlwilson Looks good. Thank you !! :)

adriaanse-deltares · 2024-07-05T07:53:32Z

Right, thanks again Chad, this setup has been running fine now with dozens of teamcity agents running on the same data directory, significantly reducing our load on the NVD backend,

njefsky · 2024-07-05T10:54:51Z

Still same issue after upgrading to 10.0.1.
`
========================== Starting Command Output ===========================
"C:\Program Files\PowerShell\7\pwsh.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'C:\agent_work_temp\e3476919-ec8c-430a-ac54-f3bd7695d183.ps1'"
09:19:52,511 |-INFO in ch.qos.logback.classic.LoggerContext[dependency-check] - Could NOT find resource [logback-test.xml]
09:19:52,511 |-INFO in ch.qos.logback.classic.LoggerContext[dependency-check] - Found resource [logback.xml] at [jar:file:/C:/dependency-check/dependency-check/lib/dependency-check-cli-10.0.1.jar!/logback.xml]
09:19:52,511 |-WARN in ch.qos.logback.classic.LoggerContext[dependency-check] - Resource [logback.xml] occurs multiple times on the classpath.
09:19:52,511 |-WARN in ch.qos.logback.classic.LoggerContext[dependency-check] - Resource [logback.xml] occurs at [jar:file:/C:/dependency-check/dependency-check/lib/dependency-check-cli-10.0.1.jar!/logback.xml]
09:19:52,511 |-WARN in ch.qos.logback.classic.LoggerContext[dependency-check] - Resource [logback.xml] occurs at [jar:file:/C:/dependency-check/dependency-check/lib/dependency-check-cli-9.0.9.jar!/logback.xml]
09:19:52,527 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@4d3167f4 - URL [jar:file:/C:/dependency-check/dependency-check/lib/dependency-check-cli-10.0.1.jar!/logback.xml] is not of type file
09:19:52,667 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
09:19:52,667 |-INFO in ch.qos.logback.classic.joran.action.ContextNameAction - Setting logger context name as [dependency-check]
09:19:52,667 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
09:19:52,683 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [console]
09:19:52,683 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
09:19:52,699 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.commons.jcs] to ERROR
09:19:52,699 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.hc] to ERROR
09:19:52,699 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
09:19:52,699 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [console] to Logger[ROOT]
09:19:52,699 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
09:19:52,699 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@ed9d034 - Registering current configuration as safe fallback point

[INFO] Checking for updates
[WARN] NVD API request failures are occurring; retrying request for the 1 time
[WARN] NVD API request failures are occurring; retrying request for the 1 time
[WARN] NVD API request failures are occurring; retrying request for the 2 time
[WARN] NVD API request failures are occurring; retrying request for the 3 time
[WARN] NVD API request failures are occurring; retrying request for the 1 time
[WARN] NVD API request failures are occurring; retrying request for the 1 time
[WARN] NVD API request failures are occurring; retrying request for the 2 time
[WARN] NVD API request failures are occurring; retrying request for the 3 time
[WARN] NVD API request failures are occurring; retrying request for the 4 time
[ERROR] Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:389)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:116)
at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
at org.owasp.dependencycheck.Engine.initializeAndUpdateDatabase(Engine.java:711)
at org.owasp.dependencycheck.Engine.analyzeDependencies(Engine.java:637)
at org.owasp.dependencycheck.App.runScan(App.java:262)
at org.owasp.dependencycheck.App.run(App.java:194)
at org.owasp.dependencycheck.App.main(App.java:89)
Caused by: io.github.jeremylong.openvulnerability.client.nvd.NvdApiException: NVD Returned Status Code: 503
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:357)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
at io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient.next(NvdCveClient.java:341)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:349)
... 7 common frames omitted
[INFO] Skipping Known Exploited Vulnerabilities update check since last check was within 24 hours.
[WARN] Unable to update 1 or more Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities.
[ERROR] Unable to continue dependency-check analysis.
[ERROR] One or more fatal errors occurred
[ERROR] Error updating the NVD Data
[ERROR] No documents exist
##[error]PowerShell exited with code '1'.

`

Chris-Turner-NIST · 2024-07-08T16:53:20Z

Looks like this issue is referenced in a lot of places. Adding this here for awareness.

See Mandatory Upgrade Notice (10.0.2)

See Reason for Upgrade

AmyChihHi · 2024-07-09T10:05:12Z

Workaround in our case: We have private build servers on which we have OWASP DC installed. Each team running their build runs with --noupdate, and every 4 hours we run a separate job to update all OWASP DC databases. In this case at least the builds from the teams keep running, and they never have to wait for updates. In case updating fails, and it has failed before, only we, the system team, are impacted.

@richardzaat
can you show that about how do you run a separate job to update all OWASP DC databases?

jeremylong · 2024-07-09T10:09:16Z

@AmyChihHi see the documentation here; it sounds like they are using option 3 and caching the data directory.

chadlwilson · 2024-07-09T10:14:23Z

To add on further, probably with the single node database updater option if --noupdate is involved as described by @richardzaat earlier. To summarise what is documented, you typically either have shared (readonly) storage for the database, or you have all the readers download-unpack or rsync a copy of the data dir from somewhere central prior to running.

The specific methodology for doing so is highly dependent on your environment and what you have available to you.

AmyChihHi · 2024-07-09T10:33:26Z

@AmyChihHi see the documentation here; it sounds like they are using option 3 and caching the data directory.

I tried it but got the following error message:

[INFO] Checking for updates
[ERROR] Error updating the NVD Data; the NVD returned a 403 or 404 error

Please ensure your API Key is valid; see https://github.com/jeremylong/Open-Vulnerability-Project/tree/main/vulnz#api-key-is-used-and-a-403-or-404-error-occurs

If your NVD API Key is valid try increasing the NVD API Delay.

If this is ocurring in a CI environment
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data; the NVD returned a 403 or 404 error

Please ensure your API Key is valid; see https://github.com/jeremylong/Open-Vulnerability-Project/tree/main/vulnz#api-key-is-used-and-a-403-or-404-error-occurs

If your NVD API Key is valid try increasing the NVD API Delay.

If this is ocurring in a CI environment
at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:387)
at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:116)
at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:878)
at org.owasp.dependencycheck.App.runUpdateOnly(App.java:427)
at org.owasp.dependencycheck.App.run(App.java:172)
at org.owasp.dependencycheck.App.main(App.java:89)

chadlwilson · 2024-07-09T10:42:57Z

@AmyChihHi see #6816 and #6817 - unless you are getting 503s, you have a different problem than this one.

ftiercelin · 2024-07-09T12:21:55Z

@AmyChihHi in my case the error
... was actually a SSL handshake error with services.nvd.nist.gov
when debugging, the error was printed, but otherwise, I was just getting
[ERROR] Error updating the NVD Data; the NVD returned a 403 or 404 error

I used https://github.com/jeremylong/InstallCert to install the new cert for services.nvd.nist.gov.
In my case I am behind a corporate proxy (Zscaler) which probably explains why I had to add the certs for this new url

@jeremylong : it seems the "Error updating the NVD Data; the NVD returned a 403 or 404 error" error message is not always accurate. It seems to be also triggered by
caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed:sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target '

chadlwilson · 2024-07-09T13:02:05Z

@ftiercelin it'd probably be helpful if you filed any misleading error messages as a separate issue with full stack trace, as opportunities for improvement are otherwise going to get lost here (just as they did with 9.x when the switch to this domain was first done - the use of this NVD domain is only new for folks who haven't been using 9.x earlier than this change).

Also, we are now conflating 503 errors with 403 errors in one ticket which is even more confusing for folks trying to figure out whether their problem is transient or permanent (without user action).

jeremylong · 2024-07-09T13:06:16Z

@chadlwilson I'll also be improving the error reporting around the NVD API Key (see ovc/55678e58084496f39ca6a92a9fc7dc90b5e82231). I've seen at least one CI with an incorrectly set NVD API Key provide an empty string - which can really cause some confusion and problems.

chadlwilson · 2024-07-09T13:27:08Z

Yeah, I've seen issues before with Gradle system properties and project properties being read incorrectly (or at the wrong part of Gradle lifecycle) so it'll definitely help to have a bit more "it looks like you're doing something wrong..." here.

hpriya19 added the question label Jul 1, 2024

mprins mentioned this issue Jul 1, 2024

Dependency-Check Core version 10.0.0 #6760

Closed

james-d-brown mentioned this issue Jul 1, 2024

Find alternative for dependecyCheck NOAA-OWP/wres#23

Open

jeremylong mentioned this issue Jul 2, 2024

NVD API request failures are occurring; retrying request for the time out (Error updating the NVD Data) #6768

Closed

jeremylong mentioned this issue Jul 2, 2024

Failing on processing NVD data due to unknown field: cvssMetricV40 #6746

Closed

linsomniac mentioned this issue Jul 3, 2024

NVD API update is hanging / resulting in several errors #6795

Closed

linsomniac mentioned this issue Jul 3, 2024

stops working once adding an NVD API key #6791

Closed

This was referenced Jul 3, 2024

Version 10.0.1 is taking longer to complete the 9.2.0 #6793

Closed

NVD API update is hanging #6797

Closed

chadlwilson mentioned this issue Jul 4, 2024

docs: Further improve formatting and docs of H2 database caching strats #6804

Merged

jeremylong mentioned this issue Jul 5, 2024

Dependency-Check-plugin failing while updating NVD #6806

Closed

aikebah mentioned this issue Jul 5, 2024

Error updating the NVD Data always at the same Index 224000 (82%) v9.2.0 #6808

Closed

nscuro mentioned this issue Jul 8, 2024

NVD Mirror Returned Status Code: 503 DependencyTrack/dependency-track#3927

Closed

2 tasks

This was referenced Jul 10, 2024

Getting 403 or 404 exception with NVD API key #6816

Closed

Please Read: Mandatory Upgrade to 10.0.2 #6817

Open

https://services.nvd.nist.gov/ endopint is giving 503 Service Unavailable #6758

https://services.nvd.nist.gov/ endopint is giving 503 Service Unavailable #6758

Comments

hpriya19 commented Jul 1, 2024 • edited Loading

hpriya19 commented Jul 1, 2024

richardzaat commented Jul 1, 2024

fc-skaviedes commented Jul 1, 2024

himanshukumar4642 commented Jul 1, 2024

jruhe-adesso commented Jul 1, 2024

kheyang commented Jul 1, 2024 • edited Loading

schilli91 commented Jul 1, 2024

jruhe-adesso commented Jul 1, 2024 • edited Loading

ftiercelin commented Jul 1, 2024 • edited Loading

richardzaat commented Jul 1, 2024

mebigfatguy commented Jul 1, 2024

chadlwilson commented Jul 1, 2024 • edited Loading

himanshukumar4642 commented Jul 2, 2024

LukaszRacon commented Jul 2, 2024

chadlwilson commented Jul 2, 2024

jeremylong commented Jul 2, 2024

jeremylong commented Jul 2, 2024

zg2pro commented Jul 2, 2024

chadlwilson commented Jul 2, 2024 • edited Loading

ftiercelin commented Jul 2, 2024

seancorfield commented Jul 3, 2024 • edited Loading

jeremylong commented Jul 3, 2024

juanmanuelromeraferrio commented Jul 3, 2024

jeremylong commented Jul 3, 2024

RobSHK commented Jul 3, 2024

juanmanuelromeraferrio commented Jul 3, 2024

chadlwilson commented Jul 3, 2024 • edited Loading

marcelstoer commented Jul 4, 2024

chadlwilson commented Jul 4, 2024

ankurga commented Jul 4, 2024

jeremylong commented Jul 4, 2024

chadlwilson commented Jul 4, 2024

ankurga commented Jul 5, 2024

adriaanse-deltares commented Jul 5, 2024

njefsky commented Jul 5, 2024 • edited Loading

Chris-Turner-NIST commented Jul 8, 2024

AmyChihHi commented Jul 9, 2024 • edited Loading

jeremylong commented Jul 9, 2024

chadlwilson commented Jul 9, 2024

AmyChihHi commented Jul 9, 2024

chadlwilson commented Jul 9, 2024 • edited Loading

ftiercelin commented Jul 9, 2024

chadlwilson commented Jul 9, 2024

jeremylong commented Jul 9, 2024

chadlwilson commented Jul 9, 2024

hpriya19 commented Jul 1, 2024 •

edited

Loading

kheyang commented Jul 1, 2024 •

edited

Loading

jruhe-adesso commented Jul 1, 2024 •

edited

Loading

ftiercelin commented Jul 1, 2024 •

edited

Loading

chadlwilson commented Jul 1, 2024 •

edited

Loading

chadlwilson commented Jul 2, 2024 •

edited

Loading

seancorfield commented Jul 3, 2024 •

edited

Loading

chadlwilson commented Jul 3, 2024 •

edited

Loading

njefsky commented Jul 5, 2024 •

edited

Loading

AmyChihHi commented Jul 9, 2024 •

edited

Loading

chadlwilson commented Jul 9, 2024 •

edited

Loading