Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid unecessary operations in kb-importer #537

Draft
wants to merge 85 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
85 commits
Select commit Hold shift + click to select a range
6721b1d
Check if bug exists before executing tasks. Delete bug and libraries …
pedrogalvao Mar 30, 2022
1a6297e
Fixed small mistakes
pedrogalvao Mar 30, 2022
94597d5
check if hashmap contains key
pedrogalvao Mar 30, 2022
764cef9
Fixed code style
pedrogalvao Apr 5, 2022
04b50a4
Removed files added by accident
pedrogalvao Apr 6, 2022
e6e3346
Removed repeated option variables from Tasks
pedrogalvao Apr 7, 2022
bbd4116
Removed skip condition in ImportVulnerability and changed tests accor…
Apr 8, 2022
c0ea508
Fixed code style
pedrogalvao Apr 8, 2022
3d83e6c
Created Manager and added function to extract tar (still have some er…
pedrogalvao Apr 12, 2022
e0dcbff
fixed some problems. still not working
pedrogalvao Apr 13, 2022
8247b13
Changing architecture (contains errors)
pedrogalvao Apr 15, 2022
8e8c751
Extracting and cloning in parallel, but without locks
pedrogalvao Apr 22, 2022
0e79f95
repositories locks
pedrogalvao Apr 25, 2022
93b0f7a
Parallel processing working for repos with tar (need to test more)
pedrogalvao Apr 29, 2022
4e50e90
started writing a test
pedrogalvao Apr 29, 2022
fbebcd9
Test for ExtractOrClone
pedrogalvao May 3, 2022
af7b43b
Started to write ImporterController
May 6, 2022
d9a4e6f
Removed comments, fixed small problems, added stopWatch to Import
May 11, 2022
d28d6f1
Rest service with Spring Framework
May 12, 2022
599a161
Added status and stop endpoints, removed some unused classes and othe…
May 13, 2022
59b00cf
Changes in shell scripts and Dockerfile
May 13, 2022
cc11bee
Removed export from kaybeeconf.yaml.sample
May 13, 2022
097f464
Added java command in start.sh
May 13, 2022
5ed1c79
Fixed some tests
May 16, 2022
ecfa8c1
Tests running again (with some errors)
May 17, 2022
86f6b25
Added missing resources. All tests succeed again
May 17, 2022
1964773
Properly cleaning directory after test
May 17, 2022
f46b984
More detailed status. Endpoints for single vulnerabilities
May 18, 2022
89ab062
Fixed problems with parameter refetchAllMs
May 18, 2022
16f9e37
Small changes. Removed unused classes
May 19, 2022
4b6d787
Removed some comments
May 19, 2022
38eeda3
Renamed Import to ImportCommand
pedrogalvao May 25, 2022
65346d7
Removed static variable that caused Empty Stack Error
pedrogalvao May 27, 2022
2f8a01f
Changed variable name in properties file
pedrogalvao May 27, 2022
7b12ce6
Removed thread from importSingleVuln
pedrogalvao May 27, 2022
738c8f1
Added licenses. Removed one file
pedrogalvao May 27, 2022
0930dea
Added license to kb-importer/pom.xml
pedrogalvao May 27, 2022
8e3f885
Fixed code style
pedrogalvao May 27, 2022
ee870dd
Fixed name and description in kb-importer pom.xml
pedrogalvao May 27, 2022
c827f28
function to retry failed vulnerabilities
pedrogalvao Jun 1, 2022
e170624
Fixed kaybeeconf, removed an unused dependency
pedrogalvao Jun 1, 2022
9e4e4b5
fixed code style
pedrogalvao Jun 1, 2022
bf286d1
Removed GoalClient again
pedrogalvao Jun 1, 2022
1b56295
Renamed a method and a variable in Manager
pedrogalvao Jun 1, 2022
9e929d5
Removed a Jenkinsfile
pedrogalvao Jun 1, 2022
a949408
catch IOException
pedrogalvao Jun 1, 2022
6bb6895
added curl installation in dockerfile
pedrogalvao Jun 1, 2022
5e41ded
Removed some comments, changed some logs, added GoalClient again
pedrogalvao Jun 2, 2022
dc06822
Added license in some new files
pedrogalvao Jun 2, 2022
9b07d69
Removed .gitignore form kb-importer directory
pedrogalvao Jun 2, 2022
4d2dffd
Moved snakeyaml dependency to kb-importer/pom.xml
pedrogalvao Jun 2, 2022
38b25e6
Fixed a mistake in kb-importer dockerfile
pedrogalvao Jun 2, 2022
d6a04e3
Removed a comment
pedrogalvao Jun 2, 2022
0016296
Documentation
pedrogalvao Jun 2, 2022
5fbf2a7
Removed MainTest.java
pedrogalvao Jun 2, 2022
6748bf0
Removed IOExceptio in commented code
pedrogalvao Jun 2, 2022
caf4d55
Removed printStackTrace from BackendConnectionException
pedrogalvao Jun 2, 2022
d65daaa
mend
pedrogalvao Jun 2, 2022
14a6e22
Fixed mistake in pom files (moved snakeyaml dependency)
pedrogalvao Jun 2, 2022
7999bf7
whitespace
pedrogalvao Jun 2, 2022
d9f9dc3
Removed GoalClient from wrong directory. Removed whitespace
pedrogalvao Jun 2, 2022
e1186cc
Reversed change in BasicHttpRequest
pedrogalvao Jun 2, 2022
ebdc259
Reversed change in kb-importer pom
pedrogalvao Jun 7, 2022
86a433b
Function isBugExisting throws BackendConnectionException
pedrogalvao Jun 7, 2022
62b6b17
Fixed code style
pedrogalvao Jun 7, 2022
a326de9
Substituted string literals by constants in tests
pedrogalvao Jun 7, 2022
e3d3381
feat: upgrade Docker images and fixes digest
Naramsim Jun 9, 2022
7f2829b
Added TestManager. Keeping failures as Exceptions instead of strings
pedrogalvao Jun 15, 2022
51cafa2
Added paths to properties. Other small changes
pedrogalvao Jun 21, 2022
914bdb9
Merge remote-tracking branch 'remote-2/fix_images'
pedrogalvao Jun 21, 2022
d8436f1
Corrected version + relativePath, added flatten-maven-plugin and star…
henrikplate Jul 28, 2022
1260868
Updated description and properties
henrikplate Jul 28, 2022
6c24fa5
Create CycloneDX BOM for REST services; Fix shell script starting the…
henrikplate Jul 28, 2022
59c4fb1
Added documentation with javadocs
pedrogalvao Aug 2, 2022
a4b766c
Start thread in constructor of ImporterController
pedrogalvao Aug 2, 2022
727698e
Changed Docker Compose so that start-steady works in clone and after …
henrikplate Aug 3, 2022
676edd1
Merge branch 'master' of https://github.com/pedrogalvao/steady
henrikplate Aug 3, 2022
2e966d9
Commented lang-python out
henrikplate Aug 3, 2022
29636df
Avoid mv of binaries into data folder
henrikplate Aug 3, 2022
b33c727
Using ProcessWrapper in kb-importer Manager
pedrogalvao Aug 10, 2022
2339bc9
Fixed thread names
pedrogalvao Aug 10, 2022
1aa0265
Added lang-python back in (its removal will be done in a separate PR)
henrikplate Aug 17, 2022
dfcd0a6
Fixed some bugs, added comments and improved log messages
henrikplate Aug 25, 2022
1222d2b
Changed default wait time to 1d
henrikplate Aug 26, 2022
bfc4ac4
Maintain stats for all statuses
henrikplate Sep 13, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
597 changes: 597 additions & 0 deletions java-files.txt

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
import org.eclipse.steady.kb.util.Metadata;
import org.eclipse.steady.shared.util.FileUtil;
import org.eclipse.steady.shared.util.VulasConfiguration;
import org.eclipse.steady.backend.BackendConnectionException;

/**
* import command
Expand All @@ -49,6 +50,7 @@ public class Import implements Command {
private static final String DIRECTORY_OPTION = "d";
private static final String OVERWRITE_OPTION = "o";
private static final String VERBOSE_OPTION = "v";
private static final String DELETE = "del";

private static final String UPLOAD_LONG_OPTION = "upload";
private static final String VERBOSE_LONG_OPTION = "verbose";
Expand Down Expand Up @@ -116,6 +118,25 @@ private void importVuln(HashMap<String, Object> args, String dirPath) {
return;
}

String vulnId = vuln.getVulnId();
boolean bugExists = false;
try {
bugExists = BackendConnector.getInstance().isBugExisting(vulnId);
} catch (BackendConnectionException e) {
log.error("Can't connect to the Backend");
return;
}

Boolean overwrite = (Boolean) args.get(OVERWRITE_OPTION);
if (bugExists) {
if (overwrite) {
args.put(DELETE, true);
} else {
log.info("Bug [{}] already exists in backend, analysis will be skipped", vulnId);
return;
}
}

List<Task> importTasks = TaskProvider.getInstance().getTasks(Command.NAME.IMPORT);

for (Task task : importTasks) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
*/
public class ImportAffectedLibraries implements Task {
private static final String OVERWRITE_OPTION = "o";
private static final String DELETE = "del";
private static final Logger log = org.apache.logging.log4j.LogManager.getLogger();

/** {@inheritDoc} */
Expand All @@ -56,6 +57,9 @@ public void execute(
if (artifacts == null || artifacts.isEmpty()) {
return;
}
if (args.containsKey(DELETE) && (boolean) args.get(DELETE)) {
backendConnector.deletePatchEvalResults(vuln.getVulnId(), AffectedVersionSource.KAYBEE);
}

List<AffectedLibrary> affectedLibsToUpsert = new ArrayList<AffectedLibrary>();
HashSet<org.eclipse.steady.shared.json.model.Artifact> ciaArtifactsCache = new HashSet<>();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ public class ImportVulnerability implements Task {
private static final String OVERWRITE_OPTION = "o";
private static final String DIRECTORY_OPTION = "d";
private static final String VERBOSE_OPTION = "v";
private static final String DELETE = "del";
pedrogalvao marked this conversation as resolved.
Show resolved Hide resolved

private static final Logger log = org.apache.logging.log4j.LogManager.getLogger();
private BackendConnector backendConnector = null;
Expand All @@ -67,9 +68,10 @@ public void execute(
String vulnId = vuln.getVulnId();
this.backendConnector = _backendConnector;

Boolean overwrite = (Boolean) args.get(OVERWRITE_OPTION);
if (!overwrite && getBackendConnector().isBugExisting(vulnId)) {
log.info("Bug [{}] already exists in backend, analysis will be skipped", vulnId);
if (args.containsKey(DELETE) && (boolean) args.get(DELETE)) {
backendConnector.deleteBug(vuln.getVulnId());
serenaponta marked this conversation as resolved.
Show resolved Hide resolved
} else if (backendConnector.isBugExisting(vuln.getVulnId())) {
log.info("Bug [{}] already exists in backend, analysis will be skipped", vuln.getVulnId());
return;
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1041,6 +1041,21 @@ public void uploadChangeList(String _bug, String _json) throws BackendConnection
req_list.send();
}

/**
* <p>deleteBug.</p>
*
* @param _bugId a {@link java.lang.String} object.
* @throws org.eclipse.steady.backend.BackendConnectionException if any.
*/
public void deleteBug(String _bugId) throws BackendConnectionException {

final BasicHttpRequest del_req =
new BasicHttpRequest(HttpMethod.DELETE, PathBuilder.bug(_bugId));
// payload cannot be empty otherwise request doesn t work
del_req.setPayload("[]", "application/json", true);
del_req.send();
}

/**
* <p>uploadCheckVersionResults.</p>
*
Expand Down
Empty file added non-compliant-files.txt
Empty file.