Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Benchmark against Legacy Chopper #20

Open
jordr opened this issue Dec 19, 2019 · 4 comments
Open

Benchmark against Legacy Chopper #20

jordr opened this issue Dec 19, 2019 · 4 comments
Labels
benchmark

Comments

@jordr
Copy link
Owner

jordr commented Dec 19, 2019 •
edited
Loading

Experiments with PSPA Chopper on tamworth

The PSPA paper results

https://www.doc.ic.ac.uk/~jruiz/resources/pspa-results.png (SRG-restricted link)

Summary of experiments on tamworth

Using chopper-pspa, 0d78f18afe72b99caa23ea7f6ba8cb63a1e5e933, SVF-dynamic, 7db622cda36bb2e057503615334b4f594cc749ba

Vulnerability Search SPA PSPA SPA+heur. PSPA+heur.
CVE-2012-1569 RANDOM 04:14 01:28 01:19 01:34
DFS 06:39 03:46 00:34 00:20
COVERAGE 06:36 00:47 01:31 01:34
CVE-2014-3467_1 RANDOM T.O. T.O. 05:51 08:38
DFS 04:37 02:52 00:03 00:03
COVERAGE T.O. T.O. 03:35 05:16
CVE-2014-3467_2 RANDOM 00:02 00:02 05:42 08:33
DFS T.O. T.O. 13:53 01:19
COVERAGE 00:02 00:02 05:34 07:57
CVE-2014-3467_3 RANDOM T.O. T.O. 08:33 12:10
DFS T.O. T.O. 00:06 00:05
COVERAGE T.O. T.O. 12:24 06:29
CVE-2015-2806 RANDOM 02:16 01:34 05:00 01:39
DFS T.O. 10:45 T.O. 10:56
COVERAGE 01:24 00:52 01:36 00:51
CVE-2015-3622 RANDOM 00:15 00:16 00:14 00:13
DFS T.O. 05:47 12:42 05:40
COVERAGE 00:12 00:13 00:10 00:17

Updated 2020-01-30
@jordr
Copy link
Owner Author

jordr commented Jan 13, 2020 •
edited
Loading

Experiments with Legacy Chopper on tamworth

The published results

The results of the original white paper

image

The unpublished PSPA results

https://www.doc.ic.ac.uk/~jruiz/resources/pspa-results.png

Summary of experiments on tamworth

Vulnerability Search KLEE Chopper KLEE (ICSE'18) Chopper (ICSE'18)
CVE-2012-1569 RANDOM 00:04:35 00:01:59 OOM 02:27
DFS 00:01:49 00:00:16 OOM 03:29
COVERAGE 00:04:55 00:02:11 OOM 02:45
CVE-2014-3467-1 RANDOM 20:37:49 00:09:12 00:05 00:45
DFS 00:02:53 00:00:09 16:31 00:08
COVERAGE 00:00:02 00:03:49 00:03 00:58
CVE-2014-3467-2 RANDOM 00:00:02 00:11:15 01:02:13 06:18
DFS 00:02:52 00:02:47 T/O 00:09
COVERAGE 00:00:02 00:14:11 01:33:56 02:48
CVE-2014-3467-3 RANDOM 03:58:34 00:12:28 T/O 09:55
DFS 00:02:52 00:00:18 T/O 12:31
COVERAGE 00:00:03 00:05:39 T/O 09:50
CVE-2015-2806 RANDOM 00:01:08 00:03:52 01:07:46 02:18
DFS 09:20:05 00:43:50 02:46:13 12:04
COVERAGE 00:01:01 00:01:20 OOM 01:02
CVE-2015-3622 RANDOM 12:19:21 00:00:23 T/O 00:16
DFS 10:39:49 00:16:52 T/O 18:41
COVERAGE 05:57:26 00:00:20 20:25:20 00:18

Detailed results

CVE-2012-1569

Experimental results of CVE-2012-1569

EXPERIMENT COVERAGE DFS RANDOM
KLEE 00:04:55 00:01:49 00:04:35
STANDARD 00:02:11 00:00:16 00:01:59
SPLIT-SEARCH 10 00:02:14 00:00:12 00:02:16
SPLIT-SEARCH 20 00:01:53 00:00:15 00:02:27
SPLIT-SEARCH 30 00:02:44 00:00:22 00:03:20
SPLIT-SEARCH 40 00:04:09 00:00:32 00:03:52
SPLIT-SEARCH 50 00:04:04 00:00:49 00:05:50
RECOVERY-SEARCH-RP 10 00:05:27 00:00:38 00:05:01
RECOVERY-SEARCH-RP 20 00:04:33 00:01:06 00:06:27
RECOVERY-SEARCH-RP 30 00:06:07 00:03:06 00:05:12
RECOVERY-SEARCH-RP 40 00:10:54 00:11:54 00:08:17
RECOVERY-SEARCH-RP 50 00:12:56 00:21:27 00:15:11
RECOVERY-SEARCH-DFS 10 00:02:53 00:00:25 00:02:51
RECOVERY-SEARCH-DFS 20 00:03:49 00:00:46 00:04:41
RECOVERY-SEARCH-DFS 30 00:04:04 00:02:15 00:05:24
RECOVERY-SEARCH-DFS 40 00:05:25 00:10:54 00:09:11
RECOVERY-SEARCH-DFS 50 00:11:38 00:21:35 00:14:06

CVE-2014-3467 (out of date)

Experimental results of CVE-2014-3467_1

EXPERIMENT COVERAGE DFS RANDOM
KLEE 00:00:02 00:03:09
STANDARD 00:04:08 00:00:11 00:10:51
SPLIT-SEARCH 10 00:03:57 00:00:07 00:07:42
SPLIT-SEARCH 20 00:04:27 00:00:11 00:11:15
SPLIT-SEARCH 30 00:05:18 00:00:13 00:14:51
SPLIT-SEARCH 40 00:05:40 00:00:18 00:20:35
SPLIT-SEARCH 50 00:06:22 00:00:25 TO:TO:TO
RECOVERY-SEARCH-RP 10 00:09:14 00:00:43 TO:TO:TO
RECOVERY-SEARCH-RP 20 TO:TO:TO 00:12:15 TO:TO:TO
RECOVERY-SEARCH-RP 30 TO:TO:TO 00:13:23 TO:TO:TO
RECOVERY-SEARCH-RP 40 TO:TO:TO 00:13:18 TO:TO:TO
RECOVERY-SEARCH-RP 50 TO:TO:TO 00:13:16 TO:TO:TO
RECOVERY-SEARCH-DFS 10 00:04:26 00:00:48 TO:TO:TO
RECOVERY-SEARCH-DFS 20 TO:TO:TO 00:09:55 TO:TO:TO
RECOVERY-SEARCH-DFS 30 TO:TO:TO 00:13:07 TO:TO:TO
RECOVERY-SEARCH-DFS 40 TO:TO:TO 00:12:59 TO:TO:TO
RECOVERY-SEARCH-DFS 50 TO:TO:TO 00:12:57 TO:TO:TO

Experimental results of CVE-2014-3467_2

EXPERIMENT COVERAGE DFS RANDOM
KLEE 00:00:02 00:03:10 00:00:02
STANDARD 00:14:15 00:03:10 00:13:12
SPLIT-SEARCH 10 00:11:13 00:16:01 00:08:50
SPLIT-SEARCH 20 TO:TO:TO 00:03:14 00:12:24
SPLIT-SEARCH 30 TO:TO:TO 00:05:01 00:18:18
SPLIT-SEARCH 40 TO:TO:TO 00:07:12 00:24:20
SPLIT-SEARCH 50 TO:TO:TO TO:TO:TO TO:TO:TO
RECOVERY-SEARCH-RP 10 00:00:04 00:09:10 00:00:08
RECOVERY-SEARCH-RP 20 00:00:05 00:13:59 00:00:04
RECOVERY-SEARCH-RP 30 00:00:10 00:13:53 00:00:04
RECOVERY-SEARCH-RP 40 00:00:12 TO:TO:TO 00:00:04
RECOVERY-SEARCH-RP 50 00:00:05 TO:TO:TO 00:00:05
RECOVERY-SEARCH-DFS 10 00:00:24 TO:TO:TO 00:00:06
RECOVERY-SEARCH-DFS 20 00:00:06 TO:TO:TO 00:00:06
RECOVERY-SEARCH-DFS 30 00:16:10 TO:TO:TO 00:00:06
RECOVERY-SEARCH-DFS 40 00:00:05 TO:TO:TO 00:00:06
RECOVERY-SEARCH-DFS 50 00:00:07 TO:TO:TO 00:00:07

Experimental results of CVE-2014-3467_3

EXPERIMENT COVERAGE DFS RANDOM
KLEE 00:00:02 00:03:10 04:35:27
STANDARD 00:05:47 00:00:19 00:14:28
SPLIT-SEARCH 10 00:05:33 00:00:13 00:12:48
SPLIT-SEARCH 20 00:06:09 00:00:21 00:16:03
SPLIT-SEARCH 30 00:06:55 00:00:32 00:13:54
SPLIT-SEARCH 40 00:07:41 00:00:45 00:07:44
SPLIT-SEARCH 50 00:05:36 00:01:04 00:04:17
RECOVERY-SEARCH-RP 10 00:02:44 00:04:59 00:03:20
RECOVERY-SEARCH-RP 20 00:02:27 TO:TO:TO 00:16:29
RECOVERY-SEARCH-RP 30 TO:TO:TO TO:TO:TO TO:TO:TO
RECOVERY-SEARCH-RP 40 TO:TO:TO TO:TO:TO TO:TO:TO
RECOVERY-SEARCH-RP 50 TO:TO:TO TO:TO:TO TO:TO:TO
RECOVERY-SEARCH-DFS 10 00:06:15 TO:TO:TO TO:TO:TO
RECOVERY-SEARCH-DFS 20 TO:TO:TO TO:TO:TO TO:TO:TO
RECOVERY-SEARCH-DFS 30 TO:TO:TO TO:TO:TO TO:TO:TO
RECOVERY-SEARCH-DFS 40 TO:TO:TO TO:TO:TO TO:TO:TO
RECOVERY-SEARCH-DFS 50 TO:TO:TO TO:TO:TO TO:TO:TO

CVE-2015-2806

Experimental results of CVE-2015-2806

EXPERIMENT COVERAGE DFS RANDOM
KLEE 00:01:01 09:20:05 00:01:08
STANDARD 00:01:20 00:43:50 00:03:52
SPLIT-SEARCH 10 00:01:21 00:43:45 00:04:38
SPLIT-SEARCH 20 00:01:21 00:43:48 00:04:29
SPLIT-SEARCH 30 00:01:21 00:43:56 00:04:26
SPLIT-SEARCH 40 00:01:21 00:43:51 00:04:36
SPLIT-SEARCH 50 00:01:21 00:43:44 00:04:18
RECOVERY-SEARCH-RP 10 00:01:21 00:43:58 00:04:30
RECOVERY-SEARCH-RP 20 00:01:20 00:43:57 00:04:32
RECOVERY-SEARCH-RP 30 00:01:22 00:44:06 00:04:30
RECOVERY-SEARCH-RP 40 00:01:31 00:46:11 00:04:49
RECOVERY-SEARCH-RP 50 00:01:32 00:46:04 00:04:50
RECOVERY-SEARCH-DFS 10 00:01:35 00:46:37 00:05:15
RECOVERY-SEARCH-DFS 20 00:01:31 00:45:47 00:04:56
RECOVERY-SEARCH-DFS 30 00:01:29 00:45:47 00:04:51
RECOVERY-SEARCH-DFS 40 00:01:29 00:46:04 00:04:48
RECOVERY-SEARCH-DFS 50 00:01:30 T/O 00:04:55

CVE-2015-3622

Experimental results of CVE-2015-3622

EXPERIMENT COVERAGE DFS RANDOM
KLEE 05:57:26 10:39:49 12:19:21
STANDARD 00:00:20 00:16:52 00:00:23
SPLIT-SEARCH 10 00:00:25 00:16:02 00:00:24
SPLIT-SEARCH 20 00:00:23 00:16:47 00:00:27
SPLIT-SEARCH 30 00:00:17 00:17:21 00:00:29
SPLIT-SEARCH 40 00:00:21 00:18:02 00:00:26
SPLIT-SEARCH 50 00:00:23 00:17:37 00:00:17
RECOVERY-SEARCH-RP 10 00:00:32 00:16:20 00:00:19
RECOVERY-SEARCH-RP 20 00:00:17 00:17:36 00:00:20
RECOVERY-SEARCH-RP 30 00:00:17 00:17:37 00:00:21
RECOVERY-SEARCH-RP 40 00:00:26 00:16:50 00:00:20
RECOVERY-SEARCH-RP 50 00:00:19 00:16:48 00:00:21
RECOVERY-SEARCH-DFS 10 00:00:39 00:15:20 00:00:20
RECOVERY-SEARCH-DFS 20 00:00:20 00:16:33 00:00:20
RECOVERY-SEARCH-DFS 30 00:00:17 00:16:36 00:00:20
RECOVERY-SEARCH-DFS 40 00:00:19 00:16:32 00:00:19
RECOVERY-SEARCH-DFS 50 00:00:18 00:16:04 00:00:18

@jordr jordr closed this as completed Jan 13, 2020
@jordr jordr reopened this Jan 14, 2020
@jordr
Copy link
Owner Author

jordr commented Jan 14, 2020

andreamattavelli/chopper-experiments#1 fixes one issue with print_results.sh but there is another issue that prevents proper printing on CVE-2014-3467

@jordr
Copy link
Owner Author

jordr commented Jan 14, 2020

Fixed in andreamattavelli/chopper-experiments#2; updated the experiments above (#20 (comment))

@jordr
Copy link
Owner Author

jordr commented Jan 15, 2020 •
edited
Loading

Example of issue with CVE-2015-2806/cse-no-searcher/cse-run-coverage. The time looks off: 00:01:01 but expected OOM (38:56). We get:

KLEE: ERROR: /home/ubuntu/code/chopper-experiments/libtasn1/CVE-2015-2806/libtasn1-4.3//lib/parser_aux.c:574: memory error: out of bound pointer

Is this really the expected error? There is no --error-location= flag in the CVE-2015-2806 Makefile, unlike others, e.g. CVE-2012-1569:

@jordr jordr pinned this issue Jan 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
benchmark
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jordr