Go analysis support for CodeQL.

Author: Max Schaefer

.codeqlmanifest.json

@@ -0,0 +1,3 @@
+{ "provide": [ "ql/src/qlpack.yml",
+               "ql/config/legacy-support/qlpack.yml" ],
+  "ignore": [ "the-extractor-which-needs-to-be-built" ] }

.gitignore

@@ -0,0 +1,21 @@
+# editor and OS artifacts
+*~
+.DS_STORE
+
+# query compilation caches
+.cache
+
+# build artifacts
+build/*
+
+# qltest projects and artifacts
+ql/test/**/*.testproj
+ql/test/**/*.actual
+ql/test/**/go.sum
+
+# Java class files
+**/*.class
+
+# binaries
+tools/bin
+tools/tokenizer.jar

CODE_OF_CONDUCT.md

@@ -0,0 +1,76 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to make participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+  advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies within all project spaces, and it also applies when
+an individual is representing the project or its community in public spaces.
+Examples of representing a project or community include using an official
+project e-mail address, posting via an official social media account, or acting
+as an appointed representative at an online or offline event. Representation of
+a project may be further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at [email protected]. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq

CONTRIBUTING.md

@@ -0,0 +1,44 @@
+## Contributing
+
+Hi there! We're thrilled that you'd like to contribute to this project. Your help is essential for keeping it great.
+
+Contributions to this project are [released](https://help.github.com/articles/github-terms-of-service/#6-contributions-under-repository-license) to the public under the [project's open source license](LICENSE).
+
+Please note that this project is released with a [Contributor Code of Conduct][CODE_OF_CONDUCT.md]. By participating in this project you agree to abide by its terms.
+
+## Adding a new query
+
+If you have an idea for a query that you would like to share with other CodeQL users, please open a pull request to add it to this repository. 
+Follow the steps below to help other users understand what your query does, and to ensure that your query is consistent with the other CodeQL queries.
+
+1. **Consult the documentation for query writers**
+
+   There is lots of useful documentation to help you write CodeQL queries, ranging from information about query file structure to language-specific tutorials. For more information on the documentation available, see [Writing QL queries](https://help.semmle.com/QL/learn-ql/writing-queries/writing-queries.html) on [help.semmle.com](https://help.semmle.com).
+
+2. **Format your code correctly**
+
+   All of the standard CodeQL queries and libraries are uniformly formatted for clarity and consistency, so we strongly recommend that all contributions follow the same formatting guidelines. If you use the CodeQL extension for Visual Studio Code, you can auto-format your query in the [QL editor](https://help.semmle.com/ql-for-eclipse/Content/WebHelp/ql-editor.html). For more information, see the [QL style guide](https://github.com/Semmle/ql/blob/master/docs/ql-style-guide.md).
+
+3. **Make sure your query has the correct metadata**
+
+   Query metadata is used by Semmle's analysis to identify your query and make sure the query results are displayed properly. 
+   The most important metadata to include are the `@name`, `@description`, and the `@kind`.
+   Other metadata properties (`@precision`, `@severity`, and `@tags`) are usually added after the query has been reviewed by the maintainers.
+   For more information on writing query metadata, see the [Query metadata style guide](https://github.com/Semmle/ql/blob/master/docs/query-metadata-style-guide.md).
+
+4. **Make sure the `select` statement is compatible with the query type**
+
+   The `select` statement of your query must be compatible with the query type (determined by the `@kind` metadata property) for alert or path results to be displayed correctly in LGTM and Visual Studio Code.
+   For more information on `select` statement format, see [Introduction to query files](https://help.semmle.com/QL/learn-ql/writing-queries/introduction-to-queries.html#select-clause) on help.semmle.com.
+
+5. **Write a query help file**
+
+   Query help files explain the purpose of your query to other users. Write your query help in a `.qhelp` file and save it in the same directory as your new query. 
+   For more information on writing query help, see the [Query help style guide](https://github.com/Semmle/ql/blob/master/docs/query-help-style-guide.md).
+
+## Resources
+
+- [How to Contribute to Open Source](https://opensource.guide/how-to-contribute/)
+- [Using Pull Requests](https://help.github.com/articles/about-pull-requests/)
+- [GitHub Help](https://help.github.com)
+- [A Note About Git Commit Messages](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html)

COPYRIGHT

@@ -0,0 +1,13 @@
+Copyright (c) Semmle Inc and other contributors. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use
+this file except in compliance with the License. You may obtain a copy of the
+License at http://www.apache.org/licenses/LICENSE-2.0
+
+THIS CODE IS PROVIDED ON AN *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY IMPLIED
+WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE,
+MERCHANTABLITY OR NON-INFRINGEMENT.
+
+See the Apache Version 2.0 License for specific language governing permissions
+and limitations under the License.

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2019 GitHub
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

Makefile

@@ -0,0 +1,61 @@
+all: tools ql/src/go.dbscheme
+
+ifeq ($(OS),Windows_NT)
+EXE = .exe
+else
+EXE =
+endif
+
+.PHONY: tools
+tools: tools/bin/go-extractor$(EXE) tools/bin/go-tokenizer$(EXE) tools/bin/go-autobuilder$(EXE) tools/tokenizer.jar tools/bin/go-bootstrap$(EXE)
+
+tools/bin/go-extractor$(EXE): FORCE
+	go build -mod=vendor -o $@ ./extractor/cli/go-extractor
+
+tools/bin/go-tokenizer$(EXE): FORCE
+	go build -mod=vendor -o $@ ./extractor/cli/go-tokenizer
+
+tools/bin/go-autobuilder$(EXE): FORCE
+	go build -mod=vendor -o $@ ./extractor/cli/go-autobuilder
+
+tools/bin/go-bootstrap$(EXE): FORCE
+	go build -mod=vendor -o $@ ./extractor/cli/go-bootstrap
+
+FORCE:
+
+tools/tokenizer.jar: tools/net/sourceforge/pmd/cpd/GoLanguage.class
+	jar cf $@ -C tools net
+	jar uf $@ -C tools opencsv
+
+tools/net/sourceforge/pmd/cpd/GoLanguage.class: extractor/net/sourceforge/pmd/cpd/GoLanguage.java
+	javac -cp extractor -d tools $^
+	rm tools/net/sourceforge/pmd/cpd/AbstractLanguage.class
+	rm tools/net/sourceforge/pmd/cpd/SourceCode.class
+	rm tools/net/sourceforge/pmd/cpd/TokenEntry.class
+	rm tools/net/sourceforge/pmd/cpd/Tokenizer.class
+
+ql/src/go.dbscheme: tools/bin/go-extractor$(EXE)
+	env TRAP_FOLDER=/tmp tools/bin/go-extractor --dbscheme $@
+
+ql/src/go.dbscheme.stats: ql/src/go.dbscheme
+	odasa createProject --force --template templates/project --threads 4 \
+		--variable repository https://github.com/golang/tools \
+		--variable revision 6e04913c \
+		--variable SEMMLE_REPO_URL golang.org/x/tools \
+		build/stats-project
+	odasa addSnapshot --latest --overwrite --name revision --project build/stats-project
+	odasa buildSnapshot --latest --project build/stats-project
+	odasa collectStats --dbscheme $^ --db build/stats-project/revision/working/db-go --outputFile $@
+
+test: all build/testdb/check-upgrade-path
+	odasa qltest --language go --library ql/src ql/test
+	cd extractor; go test -mod=vendor ./... | grep -vF "[no test files]"
+
+.PHONY: build/testdb/check-upgrade-path
+build/testdb/check-upgrade-path : build/testdb/go.dbscheme ql/src/go.dbscheme
+	odasa upgradeDatabase --db build/testdb --upgrade-packs upgrades
+	diff -q build/testdb/go.dbscheme ql/src/go.dbscheme
+
+build/testdb/go.dbscheme: upgrades/initial/go.dbscheme
+	echo >build/empty.trap
+	odasa cli --dbscheme upgrades/initial/go.dbscheme --import build/empty.trap --db build/testdb

README.md

@@ -0,0 +1,45 @@
+# Go analysis support for CodeQL
+
+This open-source repository contains the extractor, CodeQL libraries, and queries that power Go
+support in [LGTM](https://lgtm.com), CodeQL, and other Semmle products.
+
+It contains two major components:
+  - an extractor, itself written in Go, that parses Go source code and converts it into a database
+    that can be queried using CodeQL.
+  - static analysis libraries and queries written in [QL](https://help.semmle.com/QL) that can be
+    used to analyze such a database to find coding mistakes or security vulnerabilities.
+
+The goal of this project is to provide comprehensive static analysis support for Go in CodeQL.
+
+## Installation
+
+Simply clone this repository. There are no external dependencies.
+
+If you want to use the CodeQL extension for Visual Studio Code, import this repository into your VS
+Code workspace.
+
+## Usage
+
+To analyze a Go codebase, either use the CodeQL command-line interface to create a database
+yourself, or download a pre-built database from LGTM.com. You can then run any of the queries
+contained in this repository either on the command line or using the VS Code extension.
+
+Note that the [lgtm.com](https://github.com/github/codeql-go/tree/lgtm.com) branch of this
+repository corresponds to the version of the queries that is currently deployed on LGTM.com.
+The [master](https://github.com/github/codeql-go/tree/master) branch may contain changes that
+have not been deployed yet, so you may need to upgrade databases downloaded from LGTM.com before
+running queries on them.
+
+## Contributions
+
+Contributions are welcome! Please see our [contribution guidelines](CONTRIBUTING.md) and our
+[code of conduct](CODE_OF_CONDUCT.md) for details on how to participate in our community.
+
+## Licensing
+
+The code in this repository is licensed under the [MIT license](LICENSE).
+
+## Resources
+
+- [Writing CodeQL queries](https://help.semmle.com/QL/learn-ql/ql/writing-queries/writing-queries.html)
+- [Learning CodeQL](https://help.semmle.com/QL/learn-ql/index.html)

SECURITY.md

@@ -0,0 +1,3 @@
+If you discover a security issue in this repo, please submit it through the [GitHub Security Bug Bounty](https://hackerone.com/github).
+
+Thanks for helping make CodeQL safe for everyone.

alert_weighting.properties

@@ -0,0 +1,3 @@
+precision = ("veryhigh", "high", "medium", "low")
+severity = ("error", "warning", "recommendation")
+security = ("true", "false")

build/.gitkeep

@@ -0,0 +1,16 @@
+name: "go"
+display_name: "Go"
+version: 0.1.0
+pull_request_triggers:
+  - "**/go.mod"
+  - "**/glide.yaml"
+  - "**/Gopkg.toml"
+column_kind: "utf8"
+extra_env_vars:
+  SOURCE_ARCHIVE: ${env.CODEQL_EXTRACTOR_GO_SOURCE_ARCHIVE_DIR}
+  TRAP_FOLDER: ${env.CODEQL_EXTRACTOR_GO_TRAP_DIR}
+file_types:
+  - name: go
+    display_name: Go
+    extensions:
+      - .go

codeql-extractor.yml

@@ -0,0 +1,10 @@
+@echo off
+SETLOCAL EnableDelayedExpansion
+
+rem Some legacy environment variables for the autobuilder.
+set LGTM_SRC=%CD%
+
+type NUL && "%CODEQL_EXTRACTOR_GO_ROOT%/tools/%CODEQL_PLATFORM%/go-autobuilder.exe"
+exit /b %ERRORLEVEL%
+
+ENDLOCAL

codeql-tools/autobuild.cmd

@@ -0,0 +1,14 @@
+#!/bin/sh
+
+set -eu
+
+if [ "$CODEQL_PLATFORM" != "linux64" ] && [ "$CODEQL_PLATFORM" != "osx64" ] ; then
+    echo "Automatic build detection for $CODEQL_PLATFORM is not implemented."
+    exit 1
+fi
+
+# Some legacy environment variables used by the autobuilder.
+LGTM_SRC="$(pwd)"
+export LGTM_SRC
+
+"$CODEQL_EXTRACTOR_GO_ROOT/tools/$CODEQL_PLATFORM/go-autobuilder"