diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..c07aa32 --- /dev/null +++ b/.gitignore @@ -0,0 +1,33 @@ +# Compiled class file +*.class + +# Log file +*.log + +# BlueJ files +*.ctxt + +# Mobile Tools for Java (J2ME) +.mtj.tmp/ + +# Package Files # +*.jar +*.war +*.nar +*.ear +*.zip +*.tar.gz +*.rar + +# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml +hs_err_pid* + +# Gradle +build/ +.gradle/ + +# IntelliJ +.idea + +# Vim +*.swp diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md new file mode 100644 index 0000000..ca47b3a --- /dev/null +++ b/CODE_OF_CONDUCT.md @@ -0,0 +1,133 @@ + +# Contributor Covenant Code of Conduct + +## Our Pledge + +We as members, contributors, and leaders pledge to make participation in our +community a harassment-free experience for everyone, regardless of age, body +size, visible or invisible disability, ethnicity, sex characteristics, gender +identity and expression, level of experience, education, socio-economic status, +nationality, personal appearance, race, caste, color, religion, or sexual identity +and orientation. + +We pledge to act and interact in ways that contribute to an open, welcoming, +diverse, inclusive, and healthy community. + +## Our Standards + +Examples of behavior that contributes to a positive environment for our +community include: + +* Demonstrating empathy and kindness toward other people +* Being respectful of differing opinions, viewpoints, and experiences +* Giving and gracefully accepting constructive feedback +* Accepting responsibility and apologizing to those affected by our mistakes, + and learning from the experience +* Focusing on what is best not just for us as individuals, but for the + overall community + +Examples of unacceptable behavior include: + +* The use of sexualized language or imagery, and sexual attention or + advances of any kind +* Trolling, insulting or derogatory comments, and personal or political attacks +* Public or private harassment +* Publishing others' private information, such as a physical or email + address, without their explicit permission +* Other conduct which could reasonably be considered inappropriate in a + professional setting + +## Enforcement Responsibilities + +Community leaders are responsible for clarifying and enforcing our standards of +acceptable behavior and will take appropriate and fair corrective action in +response to any behavior that they deem inappropriate, threatening, offensive, +or harmful. + +Community leaders have the right and responsibility to remove, edit, or reject +comments, commits, code, wiki edits, issues, and other contributions that are +not aligned to this Code of Conduct, and will communicate reasons for moderation +decisions when appropriate. + +## Scope + +This Code of Conduct applies within all community spaces, and also applies when +an individual is officially representing the community in public spaces. +Examples of representing our community include using an official e-mail address, +posting via an official social media account, or acting as an appointed +representative at an online or offline event. + +## Enforcement + +Instances of abusive, harassing, or otherwise unacceptable behavior may be +reported to the community leaders responsible for enforcement at +application-security@schibsted.com. +All complaints will be reviewed and investigated promptly and fairly. + +All community leaders are obligated to respect the privacy and security of the +reporter of any incident. + +## Enforcement Guidelines + +Community leaders will follow these Community Impact Guidelines in determining +the consequences for any action they deem in violation of this Code of Conduct: + +### 1. Correction + +**Community Impact**: Use of inappropriate language or other behavior deemed +unprofessional or unwelcome in the community. + +**Consequence**: A private, written warning from community leaders, providing +clarity around the nature of the violation and an explanation of why the +behavior was inappropriate. A public apology may be requested. + +### 2. Warning + +**Community Impact**: A violation through a single incident or series +of actions. + +**Consequence**: A warning with consequences for continued behavior. No +interaction with the people involved, including unsolicited interaction with +those enforcing the Code of Conduct, for a specified period of time. This +includes avoiding interactions in community spaces as well as external channels +like social media. Violating these terms may lead to a temporary or +permanent ban. + +### 3. Temporary Ban + +**Community Impact**: A serious violation of community standards, including +sustained inappropriate behavior. + +**Consequence**: A temporary ban from any sort of interaction or public +communication with the community for a specified period of time. No public or +private interaction with the people involved, including unsolicited interaction +with those enforcing the Code of Conduct, is allowed during this period. +Violating these terms may lead to a permanent ban. + +### 4. Permanent Ban + +**Community Impact**: Demonstrating a pattern of violation of community +standards, including sustained inappropriate behavior, harassment of an +individual, or aggression toward or disparagement of classes of individuals. + +**Consequence**: A permanent ban from any sort of public interaction within +the community. + +## Attribution + +This Code of Conduct is adapted from the [Contributor Covenant][homepage], +version 2.0, available at +[https://www.contributor-covenant.org/version/2/0/code_of_conduct.html][v2.0]. + +Community Impact Guidelines were inspired by +[Mozilla's code of conduct enforcement ladder][Mozilla CoC]. + +For answers to common questions about this code of conduct, see the FAQ at +[https://www.contributor-covenant.org/faq][FAQ]. Translations are available +at [https://www.contributor-covenant.org/translations][translations]. + +[homepage]: https://www.contributor-covenant.org +[v2.0]: https://www.contributor-covenant.org/version/2/0/code_of_conduct.html +[Mozilla CoC]: https://github.com/mozilla/diversity +[FAQ]: https://www.contributor-covenant.org/faq +[translations]: https://www.contributor-covenant.org/translations diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..6c3372c --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,18 @@ +# Contributing to Artishock + +Thanks for taking the time to contribute! :+1::tada: + +## Code of Conduct +The project is governed by the [Contributor Covenant Code of Conduct](CODE_OF_CONDUCT.md). + +## License +Artishock is released under the [Apache 2.0 License](LICENSE). Any code you submit will be released under the same license, as per section 5 in the [Apache 2.0 License](LICENSE): +``` + Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. +``` diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..d645695 --- /dev/null +++ b/LICENSE @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/NOTICE b/NOTICE new file mode 100644 index 0000000..ea55b6a --- /dev/null +++ b/NOTICE @@ -0,0 +1,150 @@ +################################################################################ +# # +# Artishock # +# # +# Copyright 2021 Schibsted # +# # +# Unless required by applicable law or agreed to in writing, software # +# distributed under the License is distributed on an "AS IS" BASIS, # +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # +# # +# See the License for the specific language governing permissions and # +# limitations under the License. # +# # +################################################################################ + +The platform specific distributions are bundled with parts of the AdoptOpenJDk from +https://github.com/AdoptOpenJDK/openjdk15-binaries. + +Futher project dependencies are included as JARs. + +See the `legal` directory as well as the individual JARs in the `lib` directory for more information. + +OpenJDK + LICENSE: GPL 2.0 with Class-path Exception, + https://github.com/openjdk/ + + The Mozilla Elliptic Curve Crypotography library is used + LICENSE: LGPL 2.1 + +Groovy + LICENSE: Apache 2.0 + NOTICE: https://github.com/apache/groovy/blob/master/NOTICE + +Kotlin + LICENSE: Apache 2.0 + NOTICE: https://github.com/JetBrains/kotlin/tree/master/license + +------- + +Airline + LICENSE: Apache 2.0, + https://github.com/airlift/airline/ + +Apache Commons Lang + LICENSE: Apache 2.0 + NOTICE: https://github.com/apache/commons-lang/blob/master/NOTICE.txt + +Apache Log4j + LICENSE: Apache 2.0 + NOTICE: https://github.com/apache/log4j/blob/trunk/NOTICE + +Apache HttpComponents Core + LICENSE: Apache 2.0, + NOTICE: https://github.com/apache/httpcomponents-core/blob/master/NOTICE.txt + +Apache HttpComponents Client + LICENSE: Apache 2.0, + NOTICE: https://github.com/apache/httpcomponents-client/blob/master/NOTICE.txt + +Apache Commons Codec + LICENSE: Apache 2.0, + NOTICE: https://github.com/apache/commons-codec/blob/master/NOTICE.txt + +Apache Commons Lang + LICENSE: Apache 2.0, + NOTICE: https://github.com/apache/commons-lang/blob/master/NOTICE.txt + +Apache Commons IO + LICENSE: Apache 2.0 + NOTICE: https://github.com/apache/commons-io/blob/master/NOTICE.txt + +FasterXML Jackson Core + LICENSE: Apache 2.0, + https://github.com/FasterXML/jackson-core + +FasterXML Jackson Annotations + LICENSE: Apache 2.0, + https://github.com/FasterXML/jackson-annotations + +FasterXML Jackson Databind + LICENSE: Apache 2.0, + https://github.com/FasterXML/jackson-databind + +FasterXML Jackson Datatype JDK8 + LICENSE: Apache 2.0, + https://github.com/FasterXML/jackson-modules-java8 + +FasterXML Jackson Dataformat XML + LICENSE: Apache 2.0, + https://github.com/FasterXML/jackson-dataformat-xml + +FasterXML Jackson BOM + LICSENS: Apache 2.0, + https://github.com/FasterXML/jackson-bom + +FasterXML Jackson JAXB Annotations + LICENSE: Apache 2.0, + https://github.com/FasterXML/jackson-modules-base + +FasterXML Woodstox + LICENSE: Apache 2.0, + https://github.com/FasterXML/woodstox + +Google Guava + LICENSE: Apache 2.0, + https://github.com/google/guava/ + +Google J2ObjC + LICENSE: Apache 2.0, + https://github.com/google/j2objc + +Google Findbugs JSR-305 + LICENSE: 2-Clause BSD, + https://github.com/findbugsproject/findbugs/blob/master/findbugs/licenses/LICENSE-jsr305.txt + +Google Error Prone + LICENES: Apache 2.0, + https://github.com/google/error-prone + +Jakarta XML Binding + LICENSE: EDL 1.0 + NOTICE: https://github.com/eclipse-ee4j/jaxb-api/blob/master/NOTICE.md + +Jakarta Activation + LICENSE: EDL 1.0 + NOTICE: https://github.com/eclipse-ee4j/jaf/blob/master/NOTICE.md + +javax.inject + LICENSE: Apache 2.0, + https://search.maven.org/artifact/javax.inject/javax.inject/1/jar + +Jfrog Artifactory Client + LICENSE: Apache 2.0, + https://github.com/jfrog/artifactory-client-java + +SLF4J + LICENSE: MIT, + https://github.com/qos-ch/slf4j/blob/master/LICENSE.txt + +Square OkHttp + LICENSE: Apache 2.0, + https://github.com/square/okhttp + +Square Okio + LICENSE: Apache 2.0, + https://github.com/square/okio + +Type Tools Checker Equal + LICENSE: MIT, + https://github.com/typetools/checker-framework/blob/master/checker-qual/LICENSE.txt diff --git a/README.md b/README.md new file mode 100644 index 0000000..4673a6c --- /dev/null +++ b/README.md @@ -0,0 +1,86 @@ +![](src/main/logo/artishock.svg) +# Artishock +A tool to investigate Dependency Confusion in Artifactory. + +## Install +Binaries for Linux, Mac and Windows can be found under [releases](https://github.com/schibsted/artishock/releases) (Windows is not tested). + +Unzip `artishock-{linux,mac,win}.zip`, then run `artishock-{linux,mac,win}/bin/artishock`. + +**For macOs**: `bin/artishock` and `bin/java` are not signed so they must be allowed to run. + +## Configure +Create `~/.artishock/artishock.config` with the following +``` +{ + "artifactoryUrl": "https://example.com/artifactory/", + "artifactoryUsername": "email@example.com", + "artifactoryPassword": "" +} +``` + +You can also set these as environment variables: +``` +export ARTISHOCK_ARTIFACTORY_URL= +export ARTISHOCK_ARTIFACTORY_USERNAME= +export ARTISHOCK_ARTIFACTORY_PASSWORD= +``` + +## Run + +**Please note that some Artishock commands will look up your internal package names upstream.** These require the `--query-upstream` flag. If you do not want to disclose your internal names don't use those commands. + +Some requests will be cached to `~/.artishock/cache/`. Each request is cached for 7 days. The cache directory can be deleted to clear the cache. +``` +artishock +artishock repo-ls --help +artishock repo-ls --json +``` + +## Examples + +### NPM +``` +artishock repo-ls --package-system npm +artishock exclude-candidates --package-system npm --local npm-local +artishock not-claimed --package-system npm --local npm-local --query-upstream +artishock cached --package-system npm --local npm-local --remote npm-remote +artishock inferred-exclude --package-system npm --local npm-local --remote npm-remote --query-upstream +artishock package-stats --package-system npm --repo npm-local --package @example/package +``` + +### PyPi +``` +artishock repo-ls --package-system pypi +artishock exclude-candidates --package-system pypi --local pypi-local +artishock not-claimed --package-system pypi --local pypi-local --query-upstream +artishock cached --package-system pypi --local pypi-local --remote pypi-remote +artishock inferred-exclude --package-system pypi --local pypi-local --remote pypi-remote --query-upstream +``` + +### Maven +This is a slow command that iterates over the whole repo to gather download stats +``` +artishock repo-stats --package-system maven --repo small-remote-cache +``` + +## Debugging +Use `--verbose` for verbose output and `--stacktrace` to get the full stack trace rather than just the message. + +## Developer notes +*Prerequisite: [OpenJDK 15](https://adoptopenjdk.net/?variant=openjdk15&jvmVariant=hotspot)* + +Generate runtime images `build/image/artishock-{linux,mac,win}/` +``` +./gradlew runtime +``` + +On Linux run the program with +``` +build/image/artishock-linux/bin/artishock +``` + +Make release files `/build/artishoc-{linux,mac,win}.zip` +``` +./gradlew runtimeZip +``` diff --git a/build.gradle b/build.gradle new file mode 100644 index 0000000..0930a4d --- /dev/null +++ b/build.gradle @@ -0,0 +1,106 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +plugins { + id('java') + id('application') + id 'org.beryx.runtime' version '1.12.1' +} + +repositories { + mavenCentral() + jcenter() +} + +mainClassName = 'com.schibsted.security.artishock.ArtishockCli' +def version = '0.0.1' + +dependencies { + implementation('org.apache.logging.log4j:log4j-core:2.13.1') + implementation('org.slf4j:slf4j-simple:1.7.30') + + implementation('com.google.guava:guava:30.0-jre') + implementation('com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.12.2') + implementation('com.fasterxml.jackson.dataformat:jackson-dataformat-xml:2.12.2') + implementation("com.squareup.okhttp3:okhttp:4.9.0") + implementation("io.airlift:airline:0.8") + + implementation("org.jfrog.artifactory.client:artifactory-java-client-services:2.8.6") + implementation("org.codehaus.groovy:groovy:3.0.6") // Force upgrade in artifactory client +} + +java { + sourceCompatibility = JavaVersion.VERSION_11 + targetCompatibility = JavaVersion.VERSION_11 +} + +runtime { + options = ['--strip-debug', '--compress', '2', '--no-header-files', '--no-man-pages'] + imageZip = file("$buildDir/artishock.zip") + additive = true + modules = ["jdk.crypto.ec"] + + jpackage { + skipInstaller = true + imageName = "artishock" + } + + targetPlatform("linux") { + jdkHome = jdkDownload("https://github.com/AdoptOpenJDK/openjdk15-binaries/releases/download/jdk-15.0.2%2B7/OpenJDK15U-jdk_x64_linux_hotspot_15.0.2_7.tar.gz") + } + + targetPlatform("win") { + jdkHome = jdkDownload("https://github.com/AdoptOpenJDK/openjdk15-binaries/releases/download/jdk-15.0.2%2B7/OpenJDK15U-jdk_x64_windows_hotspot_15.0.2_7.zip") + } + + targetPlatform("mac") { + jdkHome = jdkDownload("https://github.com/AdoptOpenJDK/openjdk15-binaries/releases/download/jdk-15.0.2%2B7/OpenJDK15U-jdk_x64_mac_hotspot_15.0.2_7.tar.gz") + } +} + +jar { + enabled = true + manifest { + attributes( + 'Bundle-License': 'https://www.apache.org/licenses/LICENSE-2.0.txt', + 'Implementation-Version': version, + 'Implementation-URL': 'https://github.com/schibsted/artishock' + ) + } +} + + +// Copy LICENSE and NOTICE into the JAR +tasks.withType(Jar) { + from(project.projectDir) { + include 'LICENSE', 'NOTICE' + into 'META-INF' + } +} + +// Copy LICENSE and NOTICE into the legal directory in each image +def license = "${project.projectDir}/LICENSE" +def notice = "${project.projectDir}/NOTICE" + +task copyToLinuxImage(type: Copy) { + dependsOn("runtime") + from license, notice + into "${project.buildDir}/image/artishock-linux/legal/" +} + +task copyToMacImage(type: Copy) { + dependsOn("runtime") + from license, notice + into "${project.buildDir}/image/artishock-mac/legal/" +} + +task copyToWinImage(type: Copy) { + dependsOn("runtime") + from license, notice + into "${project.buildDir}/image/artishock-win/legal/" +} + +runtimeZip.dependsOn('copyToLinuxImage') +runtimeZip.dependsOn('copyToMacImage') +runtimeZip.dependsOn('copyToWinImage') diff --git a/gradle/wrapper/gradle-wrapper.jar b/gradle/wrapper/gradle-wrapper.jar new file mode 100644 index 0000000..490fda8 Binary files /dev/null and b/gradle/wrapper/gradle-wrapper.jar differ diff --git a/gradle/wrapper/gradle-wrapper.properties b/gradle/wrapper/gradle-wrapper.properties new file mode 100644 index 0000000..8cf6eb5 --- /dev/null +++ b/gradle/wrapper/gradle-wrapper.properties @@ -0,0 +1,5 @@ +distributionBase=GRADLE_USER_HOME +distributionPath=wrapper/dists +distributionUrl=https\://services.gradle.org/distributions/gradle-6.8.3-all.zip +zipStoreBase=GRADLE_USER_HOME +zipStorePath=wrapper/dists diff --git a/gradlew b/gradlew new file mode 100755 index 0000000..2fe81a7 --- /dev/null +++ b/gradlew @@ -0,0 +1,183 @@ +#!/usr/bin/env sh + +# +# Copyright 2015 the original author or authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +############################################################################## +## +## Gradle start up script for UN*X +## +############################################################################## + +# Attempt to set APP_HOME +# Resolve links: $0 may be a link +PRG="$0" +# Need this for relative symlinks. +while [ -h "$PRG" ] ; do + ls=`ls -ld "$PRG"` + link=`expr "$ls" : '.*-> \(.*\)$'` + if expr "$link" : '/.*' > /dev/null; then + PRG="$link" + else + PRG=`dirname "$PRG"`"/$link" + fi +done +SAVED="`pwd`" +cd "`dirname \"$PRG\"`/" >/dev/null +APP_HOME="`pwd -P`" +cd "$SAVED" >/dev/null + +APP_NAME="Gradle" +APP_BASE_NAME=`basename "$0"` + +# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. +DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"' + +# Use the maximum available, or set MAX_FD != -1 to use that value. +MAX_FD="maximum" + +warn () { + echo "$*" +} + +die () { + echo + echo "$*" + echo + exit 1 +} + +# OS specific support (must be 'true' or 'false'). +cygwin=false +msys=false +darwin=false +nonstop=false +case "`uname`" in + CYGWIN* ) + cygwin=true + ;; + Darwin* ) + darwin=true + ;; + MINGW* ) + msys=true + ;; + NONSTOP* ) + nonstop=true + ;; +esac + +CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar + +# Determine the Java command to use to start the JVM. +if [ -n "$JAVA_HOME" ] ; then + if [ -x "$JAVA_HOME/jre/sh/java" ] ; then + # IBM's JDK on AIX uses strange locations for the executables + JAVACMD="$JAVA_HOME/jre/sh/java" + else + JAVACMD="$JAVA_HOME/bin/java" + fi + if [ ! -x "$JAVACMD" ] ; then + die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME + +Please set the JAVA_HOME variable in your environment to match the +location of your Java installation." + fi +else + JAVACMD="java" + which java >/dev/null 2>&1 || die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. + +Please set the JAVA_HOME variable in your environment to match the +location of your Java installation." +fi + +# Increase the maximum file descriptors if we can. +if [ "$cygwin" = "false" -a "$darwin" = "false" -a "$nonstop" = "false" ] ; then + MAX_FD_LIMIT=`ulimit -H -n` + if [ $? -eq 0 ] ; then + if [ "$MAX_FD" = "maximum" -o "$MAX_FD" = "max" ] ; then + MAX_FD="$MAX_FD_LIMIT" + fi + ulimit -n $MAX_FD + if [ $? -ne 0 ] ; then + warn "Could not set maximum file descriptor limit: $MAX_FD" + fi + else + warn "Could not query maximum file descriptor limit: $MAX_FD_LIMIT" + fi +fi + +# For Darwin, add options to specify how the application appears in the dock +if $darwin; then + GRADLE_OPTS="$GRADLE_OPTS \"-Xdock:name=$APP_NAME\" \"-Xdock:icon=$APP_HOME/media/gradle.icns\"" +fi + +# For Cygwin or MSYS, switch paths to Windows format before running java +if [ "$cygwin" = "true" -o "$msys" = "true" ] ; then + APP_HOME=`cygpath --path --mixed "$APP_HOME"` + CLASSPATH=`cygpath --path --mixed "$CLASSPATH"` + JAVACMD=`cygpath --unix "$JAVACMD"` + + # We build the pattern for arguments to be converted via cygpath + ROOTDIRSRAW=`find -L / -maxdepth 1 -mindepth 1 -type d 2>/dev/null` + SEP="" + for dir in $ROOTDIRSRAW ; do + ROOTDIRS="$ROOTDIRS$SEP$dir" + SEP="|" + done + OURCYGPATTERN="(^($ROOTDIRS))" + # Add a user-defined pattern to the cygpath arguments + if [ "$GRADLE_CYGPATTERN" != "" ] ; then + OURCYGPATTERN="$OURCYGPATTERN|($GRADLE_CYGPATTERN)" + fi + # Now convert the arguments - kludge to limit ourselves to /bin/sh + i=0 + for arg in "$@" ; do + CHECK=`echo "$arg"|egrep -c "$OURCYGPATTERN" -` + CHECK2=`echo "$arg"|egrep -c "^-"` ### Determine if an option + + if [ $CHECK -ne 0 ] && [ $CHECK2 -eq 0 ] ; then ### Added a condition + eval `echo args$i`=`cygpath --path --ignore --mixed "$arg"` + else + eval `echo args$i`="\"$arg\"" + fi + i=`expr $i + 1` + done + case $i in + 0) set -- ;; + 1) set -- "$args0" ;; + 2) set -- "$args0" "$args1" ;; + 3) set -- "$args0" "$args1" "$args2" ;; + 4) set -- "$args0" "$args1" "$args2" "$args3" ;; + 5) set -- "$args0" "$args1" "$args2" "$args3" "$args4" ;; + 6) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" ;; + 7) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" ;; + 8) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" ;; + 9) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" "$args8" ;; + esac +fi + +# Escape application args +save () { + for i do printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/" ; done + echo " " +} +APP_ARGS=`save "$@"` + +# Collect all arguments for the java command, following the shell quoting and substitution rules +eval set -- $DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS "\"-Dorg.gradle.appname=$APP_BASE_NAME\"" -classpath "\"$CLASSPATH\"" org.gradle.wrapper.GradleWrapperMain "$APP_ARGS" + +exec "$JAVACMD" "$@" diff --git a/gradlew.bat b/gradlew.bat new file mode 100644 index 0000000..9109989 --- /dev/null +++ b/gradlew.bat @@ -0,0 +1,103 @@ +@rem +@rem Copyright 2015 the original author or authors. +@rem +@rem Licensed under the Apache License, Version 2.0 (the "License"); +@rem you may not use this file except in compliance with the License. +@rem You may obtain a copy of the License at +@rem +@rem https://www.apache.org/licenses/LICENSE-2.0 +@rem +@rem Unless required by applicable law or agreed to in writing, software +@rem distributed under the License is distributed on an "AS IS" BASIS, +@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +@rem See the License for the specific language governing permissions and +@rem limitations under the License. +@rem + +@if "%DEBUG%" == "" @echo off +@rem ########################################################################## +@rem +@rem Gradle startup script for Windows +@rem +@rem ########################################################################## + +@rem Set local scope for the variables with windows NT shell +if "%OS%"=="Windows_NT" setlocal + +set DIRNAME=%~dp0 +if "%DIRNAME%" == "" set DIRNAME=. +set APP_BASE_NAME=%~n0 +set APP_HOME=%DIRNAME% + +@rem Resolve any "." and ".." in APP_HOME to make it shorter. +for %%i in ("%APP_HOME%") do set APP_HOME=%%~fi + +@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. +set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m" + +@rem Find java.exe +if defined JAVA_HOME goto findJavaFromJavaHome + +set JAVA_EXE=java.exe +%JAVA_EXE% -version >NUL 2>&1 +if "%ERRORLEVEL%" == "0" goto init + +echo. +echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. +echo. +echo Please set the JAVA_HOME variable in your environment to match the +echo location of your Java installation. + +goto fail + +:findJavaFromJavaHome +set JAVA_HOME=%JAVA_HOME:"=% +set JAVA_EXE=%JAVA_HOME%/bin/java.exe + +if exist "%JAVA_EXE%" goto init + +echo. +echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% +echo. +echo Please set the JAVA_HOME variable in your environment to match the +echo location of your Java installation. + +goto fail + +:init +@rem Get command-line arguments, handling Windows variants + +if not "%OS%" == "Windows_NT" goto win9xME_args + +:win9xME_args +@rem Slurp the command line arguments. +set CMD_LINE_ARGS= +set _SKIP=2 + +:win9xME_args_slurp +if "x%~1" == "x" goto execute + +set CMD_LINE_ARGS=%* + +:execute +@rem Setup the command line + +set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar + +@rem Execute Gradle +"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %CMD_LINE_ARGS% + +:end +@rem End local scope for the variables with windows NT shell +if "%ERRORLEVEL%"=="0" goto mainEnd + +:fail +rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of +rem the _cmd.exe /c_ return code! +if not "" == "%GRADLE_EXIT_CONSOLE%" exit 1 +exit /b 1 + +:mainEnd +if "%OS%"=="Windows_NT" endlocal + +:omega diff --git a/src/main/java/com/schibsted/security/artishock/ArtishockCli.java b/src/main/java/com/schibsted/security/artishock/ArtishockCli.java new file mode 100644 index 0000000..de18ddf --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/ArtishockCli.java @@ -0,0 +1,87 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock; + +import com.google.common.collect.ListMultimap; +import com.schibsted.security.artishock.cli.view.Commands; +import java.util.Arrays; +import java.util.Collection; +import java.util.Map; +import io.airlift.airline.Cli; +import io.airlift.airline.Help; +import io.airlift.airline.OptionType; +import io.airlift.airline.ParseArgumentsMissingException; +import io.airlift.airline.ParseArgumentsUnexpectedException; +import io.airlift.airline.ParseOptionMissingException; +import io.airlift.airline.Parser; +import io.airlift.airline.model.GlobalMetadata; +import io.airlift.airline.model.OptionMetadata; + +public class ArtishockCli { + public static void main(String[] args) { + java.lang.Object.class.getModule().addOpens(java.lang.Object.class.getPackageName(), org.codehaus.groovy.reflection.CachedClass.class.getModule()); + org.codehaus.groovy.reflection.CachedClass.class.getModule().addReads(java.lang.Object.class.getModule()); + + var builder = Cli.builder("artishock") + .withDefaultCommand(Commands.CustomHelp.class); + + builder.withCommand(Commands.CustomHelp.class); + builder.withCommand(Commands.CustomHelpAlternativeName.class); + builder.withCommand(Commands.Version.class); + builder.withCommand(Commands.VersionAlternativeName.class); + + builder.withCommand(Commands.RepoLs.class); + builder.withCommand(Commands.RepoStats.class); + builder.withCommand(Commands.PackageStats.class); + builder.withCommand(Commands.Cached.class); + builder.withCommand(Commands.InferredExclude.class); + builder.withCommand(Commands.NotClaimed.class); + builder.withCommand(Commands.ExcludeCandidates.class); + + var parser = builder.build(); + try { + parser.parse(args).run(); + } catch (ParseArgumentsUnexpectedException e) { + System.err.println(e.getMessage()); + System.err.println("Try '--help' instead"); + System.exit(1); + } catch (ParseArgumentsMissingException | ParseOptionMissingException e) { + if (flagIsSet("--help", parser.getMetadata(), args)) { + Help.help(parser.getMetadata(), Arrays.asList(args)); + } else { + System.err.println(e.getMessage()); + System.err.println("Try adding '--help'"); + } + System.exit(1); + } catch (Exception e) { + boolean stacktrace = flagIsSet("--stacktrace", parser.getMetadata(), args); + + if (!stacktrace) { + System.err.println(e.getMessage()); + System.exit(1); + } + throw e; + } + } + + private static boolean flagIsSet(String flag, final GlobalMetadata globalMetadata, final String[] args) { + try { + Parser p = new Parser(); + ListMultimap options = p.parse(globalMetadata, args).getParsedOptions(); + + for (Map.Entry> option : options.asMap().entrySet()) { + OptionMetadata metadata = option.getKey(); + + if (metadata.getOptionType() == OptionType.COMMAND && metadata.getOptions().contains(flag)) { + return option.getValue().contains(true); + } + } + } catch (Exception e) { + System.err.println("Failed to determine if the stacktrace should be shown."); + } + + return false; + } +} diff --git a/src/main/java/com/schibsted/security/artishock/artifactory/ArtifactoryClient.java b/src/main/java/com/schibsted/security/artishock/artifactory/ArtifactoryClient.java new file mode 100644 index 0000000..4ca65dc --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/artifactory/ArtifactoryClient.java @@ -0,0 +1,263 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.artifactory; + +import com.schibsted.security.artishock.npm.NpmPackageIdentifier; +import com.schibsted.security.artishock.config.Config; +import com.schibsted.security.artishock.pypi.PyPiPackageIdentifier; +import java.io.IOException; +import java.time.ZonedDateTime; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.HashMap; +import java.util.List; +import java.util.Locale; +import java.util.Optional; +import java.util.Set; +import java.util.stream.Collectors; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; +import org.jfrog.artifactory.client.Artifactory; +import org.jfrog.artifactory.client.ArtifactoryClientBuilder; +import org.jfrog.artifactory.client.impl.ArtifactoryImpl; +import org.jfrog.artifactory.client.model.Folder; +import org.jfrog.artifactory.client.model.Item; +import org.jfrog.artifactory.client.model.impl.RemoteRepositoryImpl; +import org.jfrog.artifactory.client.model.impl.RepositoryTypeImpl; +import org.jfrog.artifactory.client.model.impl.VirtualRepositoryImpl; +import org.jfrog.artifactory.client.model.repository.settings.ComposerRepositorySettings; +import org.jfrog.artifactory.client.model.repository.settings.GradleRepositorySettings; +import org.jfrog.artifactory.client.model.repository.settings.MavenRepositorySettings; +import org.jfrog.artifactory.client.model.repository.settings.NpmRepositorySettings; +import org.jfrog.artifactory.client.model.repository.settings.NugetRepositorySettings; +import org.jfrog.artifactory.client.model.repository.settings.PypiRepositorySettings; +import org.jfrog.artifactory.client.model.repository.settings.RepositorySettings; +import org.jfrog.artifactory.client.model.repository.settings.SbtRepositorySettings; + +/** + * Fetch configs from internal Artifactory. + */ +public class ArtifactoryClient { + private static final Logger log = LogManager.getLogger(); + + private final Artifactory artifactory; + + /** + * Resolve credentials and create ArtifactoryFetcher. + */ + public ArtifactoryClient(Config config) { + artifactory = ArtifactoryClientBuilder.create() + .setUrl(config.getArtifactoryUrl()) + .setUsername(config.getArtifactoryUsername()) + .setPassword(config.getArtifactoryPassword()) + .build(); + } + + /** + * This is useful for the remote repo, but for the local repo there is only one file: /.pypi/simple.html + */ + public List getAllPyPiPackageIdentifiersFromPyPiDir(String repositoryName) { + log.info(() -> "Downloading all package identifiers for " + repositoryName); + var topLevelFolders = listSubFolders(repositoryName, "/.pypi/"); + + var result = new ArrayList(); + for (var topLevelFolder : topLevelFolders) { + var candidate = topLevelFolder.substring(1, topLevelFolder.length() - ".html".length()); + + // TODO simple.html is the index, there might be other corner cases + if (candidate.equals("simple")) { + continue; + } + + // TODO consider filtering out names that are not allowed upstream + result.add(new PyPiPackageIdentifier(candidate)); + } + return result; + } + + public List getAllPyPiPackageIdentifiers(String repositoryName) { + log.info(() -> "Downloading all package identifiers for " + repositoryName); + var topLevelFolders = listTopLevelFolders(repositoryName); + + var result = new ArrayList(); + for (var topLevelFolder : topLevelFolders) { + var candidate = topLevelFolder.substring(1); + + // TODO consider filtering out names that are not allowed upstream + result.add(new PyPiPackageIdentifier(topLevelFolder.substring(1))); + } + return result; + } + + public List getAllNpmPackageIdentifiersForCache(String repositoryName) { + return getAllNpmPackageIdentifiers(repositoryName, "/"); + } + + public List getAllNpmPackageIdentifiersForLocal(String repositoryName) { + return getAllNpmPackageIdentifiers(repositoryName, "/.npm/"); + } + + // TODO consider filtering out names that are not allowed upstream + public List getAllNpmPackageIdentifiers(String repositoryName, String path) { + log.info(() -> "Downloading all package identifiers for " + repositoryName); + var topLevelFolders = listSubFolders(repositoryName, path); + + var result = new ArrayList(); + for (var topLevelFolder : topLevelFolders) { + if (topLevelFolder.startsWith("/@")) { + var subFolders = listSubFolders(repositoryName, topLevelFolder); + var scope = topLevelFolder.substring(2); + for (var packageName : subFolders) { + result.add(new NpmPackageIdentifier(scope, packageName.substring(1))); + } + } else { + if (topLevelFolder.startsWith("/.")) { + continue; + } + result.add(new NpmPackageIdentifier(topLevelFolder.substring(1))); + } + } + return result; + } + + public List repoLs() { + log.info("Retrieving repos from Artifactory"); + var virtual = artifactory.repositories().list(RepositoryTypeImpl.VIRTUAL); + var remote = artifactory.repositories().list(RepositoryTypeImpl.REMOTE); + var local = artifactory.repositories().list(RepositoryTypeImpl.LOCAL); + + var repos = new HashMap(); + var reposRaw = new HashMap(); + + for (var v : local) { + var r = artifactory.repository(v.getKey()).get(); + var artifactType = artifactType(r.getRepositorySettings()); + repos.put(v.getKey(), new Repository(r.getKey(), "local", artifactType, null, new ArrayList<>())); + reposRaw.put(v.getKey(), r); + } + + + for (var v : remote) { + var r = (RemoteRepositoryImpl) artifactory.repository(v.getKey()).get(); + var artifactType = artifactType(r.getRepositorySettings()); + repos.put(v.getKey(), new Repository(r.getKey(), "remote", artifactType, r.getUrl(), new ArrayList<>())); + reposRaw.put(v.getKey(), r); + } + + for (var v : virtual) { + var r = (VirtualRepositoryImpl) artifactory.repository(v.getKey()).get(); + var artifactType = artifactType(r.getRepositorySettings()); + + repos.put(v.getKey(), new Repository(r.getKey(), "virtual", artifactType, null, new ArrayList<>())); + reposRaw.put(v.getKey(), r); + } + + for (var v : virtual) { + var r = (VirtualRepositoryImpl) reposRaw.get(v.getKey()); + + var repo = repos.get(v.getKey()); + for (var c : r.getRepositories()) { + repo.addChild(repos.get(c)); + } + } + + return repos.values().stream() + .sorted(Comparator.comparingInt(a -> -a.getChildren().size())) + .sorted(Comparator.comparingInt(a -> a.getParents().size())) + .sorted(Comparator.comparing(Repository::getType).reversed()) + .collect(Collectors.toList()); + } + + // TODO support more + public String artifactType(RepositorySettings repositorySettings) { + if (repositorySettings instanceof PypiRepositorySettings) { + return "pypi"; + } else if (repositorySettings instanceof GradleRepositorySettings) { + return "gradle"; + } else if (repositorySettings instanceof NpmRepositorySettings) { + return "npm"; + } else if (repositorySettings instanceof SbtRepositorySettings) { + return "sbt"; + } else if (repositorySettings instanceof MavenRepositorySettings) { + return "maven"; + } else if (repositorySettings instanceof NugetRepositorySettings) { + return "nuget"; + } else if (repositorySettings instanceof ComposerRepositorySettings) { + return "composer"; + } else { + return "other"; + } + } + + List listTopLevelFolders(String repositoryName) { + return listSubFolders(repositoryName, "/"); + } + + List listSubFolders(String repositoryName, String folderName) { + Folder folder = artifactory.repository(repositoryName).folder(folderName).info(); + + return folder.getChildren().stream() + .map(Item::getUri) + .collect(Collectors.toList()); + } + + public Stats repoStats(String repoName, String packageSystem) { + return recursiveStats(repoName, "/", archiveExtensions(packageSystem)); + } + + public Stats packageStats(String repoName, String packageSystem, NpmPackageIdentifier packageName) { + return recursiveStats(repoName, packageName.toString(), archiveExtensions(packageSystem)); + } + + List archiveExtensions(String packageSystem) { + switch (packageSystem.toLowerCase(Locale.ENGLISH)) { + case "maven": + return List.of(".jar", ".war", ".rar", ".ear", ".sar", ".apk", ".aar", ".par", ".kar"); + case "npm": + return List.of(".tgz"); + case "pypi": + return List.of(".tar.gz"); + default: + throw new RuntimeException("Unknown package system " + packageSystem); + } + } + + public Stats recursiveStats(String repository, String path, List archiveExtensions) { + Folder folder = artifactory.repository(repository).folder(path).info(); + + if (folder.getChildren() == null) { + var extensionMatch = archiveExtensions.stream().filter(path::endsWith).findAny(); + if (extensionMatch.isPresent()) { + //System.out.println(path); + var stats = getDownloadStats(repository, path); + Optional lastDownloaded = stats.getDownloadCount() > 0 ? Optional.of(stats.getLastDownloaded()) : Optional.empty(); + Optional lastDownloadedBy = stats.getDownloadCount() > 0 ? Optional.of(stats.getLastDownloadedBy()) : Optional.empty(); + return new Stats(stats.getDownloadCount(), 1, lastDownloaded, lastDownloadedBy, Set.of(extensionMatch.get())); + } else { + return Stats.empty(); + } + } + + var result = Stats.empty(); + for (var child : folder.getChildren()) { + result = result.merge(recursiveStats(repository, path + child.getUri(), archiveExtensions)); + } + + return result; + } + + /** + * https://www.jfrog.com/confluence/display/JFROG/Artifactory+REST+API#ArtifactoryRESTAPI-FileStatistics + */ + public DownloadStats getDownloadStats(String repository, String path) { + var artifactoryImpl = (ArtifactoryImpl) artifactory; + var url = "/api/storage/" + repository + "/" + path + "?stats"; + try { + return new DownloadStats(artifactoryImpl.get(url, StatsInfoImpl.class, StatsInfo.class)); + } catch (IOException e) { + throw new RuntimeException(e); + } + } +} diff --git a/src/main/java/com/schibsted/security/artishock/artifactory/DownloadStats.java b/src/main/java/com/schibsted/security/artishock/artifactory/DownloadStats.java new file mode 100644 index 0000000..ea6b1ed --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/artifactory/DownloadStats.java @@ -0,0 +1,51 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.artifactory; + +import java.time.Instant; +import java.time.ZoneOffset; +import java.time.ZonedDateTime; + +public class DownloadStats { + private final String uri; + private final long downloadCount; + private final ZonedDateTime lastDownloaded; + private final String lastDownloadedBy; + private final long remoteDownloadCount; + private final ZonedDateTime remoteLastDownloaded; + + public DownloadStats(StatsInfo statsInfo) { + this.uri = statsInfo.getUri(); + this.downloadCount = statsInfo.getDownloadCount(); + this.lastDownloaded = ZonedDateTime.ofInstant(Instant.ofEpochMilli(statsInfo.getLastDownloaded()), ZoneOffset.UTC); + this.lastDownloadedBy = statsInfo.getLastDownloadedBy(); + this.remoteDownloadCount = statsInfo.getRemoteDownloadCount(); + this.remoteLastDownloaded = ZonedDateTime.ofInstant(Instant.ofEpochMilli(statsInfo.getRemoteLastDownloaded()), ZoneOffset.UTC); + } + + public String getUri() { + return uri; + } + + public long getDownloadCount() { + return downloadCount; + } + + public ZonedDateTime getLastDownloaded() { + return lastDownloaded; + } + + public String getLastDownloadedBy() { + return lastDownloadedBy; + } + + public long getRemoteDownloadCount() { + return remoteDownloadCount; + } + + public ZonedDateTime getRemoteLastDownloaded() { + return remoteLastDownloaded; + } +} diff --git a/src/main/java/com/schibsted/security/artishock/artifactory/Repository.java b/src/main/java/com/schibsted/security/artishock/artifactory/Repository.java new file mode 100644 index 0000000..b102f28 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/artifactory/Repository.java @@ -0,0 +1,59 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.artifactory; + +import java.util.ArrayList; +import java.util.List; + +public class Repository { + private final String name; + private final String type; + private final String artifactType; + private final List children; + private final List parents; + private final String url; + + public Repository(String name, String type, String artifactType, String url, List children) { + this.name = name; + this.type = type; + this.artifactType = artifactType; + this.children = children; + this.parents = new ArrayList<>(); + this.url = url; + } + + public String getName() { + return name; + } + + public String getType() { + return type; + } + + public String getArtifactType() { + return artifactType; + } + + public String getUrl() { + return url; + } + + public List getChildren() { + return children; + } + + public List getParents() { + return parents; + } + + public void addChild(Repository repository) { + this.children.add(repository); + repository.addParent(this); + } + + public void addParent(Repository repository) { + parents.add(repository); + } +} diff --git a/src/main/java/com/schibsted/security/artishock/artifactory/Stats.java b/src/main/java/com/schibsted/security/artishock/artifactory/Stats.java new file mode 100644 index 0000000..24591b5 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/artifactory/Stats.java @@ -0,0 +1,84 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.artifactory; + +import java.time.ZonedDateTime; +import java.util.Collection; +import java.util.Optional; +import java.util.Set; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +public class Stats { + private final long downloadCount; + private final long archiveCount; + private final Optional lastDownloaded; + private final Optional lastDownloadedBy; + private final Set extensions; + + public Stats(long downloadCount, long archiveCount, + Optional lastDownloaded, + Optional lastDownloadedBy, + Set extensions) { + this.downloadCount = downloadCount; + this.archiveCount = archiveCount; + this.lastDownloaded = lastDownloaded; + this.lastDownloadedBy = lastDownloadedBy; + this.extensions = extensions; + } + + public long getDownloadCount() { + return downloadCount; + } + + public long getArchiveCount() { + return archiveCount; + } + + public Optional getLastDownloaded() { + return lastDownloaded; + } + + public Optional getLastDownloadedBy() { + return lastDownloadedBy; + } + + public Set getExtensions() { + return extensions; + } + + public Stats merge(Stats other) { + var extensions = Stream.of(this.extensions, other.getExtensions()) + .flatMap(Collection::stream) + .distinct() + .collect(Collectors.toSet()); + + return new Stats(this.downloadCount + other.getDownloadCount(), + this.archiveCount + other.getArchiveCount(), + isAfter(this.lastDownloaded, other.getLastDownloaded()) ? this.lastDownloaded : other.getLastDownloaded(), + isAfter(this.lastDownloaded, other.getLastDownloaded()) ? this.getLastDownloadedBy() : other.getLastDownloadedBy(), + extensions); + } + + public static Stats empty() { + return new Stats(0, 0, Optional.empty(), Optional.empty(), Set.of()); + } + + private static boolean isAfter(Optional a, Optional b) { + if (a.isEmpty() && b.isEmpty()) { + return true; + } + + if (a.isPresent() && b.isEmpty()) { + return true; + } + + if (b.isPresent() && a.isEmpty()) { + return false; + } + + return a.get().isAfter(b.get()); + } +} diff --git a/src/main/java/com/schibsted/security/artishock/artifactory/StatsInfo.java b/src/main/java/com/schibsted/security/artishock/artifactory/StatsInfo.java new file mode 100644 index 0000000..44545df --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/artifactory/StatsInfo.java @@ -0,0 +1,17 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.artifactory; + +import com.fasterxml.jackson.annotation.JsonIgnoreProperties; + +@JsonIgnoreProperties(ignoreUnknown = true) +public interface StatsInfo { + long getDownloadCount(); + long getRemoteDownloadCount(); + long getLastDownloaded(); + long getRemoteLastDownloaded(); + String getLastDownloadedBy(); + String getUri(); +} diff --git a/src/main/java/com/schibsted/security/artishock/artifactory/StatsInfoImpl.java b/src/main/java/com/schibsted/security/artishock/artifactory/StatsInfoImpl.java new file mode 100644 index 0000000..a8dd4c0 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/artifactory/StatsInfoImpl.java @@ -0,0 +1,68 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.artifactory; + +public class StatsInfoImpl implements StatsInfo { + private long downloadCount; + private long remoteDownloadCount; + private long lastDownloaded; + private long remoteLastDownloaded; + private String lastDownloadedBy; + private String uri; + + @Override + public long getDownloadCount() { + return downloadCount; + } + + @Override + public long getRemoteDownloadCount() { + return remoteDownloadCount; + } + + @Override + public long getLastDownloaded() { + return lastDownloaded; + } + + @Override + public long getRemoteLastDownloaded() { + return remoteLastDownloaded; + } + + @Override + public String getLastDownloadedBy() { + return lastDownloadedBy; + } + + @Override + public String getUri() { + return uri; + } + + public void setDownloadCount(long downloadCount) { + this.downloadCount = downloadCount; + } + + public void setRemoteDownloadCount(long remoteDownloadCount) { + this.remoteDownloadCount = remoteDownloadCount; + } + + public void setLastDownloaded(long lastDownloaded) { + this.lastDownloaded = lastDownloaded; + } + + public void setRemoteLastDownloaded(long remoteLastDownloaded) { + this.remoteLastDownloaded = remoteLastDownloaded; + } + + public void setLastDownloadedBy(String lastDownloadedBy) { + this.lastDownloadedBy = lastDownloadedBy; + } + + public void setUri(String uri) { + this.uri = uri; + } +} diff --git a/src/main/java/com/schibsted/security/artishock/cli/view/Commands.java b/src/main/java/com/schibsted/security/artishock/cli/view/Commands.java new file mode 100644 index 0000000..0cea610 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/cli/view/Commands.java @@ -0,0 +1,239 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.cli.view; + +import com.schibsted.security.artishock.cli.viewmodel.Artishock; +import java.util.List; +import javax.inject.Inject; +import io.airlift.airline.Command; +import io.airlift.airline.Help; +import io.airlift.airline.OptionType; +import io.airlift.airline.model.GlobalMetadata; +import org.apache.logging.log4j.Level; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.core.config.Configurator; + +public class Commands { + private static final String PACKAGE_SYSTEM_NAME = "--package-system"; + private static final String LOCAL_NAME = "--local"; + private static final String LOCAL_DESCRIPTION = "Name of an Artifactory repo of type local"; + private static final String REMOTE_NAME = "--remote"; + private static final String REPO_DESCRIPTION = "Name of Artifactory repo"; + private static final String REPO_NAME = "--repo"; + private static final String REMOTE_DESCRIPTION = "Name of an Artifactory repo of type remote"; + private static final String QUERY_UPSTREAM_NAME = "--query-upstream"; + private static final String QUERY_UPSTREAM_DESCRIPTION = "Acknowledge that this command sends internal package names upstream"; + + private static Artishock confused() { + return new Artishock(); + } + + @io.airlift.airline.Command(name = "help", description = "Help") + public static class CustomHelp extends Help { + @Override + public void run() { + super.run(); + System.exit(1); + } + } + + @io.airlift.airline.Command(name = "--help", description = "Help", hidden = true) + public static class CustomHelpAlternativeName extends CustomHelp { + } + + @io.airlift.airline.Command(name = "version", description = "Get version") + public static class Version implements Runnable { + + @Override + public void run() { + Package p = getClass().getPackage(); + String version = p.getImplementationVersion(); + + System.out.println(String.format("artishock %s", version)); + } + } + + @io.airlift.airline.Command(name = "--version", description = "Get version", hidden = true) + public static class VersionAlternativeName extends Version { + } + + public static class BaseCommand implements Runnable { + @Inject + public GlobalMetadata global; + + @io.airlift.airline.Option(type = OptionType.COMMAND, name = "--json", description = "Output as JSON") + public boolean json; + + @io.airlift.airline.Option(type = OptionType.COMMAND, name = "--verbose", description = "Make verbose") + public boolean verbose; + + @io.airlift.airline.Option(type = OptionType.COMMAND, name = "--stacktrace", description = "Enable stacktrace") + public boolean stacktrace; + + @io.airlift.airline.Option(type = OptionType.COMMAND, name = "--help", description = "Show help") + public boolean help; + + protected Renderer renderer() { + return new Renderer(json ? OutputFormat.JSON : OutputFormat.TEXT, System.out); + } + + private void help(List commands) { + if (help) { + Help.help(global, commands); + System.exit(0); + } + } + + private void configureVerbose() { + if (verbose) { + Configurator.setAllLevels(LogManager.getRootLogger().getName(), Level.INFO); + } else { + Configurator.setAllLevels(LogManager.getRootLogger().getName(), Level.OFF); + } + } + + protected void verboseAndHelp(String command) { + configureVerbose(); + help(List.of(command)); + } + + @Override + public void run() { + } + } + + @Command(name = "repo-ls", description = "List Artifactory repositories") + public static class RepoLs extends BaseCommand { + @io.airlift.airline.Option(type = OptionType.COMMAND, name = PACKAGE_SYSTEM_NAME, description = "maven, npm, pypi") + public String packageSystem; + + @Override + public void run() { + verboseAndHelp("repo-ls"); + + renderer().render(confused().repoLs(packageSystem)); + } + } + + + @Command(name = "repo-stats", description = "Stats for a given Artifactory repository by iterating over all files (slow)") + public static class RepoStats extends BaseCommand { + + @io.airlift.airline.Option(name = REPO_NAME, description = REPO_DESCRIPTION, required = true) + public String repoName; + + @io.airlift.airline.Option(name = PACKAGE_SYSTEM_NAME, description = "maven, npm, pypi", required = true) + public String packageSystem; + + @Override + public void run() { + verboseAndHelp("repo-stats"); + + renderer().render(confused().repoStats(repoName, packageSystem)); + } + } + + @Command(name = "package-stats", description = "Stats for a given package in Artifactory by iterating recursively (can be slow)") + public static class PackageStats extends BaseCommand { + + @io.airlift.airline.Option(name = REPO_NAME, description = REPO_DESCRIPTION, required = true) + public String repoName; + + @io.airlift.airline.Option(name = PACKAGE_SYSTEM_NAME, description = "npm", required = true) + public String packageSystem; + + @io.airlift.airline.Option(name = "--package", description = "Name of the package", required = true) + public String packageName; + + @Override + public void run() { + verboseAndHelp("package-stats"); + + renderer().render(confused().packageStats(repoName, packageSystem, packageName)); + } + } + + @Command(name = "exclude-candidates", description = "Packages that are candidates to be excluded") + public static class ExcludeCandidates extends BaseCommand { + @io.airlift.airline.Option(name = PACKAGE_SYSTEM_NAME, description = "npm, pypi", required = true) + public String packageSystem; + + @io.airlift.airline.Option(name = LOCAL_NAME, description = LOCAL_DESCRIPTION, required = true) + public String local; + + @io.airlift.airline.Option(name = "--trusted", description = "File containing trusted packages") + public String trusted; + + @io.airlift.airline.Option(name = "--excluded", description = "File containing excluded packages") + public String excluded; + + @Override + public void run() { + verboseAndHelp("exclude-candidates"); + + renderer().render(confused().excludeCandidates(packageSystem, local, trusted, excluded)); + } + } + + @Command(name = "cached", description = "Local packages that exist upstream and have been cached by Artifactory") + public static class Cached extends BaseCommand { + @io.airlift.airline.Option(name = PACKAGE_SYSTEM_NAME, description = "npm, pypi", required = true) + public String packageSystem; + + @io.airlift.airline.Option(name = LOCAL_NAME, description = LOCAL_DESCRIPTION, required = true) + public String local; + + @io.airlift.airline.Option(name = REMOTE_NAME, description = REMOTE_DESCRIPTION, required = true) + public String remote; + + @Override + public void run() { + verboseAndHelp("cached"); + + renderer().render(confused().cached(packageSystem, local, remote)); + } + } + + @Command(name = "inferred-exclude", description = "Infer excluded packages (best effort)") + public static class InferredExclude extends BaseCommand { + @io.airlift.airline.Option(name = PACKAGE_SYSTEM_NAME, description = "npm, pypi", required = true) + public String packageSystem; + + @io.airlift.airline.Option(name = LOCAL_NAME, description = LOCAL_DESCRIPTION, required = true) + public String local; + + @io.airlift.airline.Option(name = REMOTE_NAME, description = REMOTE_DESCRIPTION, required = true) + public String remote; + + @io.airlift.airline.Option(name = QUERY_UPSTREAM_NAME, description = QUERY_UPSTREAM_DESCRIPTION, required = false) + boolean queryUpstream; + + @Override + public void run() { + verboseAndHelp("inferred-exclude"); + + renderer().render(confused().inferredExclude(packageSystem, local, remote, queryUpstream)); + } + } + + @Command(name = "not-claimed", description = "Local packages not claimed upstream") + public static class NotClaimed extends BaseCommand { + @io.airlift.airline.Option(name = PACKAGE_SYSTEM_NAME, description = "npm, pypi", required = true) + public String packageSystem; + + @io.airlift.airline.Option(name = LOCAL_NAME, description = LOCAL_DESCRIPTION, required = true) + public String local; + + @io.airlift.airline.Option(name = QUERY_UPSTREAM_NAME, description = QUERY_UPSTREAM_DESCRIPTION, required = false) + boolean queryUpstream; + + @Override + public void run() { + verboseAndHelp("not-claimed"); + + renderer().render(confused().notClaimed(packageSystem, local, queryUpstream)); + } + } +} diff --git a/src/main/java/com/schibsted/security/artishock/cli/view/OutputFormat.java b/src/main/java/com/schibsted/security/artishock/cli/view/OutputFormat.java new file mode 100644 index 0000000..45fa18e --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/cli/view/OutputFormat.java @@ -0,0 +1,9 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.cli.view; + +public enum OutputFormat { + TEXT, JSON; +} diff --git a/src/main/java/com/schibsted/security/artishock/cli/view/Renderer.java b/src/main/java/com/schibsted/security/artishock/cli/view/Renderer.java new file mode 100644 index 0000000..be36e69 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/cli/view/Renderer.java @@ -0,0 +1,84 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.cli.view; + +import com.google.common.base.Joiner; +import java.io.IOException; +import java.io.PrintStream; +import java.nio.charset.StandardCharsets; +import java.util.Collection; +import java.util.stream.Collectors; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.databind.SerializationFeature; +import com.fasterxml.jackson.datatype.jdk8.Jdk8Module; + +public class Renderer { + private final OutputFormat outputFormat; + private final PrintStream outputStream; + + public Renderer(OutputFormat outputFormat, PrintStream outputStream) { + this.outputStream = outputStream; + + this.outputFormat = outputFormat; + switch (outputFormat) { + case JSON: + case TEXT: + break; + default: + throw new IllegalArgumentException(String.format("Unexpected output format '%s', reverting to text", outputFormat.toString())); + } + } + + public > void render(T object) { + writeToStream(arrayOutput(object).getBytes(StandardCharsets.UTF_8)); + } + + public void render(T object) { + writeToStream(singleOutput(object).getBytes(StandardCharsets.UTF_8)); + } + + public void writeToStream(byte[] data) { + try { + this.outputStream.write(data); + this.outputStream.flush(); + } catch (IOException e) { + throw new RuntimeException("Failed to write output from CLI", e); + } + } + + private > String arrayOutput(T object) { + switch (outputFormat) { + case JSON: + return serializeToJSON(object) + "\n"; + case TEXT: + return Joiner.on("\n").join(object.stream().map(Object::toString).collect(Collectors.toList())) + "\n"; + default: + throw new RuntimeException("Unexpected output format"); + } + } + + public String singleOutput(T object) { + switch (outputFormat) { + case JSON: + return serializeToJSON(object) + "\n"; + case TEXT: + return object.toString() + "\n"; + default: + throw new RuntimeException("Unexpected output format"); + } + } + + private String serializeToJSON(Object object) { + ObjectMapper m = new ObjectMapper() + .registerModule(new Jdk8Module()); + m.enable(SerializationFeature.INDENT_OUTPUT); + try { + return m.writeValueAsString(object); + } catch (JsonProcessingException e) { + throw new RuntimeException("Failed to serialize output to JSON", e); + } + } +} diff --git a/src/main/java/com/schibsted/security/artishock/cli/viewmodel/Artishock.java b/src/main/java/com/schibsted/security/artishock/cli/viewmodel/Artishock.java new file mode 100644 index 0000000..a503776 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/cli/viewmodel/Artishock.java @@ -0,0 +1,147 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.cli.viewmodel; + +import com.google.common.base.Joiner; +import com.schibsted.security.artishock.cli.viewmodel.types.PyPiPackageIdentifier; +import com.schibsted.security.artishock.npm.Npm; +import com.schibsted.security.artishock.artifactory.ArtifactoryClient; +import com.schibsted.security.artishock.cli.viewmodel.types.ArtifactoryRepository; +import com.schibsted.security.artishock.cli.viewmodel.types.ArtifactoryRepositoryStats; +import com.schibsted.security.artishock.cli.viewmodel.types.NpmPackageIdentifier; +import com.schibsted.security.artishock.cli.viewmodel.types.NpmPackageOrScope; +import com.schibsted.security.artishock.config.ConfigResolver; +import com.schibsted.security.artishock.pypi.Pypi; +import com.schibsted.security.artishock.shared.PackageSystem; +import java.util.List; +import java.util.Optional; +import java.util.stream.Collectors; + +public class Artishock { + private final Npm npm; + private final Pypi pypi; + private final ArtifactoryClient artifactoryClient; + + public Artishock() { + var configResolver = new ConfigResolver(); + var config = configResolver.resolveConfig(); + this.artifactoryClient = new ArtifactoryClient(config); + this.npm = new Npm(config, artifactoryClient); + this.pypi = new Pypi(config, artifactoryClient); + } + + public List repoLs(String packageSystem) { + throwIfNotSupportedOptional(packageSystem, List.of(PackageSystem.MAVEN, PackageSystem.NPM, PackageSystem.PYPI)); + + return artifactoryClient.repoLs().stream() + .filter(f -> packageSystem == null || f.getArtifactType().equals(packageSystem)) + .map(ArtifactoryRepository::new) + .collect(Collectors.toList()); + } + + public Object repoStats(String repoName, String packageSystem) { + getPackageSystemOrThrow(packageSystem, List.of(PackageSystem.MAVEN, PackageSystem.NPM, PackageSystem.PYPI)); + + return new ArtifactoryRepositoryStats(repoName, artifactoryClient.repoStats(repoName, packageSystem)); + } + + public Object packageStats(String repoName, String packageSystem, String packageName) { + getPackageSystemOrThrow(packageSystem, List.of(PackageSystem.NPM)); + + var identifier = new com.schibsted.security.artishock.npm.NpmPackageIdentifier(packageName); + return new ArtifactoryRepositoryStats(packageName, artifactoryClient.packageStats(repoName, packageSystem, identifier)); + } + + public List excludeCandidates(String packageSystem, String local, String trusted, String excluded) { + var system = getPackageSystemOrThrow(packageSystem, List.of(PackageSystem.NPM, PackageSystem.PYPI)); + + if (system.equals(PackageSystem.NPM)) { + return npm.excludeCandidates(local, Optional.ofNullable(trusted), Optional.ofNullable(excluded)).stream() + .map(NpmPackageIdentifier::new) + .collect(Collectors.toList()); + } else if (system.equals(PackageSystem.PYPI)) { + return pypi.excludeCandidates(local, Optional.ofNullable(trusted), Optional.ofNullable(excluded)).stream() + .map(PyPiPackageIdentifier::new) + .collect(Collectors.toList()); + } + + throw new RuntimeException("Implementation bug"); + } + + public List cached(String packageSystem, String local, String remote) { + var system = getPackageSystemOrThrow(packageSystem, List.of(PackageSystem.NPM, PackageSystem.PYPI)); + + if (system.equals(PackageSystem.NPM)) { + return npm.cached(local, remote).stream() + .map(NpmPackageIdentifier::new) + .collect(Collectors.toList()); + } else if (system.equals(PackageSystem.PYPI)) { + return pypi.cached(local, remote).stream() + .map(PyPiPackageIdentifier::new) + .collect(Collectors.toList()); + } + + throw new RuntimeException("Implementation bug"); + } + + public List inferredExclude(String packageSystem, String local, String remote, boolean enableUpstream) { + verifyEnableUpstreamOrThrow(enableUpstream); + + var system = getPackageSystemOrThrow(packageSystem, List.of(PackageSystem.NPM, PackageSystem.PYPI)); + + if (system.equals(PackageSystem.NPM)) { + return npm.inferredExclude(local, remote).stream() + .map(NpmPackageIdentifier::new) + .collect(Collectors.toList()); + } else if (system.equals(PackageSystem.PYPI)){ + return pypi.inferredExclude(local, remote).stream() + .map(PyPiPackageIdentifier::new) + .collect(Collectors.toList()); + } + + throw new RuntimeException("Implementation bug"); + } + + public List notClaimed(String packageSystem, String local, boolean enableUpstream) { + verifyEnableUpstreamOrThrow(enableUpstream); + + var system = getPackageSystemOrThrow(packageSystem, List.of(PackageSystem.NPM, PackageSystem.PYPI)); + + if (system.equals(PackageSystem.NPM)) { + return npm.notClaimed(local).stream() + .map(NpmPackageOrScope::new) + .collect(Collectors.toList()); + } else if (system.equals(PackageSystem.PYPI)) { + return pypi.notClaimed(local).stream() + .map(PyPiPackageIdentifier::new) + .collect(Collectors.toList()); + } + + throw new RuntimeException("Implementation bug"); + } + + void throwIfNotSupportedOptional(String packageSystem, List supported) { + if (packageSystem != null) { + getPackageSystemOrThrow(packageSystem, supported); + } + } + + PackageSystem getPackageSystemOrThrow(String packageSystem, List supported) { + var type = PackageSystem.fromString(packageSystem); + if (type.isEmpty() || !supported.contains(type.get())) { + throw new RuntimeException(String.format("Unsupported package system '%s', try one of {%s}", + packageSystem, + Joiner.on(", ").join(supported))); + } + + return type.get(); + } + + void verifyEnableUpstreamOrThrow(boolean enableUpstream) { + if (!enableUpstream) { + throw new RuntimeException("This command only works by querying internal packages upstream, add `--query-upstream` if this is OK or don't use this command"); + } + } +} diff --git a/src/main/java/com/schibsted/security/artishock/cli/viewmodel/types/ArtifactoryRepository.java b/src/main/java/com/schibsted/security/artishock/cli/viewmodel/types/ArtifactoryRepository.java new file mode 100644 index 0000000..0209dfb --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/cli/viewmodel/types/ArtifactoryRepository.java @@ -0,0 +1,68 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.cli.viewmodel.types; + +import com.schibsted.security.artishock.artifactory.Repository; +import java.util.List; +import java.util.stream.Collectors; + +public class ArtifactoryRepository { + private final String name; + private final String repositoryType; + private final String packageSystem; + private final List repositories; + private final String upstreamUrl; + + public ArtifactoryRepository(Repository repository) { + this.name = repository.getName(); + this.repositoryType = repository.getType(); + this.packageSystem = repository.getArtifactType(); + this.repositories = repository.getChildren().stream().map(ArtifactoryRepository::new).collect(Collectors.toList()); + this.upstreamUrl = repository.getUrl(); + } + + public String getName() { + return name; + } + + public String getRepositoryType() { + return repositoryType; + } + + public String getPackageSystem() { + return packageSystem; + } + + public String getUpstreamUrl() { + return upstreamUrl; + } + + public List getRepositories() { + return repositories; + } + + @Override + public String toString() { + var sb = new StringBuilder(); + print(sb, this, 0); + return sb.toString(); + } + + private void print(StringBuilder sb, ArtifactoryRepository repo, int indent) { + var prefix = "-".repeat(indent); + if (indent > 0) { + prefix = prefix + " "; + } + if (repo.getUpstreamUrl() != null) { + sb.append(String.format("%s%s [%s:%s:%s]", prefix, repo.getName(), repo.getPackageSystem(), repo.getRepositoryType(), repo.getUpstreamUrl())); + } else { + sb.append(String.format("%s%s [%s:%s]", prefix, repo.getName(), repo.getPackageSystem(), repo.getRepositoryType())); + } + for (var c : repo.getRepositories()) { + sb.append("\n"); + print(sb, c, indent + 1); + } + } +} diff --git a/src/main/java/com/schibsted/security/artishock/cli/viewmodel/types/ArtifactoryRepositoryStats.java b/src/main/java/com/schibsted/security/artishock/cli/viewmodel/types/ArtifactoryRepositoryStats.java new file mode 100644 index 0000000..478ecd9 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/cli/viewmodel/types/ArtifactoryRepositoryStats.java @@ -0,0 +1,68 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.cli.viewmodel.types; + +import com.google.common.base.Joiner; +import com.schibsted.security.artishock.artifactory.Stats; +import java.time.ZonedDateTime; +import java.util.Optional; +import java.util.Set; + +public class ArtifactoryRepositoryStats { + private final String name; + private final long downloadCount; + private final long archiveCount; + private final Optional lastDownloaded; + private final Optional lastDownloadedBy; + private final Set extensions; + + public ArtifactoryRepositoryStats(String name, Stats stats) { + this.name = name; + this.downloadCount = stats.getDownloadCount(); + this.archiveCount = stats.getArchiveCount(); + this.lastDownloaded = stats.getLastDownloaded().map(ZonedDateTime::toString); + this.lastDownloadedBy = stats.getLastDownloadedBy(); + this.extensions = stats.getExtensions(); + } + + public String getName() { + return name; + } + + public long getDownloadCount() { + return downloadCount; + } + + public long getArchiveCount() { + return archiveCount; + } + + public Optional getLastDownloaded() { + return lastDownloaded; + } + + public Optional getLastDownloadedBy() { + return lastDownloadedBy; + } + + public Set getExtensions() { + return extensions; + } + + @Override + public String toString() { + var sb = new StringBuilder(); + + sb.append(String.format("name: %s\n", name)); + sb.append(String.format("archives: %d\n", archiveCount)); + sb.append(String.format("archives types: %s\n", Joiner.on(",").join(extensions))); + sb.append(String.format("downloads: %d\n", downloadCount)); + + lastDownloaded.ifPresent(s -> sb.append(String.format("last downloaded: %s\n", s))); + lastDownloadedBy.ifPresent(s -> sb.append(String.format("last downloaded by: %s\n", s))); + + return sb.toString(); + } +} diff --git a/src/main/java/com/schibsted/security/artishock/cli/viewmodel/types/NpmPackageIdentifier.java b/src/main/java/com/schibsted/security/artishock/cli/viewmodel/types/NpmPackageIdentifier.java new file mode 100644 index 0000000..d807fb4 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/cli/viewmodel/types/NpmPackageIdentifier.java @@ -0,0 +1,30 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.cli.viewmodel.types; + +import java.util.Optional; + +public class NpmPackageIdentifier { + private final String packageName; + private final Optional scope; + + public NpmPackageIdentifier(com.schibsted.security.artishock.npm.NpmPackageIdentifier packageIdentifier) { + this.packageName = packageIdentifier.getPackageName(); + this.scope = packageIdentifier.getScope(); + } + + public String getPackageName() { + return packageName; + } + + public Optional getScope() { + return scope; + } + + @Override + public String toString() { + return scope.map(s -> "@" + s + "/" + packageName).orElse(packageName); + } +} diff --git a/src/main/java/com/schibsted/security/artishock/cli/viewmodel/types/NpmPackageOrScope.java b/src/main/java/com/schibsted/security/artishock/cli/viewmodel/types/NpmPackageOrScope.java new file mode 100644 index 0000000..c1be3cd --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/cli/viewmodel/types/NpmPackageOrScope.java @@ -0,0 +1,32 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.cli.viewmodel.types; + +import java.util.Optional; + +public class NpmPackageOrScope { + private final Optional packageName; + private final Optional scope; + private final String name; + + public NpmPackageOrScope(com.schibsted.security.artishock.npm.NpmPackageOrScope packageOrScope) { + this.packageName = packageOrScope.getPackageName(); + this.scope = packageOrScope.getScope(); + this.name = packageOrScope.toString(); + } + + public Optional getPackageName() { + return packageName; + } + + public Optional getScope() { + return scope; + } + + @Override + public String toString() { + return name; + } +} diff --git a/src/main/java/com/schibsted/security/artishock/cli/viewmodel/types/PyPiPackageIdentifier.java b/src/main/java/com/schibsted/security/artishock/cli/viewmodel/types/PyPiPackageIdentifier.java new file mode 100644 index 0000000..7db7b5e --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/cli/viewmodel/types/PyPiPackageIdentifier.java @@ -0,0 +1,22 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.cli.viewmodel.types; + +public class PyPiPackageIdentifier { + private final String packageName; + + public PyPiPackageIdentifier(com.schibsted.security.artishock.pypi.PyPiPackageIdentifier pyPiPackageIdentifier) { + this.packageName = pyPiPackageIdentifier.getPackageName(); + } + + public String getPackageName() { + return packageName; + } + + @Override + public String toString() { + return packageName; + } +} diff --git a/src/main/java/com/schibsted/security/artishock/config/Config.java b/src/main/java/com/schibsted/security/artishock/config/Config.java new file mode 100644 index 0000000..90b29d6 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/config/Config.java @@ -0,0 +1,29 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.config; + +public class Config { + private final String artifactoryUrl; + private final String artifactoryUsername; + private final String artifactoryPassword; + + public Config(String artifactoryUrl, String artifactoryUsername, String artifactoryPassword) { + this.artifactoryUrl = artifactoryUrl; + this.artifactoryUsername = artifactoryUsername; + this.artifactoryPassword = artifactoryPassword; + } + + public String getArtifactoryUrl() { + return artifactoryUrl; + } + + public String getArtifactoryUsername() { + return artifactoryUsername; + } + + public String getArtifactoryPassword() { + return artifactoryPassword; + } +} diff --git a/src/main/java/com/schibsted/security/artishock/config/ConfigRaw.java b/src/main/java/com/schibsted/security/artishock/config/ConfigRaw.java new file mode 100644 index 0000000..39e28e0 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/config/ConfigRaw.java @@ -0,0 +1,11 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.config; + +public class ConfigRaw { + public String artifactoryUrl; + public String artifactoryUsername; + public String artifactoryPassword; +} diff --git a/src/main/java/com/schibsted/security/artishock/config/ConfigResolver.java b/src/main/java/com/schibsted/security/artishock/config/ConfigResolver.java new file mode 100644 index 0000000..87e62bc --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/config/ConfigResolver.java @@ -0,0 +1,71 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.config; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Optional; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.DeserializationFeature; +import com.fasterxml.jackson.databind.ObjectMapper; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; + +public class ConfigResolver { + private static final Logger log = LogManager.getLogger(); + + public static Config resolveConfig() { + var file = configDirectory().resolve("artishock.config").toFile(); + + var mapper = new ObjectMapper().configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false); + + try { + var config = new ConfigRaw(); + + if (file.exists()) { + log.info(() -> String.format("Reading config file %s", file.toPath())); + var content = Files.readString(file.toPath()); + config = mapper.readValue(content, ConfigRaw.class); + } else { + log.info(() -> String.format("Could not read config file %s", file.toPath())); + } + + var artifactoryUrl = getValue(config.artifactoryUrl, "ARTISHOCK_ARTIFACTORY_URL", "artifactory url"); + var artifactoryUser = getValue(config.artifactoryUsername, "ARTISHOCK_ARTIFACTORY_USERNAME", "artifactory user"); + var artifactoryPassword = getValue(config.artifactoryPassword, "ARTISHOCK_ARTIFACTORY_PASSWORD", "artifactory api key"); + + return new Config(artifactoryUrl, artifactoryUser, artifactoryPassword); + } catch (JsonProcessingException e) { + throw new RuntimeException("Could deserialize config"); + } catch (IOException e) { + throw new RuntimeException("Could not read config from"); + } + } + + public static Path configDirectory() { + return Path.of(userHome() + "/.artishock/"); + } + + static String userHome() { + return System.getProperty("user.home"); + } + + static String getValue(String configFileValue, String env, String description) { + Optional envOverride = Optional.ofNullable(System.getenv(env)); + + if (envOverride.isEmpty() && configFileValue == null) { + throw new RuntimeException("Unable to get config " + description); + } + + if (envOverride.isPresent()) { + log.info(() -> String.format("Using %s from environment variable", description)); + return envOverride.get(); + } else { + log.info(() -> String.format("Using %s from config file", description)); + return configFileValue; + } + } +} diff --git a/src/main/java/com/schibsted/security/artishock/npm/Npm.java b/src/main/java/com/schibsted/security/artishock/npm/Npm.java new file mode 100644 index 0000000..d9288bf --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/npm/Npm.java @@ -0,0 +1,132 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.npm; + +import com.schibsted.security.artishock.artifactory.ArtifactoryClient; +import com.schibsted.security.artishock.npm.client.NpmClient; +import com.schibsted.security.artishock.config.Config; +import com.schibsted.security.artishock.shared.ConnectionInfo; +import com.schibsted.security.artishock.shared.FileReader; +import com.schibsted.security.artishock.shared.Intersection; +import java.util.ArrayList; +import java.util.Comparator; +import java.util.List; +import java.util.Optional; +import java.util.stream.Collectors; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; + +public class Npm { + private static final Logger log = LogManager.getLogger(); + + private final ArtifactoryClient artifactoryClient; + private final Config config; + + private final NpmClient npmClient; + + public Npm(Config config, ArtifactoryClient artifactoryClient) { + this.config = config; + this.artifactoryClient = artifactoryClient; + this.npmClient = new NpmClient(); + } + + public List excludeCandidates(String local, Optional trusted, Optional excluded) { + var candidates = artifactoryClient.getAllNpmPackageIdentifiersForLocal(local); + + if (trusted.isPresent()) { + candidates = filterOutPackageOrScope(candidates, packageOrScopes(trusted.get())); + } + + if (excluded.isPresent()) { + candidates = filterOutPackageOrScope(candidates, packageOrScopes(excluded.get())); + } + + return candidates; + } + + public List cached(String local, String remote) { + var localPackages = artifactoryClient.getAllNpmPackageIdentifiersForLocal(local); + + var remoteCached = artifactoryClient.getAllNpmPackageIdentifiersForCache(remote + "-cache"); + return Intersection.cacheIntersection(localPackages, remoteCached); + } + + public List inferredExclude(String local, String remote) { + var localPackages = artifactoryClient.getAllNpmPackageIdentifiersForLocal(local); + + // TODO search for scoped packages? This will leak additional names upstream + var localPackagesWithoutScope = packagesWithoutScope(localPackages); + + var upstreamPackages = npmClient.getPackageList(localPackagesWithoutScope, npmClient.upstream()); + + var remotePackages = npmClient.getPackageList(localPackagesWithoutScope, + new ConnectionInfo(npmApi(config.getArtifactoryUrl(), remote), config.getArtifactoryUsername(), config.getArtifactoryPassword())); + + upstreamPackages.removeAll(remotePackages); + upstreamPackages.sort(Comparator.comparing(NpmPackageIdentifier::toString)); + return upstreamPackages; + } + + public List notClaimed(String local) { + var localPackages = new ArrayList<>(artifactoryClient.getAllNpmPackageIdentifiersForLocal(local)); + + var localPackagesWithoutScope = packagesWithoutScope(localPackages); + + var localScopes = localPackages.stream() + .flatMap(f -> f.getScope().stream()) + .distinct() + .collect(Collectors.toList()); + + var upstreamPackages = npmClient.getPackageList(localPackagesWithoutScope, npmClient.upstream()); + + localPackagesWithoutScope.removeAll(upstreamPackages); + + var result = localPackagesWithoutScope.stream() + .map(NpmPackageOrScope::new) + .collect(Collectors.toList()); + + var notClaimedScopes = npmClient.notClaimedOrg(localScopes); + + result.addAll(notClaimedScopes); + + return result; + } + + String npmApi(String artifactoryBase, String repositoryName) { + return artifactoryBase + "/api/npm/" + repositoryName; + } + + List filterOutPackageOrScope(List all, List exclude) { + var excludeScopes = exclude.stream() + .filter(f -> f.getScope().isPresent() && f.getPackageName().isEmpty()) + .flatMap(f -> f.getScope().stream()) + .collect(Collectors.toSet()); + var excludePackages = exclude.stream() + .filter(f -> f.getScope().isEmpty()) + .flatMap(f -> f.getPackageName().stream()) + .collect(Collectors.toSet()); + var excludePackageAndScope = exclude.stream() + .filter(f -> f.getScope().isPresent() && f.getPackageName().isPresent()) + .collect(Collectors.toSet()); + + return all.stream() + .filter(p -> p.getScope().isEmpty() || !excludeScopes.contains(p.getScope().get())) + .filter(p -> p.getScope().isPresent() || !excludePackages.contains(p.getPackageName())) + .filter(p -> p.getScope().isEmpty() || !excludePackageAndScope.contains(new NpmPackageOrScope(p))) + .collect(Collectors.toList()); + } + + List packageOrScopes(String location) { + return FileReader.linesFromFile(location).stream() + .map(NpmPackageOrScope::new) + .collect(Collectors.toList()); + } + + static List packagesWithoutScope(List packages) { + return packages.stream() + .filter(p -> p.getScope().isEmpty()) + .collect(Collectors.toList()); + } +} diff --git a/src/main/java/com/schibsted/security/artishock/npm/NpmPackageIdentifier.java b/src/main/java/com/schibsted/security/artishock/npm/NpmPackageIdentifier.java new file mode 100644 index 0000000..471fb2a --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/npm/NpmPackageIdentifier.java @@ -0,0 +1,60 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.npm; + +import java.util.Objects; +import java.util.Optional; + +public class NpmPackageIdentifier { + private final Optional scope; + private final String packageName; + + public NpmPackageIdentifier(String scope, String packageName) { + this.scope = Optional.of(scope); + this.packageName = packageName; + } + + public NpmPackageIdentifier(String name) { + var parts = name.split("/"); + if (parts.length == 2) { + if (!parts[0].startsWith("@")) { + throw new RuntimeException(String.format("Scope '%s' must start with '@'", parts[0])); + } + this.scope = Optional.of(parts[0].substring(1)); + this.packageName = parts[1]; + } else if (parts.length == 1){ + this.scope = Optional.empty(); + this.packageName = parts[0]; + } else { + throw new RuntimeException(String.format("Expected at most one '/' to separate scope and package name in '%s'", name)); + } + } + + public Optional getScope() { + return scope; + } + + public String getPackageName() { + return packageName; + } + + @Override + public boolean equals(Object o) { + if (this == o) return true; + if (o == null || getClass() != o.getClass()) return false; + NpmPackageIdentifier that = (NpmPackageIdentifier) o; + return scope.equals(that.scope) && packageName.equals(that.packageName); + } + + @Override + public int hashCode() { + return Objects.hash(scope, packageName); + } + + @Override + public String toString() { + return scope.map(s -> "@" + s + "/" + packageName).orElse(packageName); + } +} diff --git a/src/main/java/com/schibsted/security/artishock/npm/NpmPackageOrScope.java b/src/main/java/com/schibsted/security/artishock/npm/NpmPackageOrScope.java new file mode 100644 index 0000000..062c621 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/npm/NpmPackageOrScope.java @@ -0,0 +1,74 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.npm; + +import java.util.Objects; +import java.util.Optional; + +public class NpmPackageOrScope { + private final Optional packageName; + private final Optional scope; + + public NpmPackageOrScope(String name) { + var parts = name.split("/"); + if (parts.length > 1) { + if (!parts[0].startsWith("@")) { + throw new RuntimeException(String.format("Scope '%s' must start with '@'", parts[0])); + } + this.scope = Optional.of(parts[0].substring(1)); + this.packageName = Optional.of(parts[1]); + } else { + if (parts[0].startsWith("@")) { + this.scope = Optional.of(parts[0].substring(1)); + this.packageName = Optional.empty(); + } else { + this.packageName = Optional.of(parts[0]); + this.scope = Optional.empty(); + } + } + } + + public NpmPackageOrScope(Optional scope, Optional packageName) { + this.scope = scope; + this.packageName = packageName; + } + + public NpmPackageOrScope(NpmPackageIdentifier npmPackageIdentifier) { + this.scope = npmPackageIdentifier.getScope(); + this.packageName = Optional.of(npmPackageIdentifier.getPackageName()); + } + + public Optional getPackageName() { + return packageName; + } + + public Optional getScope() { + return scope; + } + + @Override + public boolean equals(Object o) { + if (this == o) return true; + if (o == null || getClass() != o.getClass()) return false; + NpmPackageOrScope that = (NpmPackageOrScope) o; + return Objects.equals(packageName, that.packageName) && Objects.equals(scope, that.scope); + } + + @Override + public int hashCode() { + return Objects.hash(packageName, scope); + } + + @Override + public String toString() { + if (packageName.isPresent()) { + return scope.map(s -> "@" + s + "/" + packageName).orElse(packageName.get()); + } else if (scope.isPresent()) { + return "@" + scope.get(); + } else { + throw new IllegalStateException("Neither scope nor package name present"); + } + } +} diff --git a/src/main/java/com/schibsted/security/artishock/npm/client/NpmClient.java b/src/main/java/com/schibsted/security/artishock/npm/client/NpmClient.java new file mode 100644 index 0000000..d617e54 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/npm/client/NpmClient.java @@ -0,0 +1,115 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.npm.client; + +import com.schibsted.security.artishock.shared.CacheCategory; +import com.schibsted.security.artishock.shared.ConnectionInfo; +import com.schibsted.security.artishock.npm.NpmPackageIdentifier; +import com.schibsted.security.artishock.npm.NpmPackageOrScope; +import com.schibsted.security.artishock.shared.HttpClient; +import com.schibsted.security.artishock.shared.SimpleCache; +import java.io.IOException; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.function.Supplier; +import com.fasterxml.jackson.databind.DeserializationFeature; +import com.fasterxml.jackson.databind.ObjectMapper; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; + +public class NpmClient { + private static final Logger log = LogManager.getLogger(); + + private final ObjectMapper mapper; + + public NpmClient() { + mapper = new ObjectMapper().configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false); + } + + public List getPackageList(List packages, ConnectionInfo connectionInfo) { + log.info(() -> "Fetching select packages from " + connectionInfo.getPrefix()); + return new ArrayList<>(getNpmPackages(packages, connectionInfo).keySet()); + } + + public List notClaimedOrg(List scopes) { + var result = new ArrayList(); + for (var scope : scopes) { + if (!claimedOrgCached(scope)) { + result.add(new NpmPackageOrScope(Optional.of(scope), Optional.empty())); + } + } + + return result; + } + + boolean claimedOrgCached(String scope) { + var connectionInfo = new ConnectionInfo("https://www.npmjs.com"); + Supplier f = () -> Boolean.toString(fetchOrgIsClaimed(connectionInfo, scope)); + var result = SimpleCache.getFromCacheOrExecute(connectionInfo, scope, CacheCategory.ORG, f); + + if (result.equals("true")) { + return true; + } else if (result.equals("false")) { + return false; + } else { + throw new RuntimeException("Wrong code"); + } + } + + Map getNpmPackages(List packages, ConnectionInfo connectionInfo) { + var result = new HashMap(); + for (var packageName : packages) { + var npmjs = getPackageInfo(packageName, connectionInfo); + var p = convert(npmjs); + p.ifPresent(npmPackageInfo -> result.put(packageName, npmPackageInfo)); + } + + return result; + } + + Optional convert(ViewRaw viewRaw) { + if (viewRaw.error == null && viewRaw.errors == null) { + return Optional.of(new NpmPackageInfo(viewRaw)); + } else { + if (viewRaw.errors != null && viewRaw.errors.get(0).status == 404) { + return Optional.empty(); + } else if (viewRaw.error != null && (viewRaw.error.equals("Not found") || viewRaw.error.equals("not_found"))) { + return Optional.empty(); + } + + if (viewRaw.errors != null) { + throw new RuntimeException(String.format("Got error %s", viewRaw.errors.get(0).message)); + } else { + throw new RuntimeException(String.format("Got error %s", viewRaw.error)); + } + } + } + + ViewRaw getPackageInfo(NpmPackageIdentifier packageName, ConnectionInfo connectionInfo) { + try { + Supplier f = () -> fetchPackageInfo(packageName.toString(), connectionInfo); + var result = SimpleCache.getFromCacheOrExecute(connectionInfo, packageName.toString(), CacheCategory.PACKAGE_INFO, f); + + return mapper.readValue(result, ViewRaw.class); + } catch (IOException e) { + throw new RuntimeException(e); + } + } + + boolean fetchOrgIsClaimed(ConnectionInfo connectionInfo, String scope) { + return HttpClient.exists(connectionInfo, "/org/" + scope); + } + + private String fetchPackageInfo(String packageName, ConnectionInfo connectionInfo) { + return HttpClient.fetch(connectionInfo, "/" + packageName); + } + + public ConnectionInfo upstream() { + return new ConnectionInfo("https://registry.npmjs.org"); + } +} diff --git a/src/main/java/com/schibsted/security/artishock/npm/client/NpmPackageInfo.java b/src/main/java/com/schibsted/security/artishock/npm/client/NpmPackageInfo.java new file mode 100644 index 0000000..5ddc4bb --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/npm/client/NpmPackageInfo.java @@ -0,0 +1,91 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.npm.client; + +import java.time.ZonedDateTime; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Optional; +import java.util.stream.Collectors; + +public class NpmPackageInfo { + private final String name; + private final List releases; + private final ZonedDateTime created; + private final ZonedDateTime modified; + private final List maintainers; + + public NpmPackageInfo(ViewRaw viewRaw) { + if (viewRaw.errors != null) { + throw new RuntimeException(String.format("code %d, message %s", viewRaw.errors.get(0).status, viewRaw.errors.get(0).message)); + } + this.name = viewRaw.name; + + var releases = new ArrayList(); + for (var version : viewRaw.versions.keySet()) { + var time = ZonedDateTime.parse(viewRaw.time.get(version)); + releases.add(new NpmRelease(version, time)); + } + Collections.sort(releases); + this.releases = releases; + + this.created = ZonedDateTime.parse(viewRaw.time.get("created")); + this.modified = ZonedDateTime.parse(viewRaw.time.get("modified")); + + if (viewRaw.maintainers != null) { + this.maintainers = viewRaw.maintainers.stream() + .map(m -> m.name + " <" + m.email + ">") + .collect(Collectors.toList()); + } else { + this.maintainers = List.of(); + } + } + + public String getName() { + return name; + } + + public List getReleases() { + return releases; + } + + public ZonedDateTime getCreated() { + return created; + } + + public ZonedDateTime getModified() { + return modified; + } + + public List getMaintainers() { + return maintainers; + } + + public List majorVersions() { + return releases.stream() + .map(r -> r.getNpmVersion().getMajor()) + .distinct() + .sorted() + .collect(Collectors.toList()); + } + + public NpmVersion highestVersion() { + return releases.get(releases.size() - 1).getNpmVersion(); + } + + public Optional highestVersion(int majorVersion) { + var candidates = releases.stream() + .filter(r -> r.getNpmVersion().getMajor() == majorVersion) + .sorted() + .collect(Collectors.toList()); + + if (candidates.isEmpty()) { + return Optional.empty(); + } else { + return Optional.of(candidates.get(candidates.size() - 1).getNpmVersion()); + } + } +} diff --git a/src/main/java/com/schibsted/security/artishock/npm/client/NpmRelease.java b/src/main/java/com/schibsted/security/artishock/npm/client/NpmRelease.java new file mode 100644 index 0000000..2a87bcc --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/npm/client/NpmRelease.java @@ -0,0 +1,53 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.npm.client; + +import java.time.ZonedDateTime; + +public class NpmRelease implements Comparable { + private final String version; + private final NpmVersion npmVersion; + + private final ZonedDateTime time; + + public NpmRelease(String version, ZonedDateTime time) { + this.version = version; + + var dash = version.split("[-+]"); + if (dash.length > 1) { + // System.out.println("version has - or +, ignoring"); + } + + var parts = dash[0].split("\\."); + if (parts.length != 3) { + throw new RuntimeException("Unexpected version " + version); + } + + var major = Integer.parseInt(parts[0]); + var minor = Integer.parseInt(parts[1]); + var patch = Integer.parseInt(parts[2]); + this.npmVersion = new NpmVersion(major, minor, patch); + + this.time = time; + } + + public String getVersion() { + return version; + } + + public ZonedDateTime getTime() { + return time; + } + + public NpmVersion getNpmVersion() { + return npmVersion; + } + + @Override + public int compareTo(NpmRelease o) { + return this.npmVersion.compareTo(o.getNpmVersion()); + } +} + diff --git a/src/main/java/com/schibsted/security/artishock/npm/client/NpmSearchResult.java b/src/main/java/com/schibsted/security/artishock/npm/client/NpmSearchResult.java new file mode 100644 index 0000000..d48b6a7 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/npm/client/NpmSearchResult.java @@ -0,0 +1,41 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.npm.client; + +import java.time.ZonedDateTime; + +public class NpmSearchResult { + private final String name; + private final String scope; + private final String description; + private final ZonedDateTime date; + + public NpmSearchResult(SearchRaw.Package packageAlt) { + this.name = packageAlt.name; + this.scope = packageAlt.scope; // can be null + this.description = packageAlt.description; // can be null + if (packageAlt.date != null) { + this.date = ZonedDateTime.parse(packageAlt.date); + } else { + this.date = null; + } + } + + public String getName() { + return name; + } + + public String getScope() { + return scope; + } + + public String getDescription() { + return description; + } + + public ZonedDateTime getDate() { + return date; + } +} diff --git a/src/main/java/com/schibsted/security/artishock/npm/client/NpmVersion.java b/src/main/java/com/schibsted/security/artishock/npm/client/NpmVersion.java new file mode 100644 index 0000000..db2c0b6 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/npm/client/NpmVersion.java @@ -0,0 +1,46 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.npm.client; + +public class NpmVersion implements Comparable { + private final int major; + private final int minor; + private final int patch; + + public NpmVersion(int major, int minor, int patch) { + this.major = major; + this.minor = minor; + this.patch = patch; + } + + public int getMajor() { + return major; + } + + public int getMinor() { + return minor; + } + + public int getPatch() { + return patch; + } + + @Override + public int compareTo(NpmVersion o) { + if (this.major == o.getMajor()) { + if (this.minor == o.getMinor()) { + return Integer.compare(this.patch, o.getPatch()); + } else { + return Integer.compare(this.minor, o.getMinor()); + } + } else { + return Integer.compare(this.major, o.getMajor()); + } + } + + public String versionAsString() { + return major + "." + minor + "." + patch; + } +} diff --git a/src/main/java/com/schibsted/security/artishock/npm/client/SearchRaw.java b/src/main/java/com/schibsted/security/artishock/npm/client/SearchRaw.java new file mode 100644 index 0000000..2775061 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/npm/client/SearchRaw.java @@ -0,0 +1,31 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.npm.client; + +import java.util.List; +import com.fasterxml.jackson.annotation.JsonProperty; + +public class SearchRaw { + public List objects; + + public static class Object { + @JsonProperty("package") + public Package aPackage; + } + + public static class Package { + public String name; + public String scope; + public String date; + public String description; + public UsernameEmail publisher; + public List maintainers; + } + + public static class UsernameEmail { + public String username; + public String email; + } +} diff --git a/src/main/java/com/schibsted/security/artishock/npm/client/ViewRaw.java b/src/main/java/com/schibsted/security/artishock/npm/client/ViewRaw.java new file mode 100644 index 0000000..10e7f71 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/npm/client/ViewRaw.java @@ -0,0 +1,37 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.npm.client; + +import java.util.List; +import java.util.Map; + +public class ViewRaw { + public String name; + public Map versions; + public Map time; + public List maintainers; + public String error; + public List errors; + + public static class Version { + public String version; + public String name; + } + + public static class Repository { + public String type; + public String url; + } + + public static class Maintainers { + public String name; + public String email; + } + + public static class Error { + public int status; + public String message; + } +} diff --git a/src/main/java/com/schibsted/security/artishock/pypi/PyPiPackageIdentifier.java b/src/main/java/com/schibsted/security/artishock/pypi/PyPiPackageIdentifier.java new file mode 100644 index 0000000..914f24c --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/pypi/PyPiPackageIdentifier.java @@ -0,0 +1,37 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.pypi; + +import java.util.Objects; + +public class PyPiPackageIdentifier { + private final String packageName; + + public PyPiPackageIdentifier(String packageName) { + this.packageName = packageName; + } + + public String getPackageName() { + return packageName; + } + + @Override + public boolean equals(Object o) { + if (this == o) return true; + if (o == null || getClass() != o.getClass()) return false; + PyPiPackageIdentifier that = (PyPiPackageIdentifier) o; + return Objects.equals(packageName, that.packageName); + } + + @Override + public int hashCode() { + return Objects.hash(packageName); + } + + @Override + public String toString() { + return packageName; + } +} diff --git a/src/main/java/com/schibsted/security/artishock/pypi/Pypi.java b/src/main/java/com/schibsted/security/artishock/pypi/Pypi.java new file mode 100644 index 0000000..341a471 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/pypi/Pypi.java @@ -0,0 +1,104 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.pypi; + +import com.schibsted.security.artishock.artifactory.ArtifactoryClient; +import com.schibsted.security.artishock.config.Config; +import com.schibsted.security.artishock.pypi.client.PyPiClient; +import com.schibsted.security.artishock.shared.ConnectionInfo; +import com.schibsted.security.artishock.shared.FileReader; +import com.schibsted.security.artishock.shared.Intersection; +import java.util.ArrayList; +import java.util.List; +import java.util.Optional; +import java.util.stream.Collectors; + +public class Pypi { + private final ArtifactoryClient artifactoryClient; + private final PyPiClient pyPiClient; + private final Config config; + + public Pypi(Config config, ArtifactoryClient artifactoryClient) { + this.artifactoryClient = artifactoryClient; + this.pyPiClient = new PyPiClient(config); + this.config = config; + } + + public List excludeCandidates(String local, Optional trusted, Optional excluded) { + var candidates = pyPiClient.getAllPyPiPackageIdentifierFromIndex(local); + + var exclude = new ArrayList(); + trusted.ifPresent(s -> exclude.addAll(fromFile(s))); + excluded.ifPresent(s -> exclude.addAll(fromFile(s))); + + candidates.removeAll(exclude); + + return candidates; + } + + public List cached(String local, String remote) { + var localPackages = pyPiClient.getAllPyPiPackageIdentifierFromIndex(local); + var remoteCached = artifactoryClient.getAllPyPiPackageIdentifiersFromPyPiDir(remote + "-cache"); + + return Intersection.cacheIntersection(localPackages, remoteCached); + } + + public List inferredExclude(String local, String remote) { + var localPackages = pyPiClient.getAllPyPiPackageIdentifierFromIndex(local); + + var upstreamPackages = checkUpstream(localPackages); + var remotePackages = checkLocal(remote, localPackages); + + upstreamPackages.removeAll(remotePackages); + + return upstreamPackages; + } + + public List notClaimed(String local) { + var localPackages = pyPiClient.getAllPyPiPackageIdentifierFromIndex(local); + + var upstreamPackages = checkUpstream(localPackages); + localPackages.removeAll(upstreamPackages); + + return localPackages; + } + + List checkUpstream(List local) { + var result = new ArrayList(); + for (var l : local) { + if (existsUpstream(l)) { + result.add(l); + } + } + return result; + } + + List checkLocal(String repoName, List local) { + var result = new ArrayList(); + for (var l : local) { + if (existsInArtifactory(repoName, l)) { + result.add(l); + } + } + return result; + } + + boolean existsUpstream(PyPiPackageIdentifier packageIdentifier) { + return pyPiClient.packageExistsCached(pyPiClient.upstream(), packageIdentifier); + } + + boolean existsInArtifactory(String repo, PyPiPackageIdentifier packageIdentifier) { + return pyPiClient.packageExistsCached(new ConnectionInfo(config.getArtifactoryUrl() + "/api/pypi/" + repo, + config.getArtifactoryUsername(), + config.getArtifactoryPassword()), + packageIdentifier); + } + + List fromFile(String location) { + return FileReader.linesFromFile(location).stream() + .map(PyPiPackageIdentifier::new) + .collect(Collectors.toList()); + } +} diff --git a/src/main/java/com/schibsted/security/artishock/pypi/client/PyPiClient.java b/src/main/java/com/schibsted/security/artishock/pypi/client/PyPiClient.java new file mode 100644 index 0000000..891756a --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/pypi/client/PyPiClient.java @@ -0,0 +1,80 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.pypi.client; + +import com.schibsted.security.artishock.config.Config; +import com.schibsted.security.artishock.pypi.PyPiPackageIdentifier; +import com.schibsted.security.artishock.shared.CacheCategory; +import com.schibsted.security.artishock.shared.ConnectionInfo; +import com.schibsted.security.artishock.shared.HttpClient; +import com.schibsted.security.artishock.shared.SimpleCache; +import java.util.List; +import java.util.function.Supplier; +import java.util.stream.Collectors; +import com.fasterxml.jackson.databind.DeserializationFeature; +import com.fasterxml.jackson.dataformat.xml.XmlMapper; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; + +public class PyPiClient { + private static final Logger log = LogManager.getLogger(); + + private final Config config; + + public PyPiClient(Config config) { + this.config = config; + } + + public ConnectionInfo upstream() { + return new ConnectionInfo("https://pypi.org"); + } + + public List getAllPyPiPackageIdentifierFromIndex(String repositoryName) { + log.info(() -> "Fetching PyPi package from index in " + repositoryName); + + var raw = getPackagesFromIndex(new ConnectionInfo(config.getArtifactoryUrl() + "/api/pypi/" + repositoryName, + config.getArtifactoryUsername(), config.getArtifactoryPassword()), "/simple/"); + + var preprocessed = raw.replaceAll("\".*\"", "\"\"") + .replace("\n", "") + .replaceAll("
", "") + .replaceAll(".*", ""); + + var xmlMapper = new XmlMapper().configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false); + + try + { + var r = xmlMapper.readValue(preprocessed, SimpleRaw.class); + + // TODO consider filtering out names that are not allowed upstream + return r.body.stream() + .map(PyPiPackageIdentifier::new) + .collect(Collectors.toList()); + } catch (Exception e) { + throw new RuntimeException(e); + } + } + + public boolean packageExistsCached(ConnectionInfo connectionInfo, PyPiPackageIdentifier packageIdentifier) { + Supplier f = () -> Boolean.toString(packageExists(connectionInfo, packageIdentifier)); + var result = SimpleCache.getFromCacheOrExecute(connectionInfo, packageIdentifier.toString(), CacheCategory.PACKAGE_EXISTS, f); + + if (result.equals("true")) { + return true; + } else if (result.equals("false")) { + return false; + } else { + throw new RuntimeException(String.format("Must be 'true' or 'false' got '%s'", result)); + } + } + + public boolean packageExists(ConnectionInfo connectionInfo, PyPiPackageIdentifier packageName) { + return HttpClient.exists(connectionInfo, "/simple/" + packageName.getPackageName() + "/"); + } + + String getPackagesFromIndex(ConnectionInfo connectionInfo, String path) { + return HttpClient.fetch(connectionInfo, path); + } +} diff --git a/src/main/java/com/schibsted/security/artishock/pypi/client/SimpleRaw.java b/src/main/java/com/schibsted/security/artishock/pypi/client/SimpleRaw.java new file mode 100644 index 0000000..a331b34 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/pypi/client/SimpleRaw.java @@ -0,0 +1,11 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.pypi.client; + +import java.util.List; + +public class SimpleRaw { + public List body; +} diff --git a/src/main/java/com/schibsted/security/artishock/shared/CacheCategory.java b/src/main/java/com/schibsted/security/artishock/shared/CacheCategory.java new file mode 100644 index 0000000..ae08b55 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/shared/CacheCategory.java @@ -0,0 +1,22 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.shared; + +public enum CacheCategory { + PACKAGE_INFO("/package-info/"), + SEARCH("/search/"), + ORG("/org/"), + PACKAGE_EXISTS("/package-exists/"); + + private final String path; + + CacheCategory(String path) { + this.path = path; + } + + public String getPath() { + return path; + } +} diff --git a/src/main/java/com/schibsted/security/artishock/shared/ConnectionInfo.java b/src/main/java/com/schibsted/security/artishock/shared/ConnectionInfo.java new file mode 100644 index 0000000..f85ce71 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/shared/ConnectionInfo.java @@ -0,0 +1,40 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.shared; + +import java.net.URLEncoder; +import java.nio.charset.StandardCharsets; +import java.util.Optional; + +public class ConnectionInfo { + private final String prefix; + private final Optional username; + private final Optional password; + + public ConnectionInfo(String prefix, String username, String password) { + this.prefix = prefix; + this.username = Optional.of(username); + this.password = Optional.of(password); + } + + + public ConnectionInfo(String prefix) { + this.prefix = prefix; + this.username = Optional.empty(); + this.password = Optional.empty(); + } + + public String getPrefix() { + return prefix; + } + + public Optional getUsername() { + return username; + } + + public Optional getPassword() { + return password; + } +} diff --git a/src/main/java/com/schibsted/security/artishock/shared/FileReader.java b/src/main/java/com/schibsted/security/artishock/shared/FileReader.java new file mode 100644 index 0000000..2256722 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/shared/FileReader.java @@ -0,0 +1,21 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.shared; + +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; + +public class FileReader { + public static List linesFromFile(String location) { + var path = Path.of(location); + try { + return Files.readAllLines(Path.of(location)); + } catch (IOException e) { + throw new RuntimeException(String.format("Failed to read file at '%s'", path.toString()), e); + } + } +} diff --git a/src/main/java/com/schibsted/security/artishock/shared/HttpClient.java b/src/main/java/com/schibsted/security/artishock/shared/HttpClient.java new file mode 100644 index 0000000..e100a48 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/shared/HttpClient.java @@ -0,0 +1,94 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.shared; + +import java.io.IOException; +import java.util.concurrent.TimeUnit; +import okhttp3.Credentials; +import okhttp3.OkHttpClient; +import okhttp3.Request; +import okhttp3.Response; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; + +public class HttpClient { + private static final Logger log = LogManager.getLogger(); + + private static final OkHttpClient client = new OkHttpClient.Builder() + .connectTimeout(50, TimeUnit.SECONDS) + .readTimeout(300, TimeUnit.SECONDS) + .writeTimeout(50, TimeUnit.SECONDS) + .build(); + + public static Request prepareRequest(ConnectionInfo connectionInfo, String path) { + var url = connectionInfo.getPrefix() + path; + var requestBuilder = new Request.Builder() + .url(url); + + if (connectionInfo.getUsername().isPresent() && connectionInfo.getPassword().isPresent()) { + var credentials = Credentials.basic(connectionInfo.getUsername().get(), connectionInfo.getPassword().get()); + requestBuilder.addHeader("Authorization", credentials); + } else { + // External request: be polite + try { + Thread.sleep(1000); + } catch (InterruptedException e) { + // do nothing + } + } + + return requestBuilder.build(); + } + + public static Response execute(Request request) { + log.info(() -> "Fetching " + request.url()); + + try { + var response = client.newCall(request).execute(); + throwIfUnauthorized(response); + + return response; + } catch (IOException e) { + throw new RuntimeException("Failed to fetch " + request.url(), e); + } + } + + public static void throwIfUnauthorized(Response response) { + if (response.code() == 401) { + response.close(); + throw new RuntimeException("401 Unauthorized " + response.request().url()); + } + if (response.code() == 429) { + response.close(); + throw new RuntimeException("429 Too many requests " + response.request().url()); + } + } + + public static String fetch(ConnectionInfo connectionInfo, String path) { + var request = HttpClient.prepareRequest(connectionInfo, path); + + try (var response = HttpClient.execute(request)) { + return response.body().string(); + } catch (IOException e) { + throw new RuntimeException("Failed to get body from " + request.url()); + } + } + + /** + * Returns true if response is 200, false if response is 404, throws otherwise + */ + public static boolean exists(ConnectionInfo connectionInfo, String path) { + var request = HttpClient.prepareRequest(connectionInfo, path); + + try (var response = HttpClient.execute(request)) { + if (response.code() == 200) { + return true; + } else if (response.code() == 404) { + return false; + } + throw new RuntimeException(String.format("Expected code '200' or '404' for '%s', got '%d'", request.url(), response.code())); + } + } +} diff --git a/src/main/java/com/schibsted/security/artishock/shared/Intersection.java b/src/main/java/com/schibsted/security/artishock/shared/Intersection.java new file mode 100644 index 0000000..68071b4 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/shared/Intersection.java @@ -0,0 +1,29 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.shared; + +import java.util.HashSet; +import java.util.List; +import java.util.stream.Collectors; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; + +public class Intersection { + private static final Logger log = LogManager.getLogger(); + + public static List cacheIntersection(List local, List cached) { + var localSet = new HashSet<>(local); + var cachedSet = new HashSet<>(cached); + + var intersection = local.stream() + .filter(cachedSet::contains) + .collect(Collectors.toList()); + + log.info("Without cache " + (localSet.size() - intersection.size())); + log.info("With cache " + intersection.size()); + + return intersection; + } +} diff --git a/src/main/java/com/schibsted/security/artishock/shared/PackageSystem.java b/src/main/java/com/schibsted/security/artishock/shared/PackageSystem.java new file mode 100644 index 0000000..1181d49 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/shared/PackageSystem.java @@ -0,0 +1,42 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.shared; + +import java.util.HashMap; +import java.util.Map; +import java.util.Optional; + +public enum PackageSystem { + NPM("npm"), + MAVEN("maven"), + GRADLE("gradle"), + PYPI("pypi"); + + private final String alias; + + private static final Map map = new HashMap<>(); + static { + for (var v : values()) { + map.put(v.getAlias(), v); + } + } + + PackageSystem(String alias) { + this.alias = alias; + } + + public String getAlias() { + return alias; + } + + public static Optional fromString(String alias) { + return Optional.ofNullable(map.get(alias)); + } + + @Override + public String toString() { + return alias; + } +} diff --git a/src/main/java/com/schibsted/security/artishock/shared/SimpleCache.java b/src/main/java/com/schibsted/security/artishock/shared/SimpleCache.java new file mode 100644 index 0000000..dbda086 --- /dev/null +++ b/src/main/java/com/schibsted/security/artishock/shared/SimpleCache.java @@ -0,0 +1,92 @@ +/* + * Copyright 2021 Schibsted. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root. + */ + +package com.schibsted.security.artishock.shared; + +import com.schibsted.security.artishock.config.ConfigResolver; +import java.io.File; +import java.io.IOException; +import java.net.URLEncoder; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Instant; +import java.time.temporal.ChronoUnit; +import java.util.function.Supplier; + +public class SimpleCache { + public static boolean notInCache(String key, String repo, CacheCategory npmCacheCategory) { + var cacheFile = cacheFile(key, repo, npmCacheCategory); + + return !cacheFile.exists() || olderThanOneWeek(cacheFile); + } + + private static boolean olderThanOneWeek(File cacheFile) { + if (cacheFile.exists()) { + try { + var lastModified = Files.getLastModifiedTime(cacheFile.toPath()); + var ageInDays = ChronoUnit.DAYS.between(lastModified.toInstant(), Instant.now()); + + return ageInDays >= 7; + } catch (IOException e) { + throw new RuntimeException("Failed to find age of cache " + cacheFile); + } + } + + return false; + } + + private static File cacheFile(String key, String repo, CacheCategory npmCacheCategory) { + var urlEncodedKey = urlEncode(key); + var urlEncodedRepo = urlEncode(repo); + return ConfigResolver.configDirectory().resolve("cache/" + urlEncodedRepo + npmCacheCategory.getPath() + urlEncodedKey).toFile(); + } + + private static String urlEncode(String s) { + return URLEncoder.encode(s, StandardCharsets.UTF_8); + } + + public static String getFromCache(String key, String repo, CacheCategory npmCacheCategory) { + var path = cacheFile(key, repo, npmCacheCategory).toPath(); + try { + return Files.readString(path); + } catch (IOException e) { + throw new RuntimeException(String.format("Failed read cached data from '%s'", path), e); + } + } + + public static void putInCache(String key, String repo, CacheCategory npmCacheCategory, String content) { + var configDirectory = ConfigResolver.configDirectory(); + var urlEncodedRepo = urlEncode(repo); + createDirectoryIfNonexistent(configDirectory); + createDirectoryIfNonexistent(configDirectory.resolve("cache/")); + createDirectoryIfNonexistent(configDirectory.resolve("cache/" + urlEncodedRepo)); + createDirectoryIfNonexistent(configDirectory.resolve("cache/" + urlEncodedRepo + npmCacheCategory.getPath())); + + var path = cacheFile(key, repo, npmCacheCategory).toPath(); + try { + Files.writeString(path, content); + } catch (IOException e) { + throw new RuntimeException(String.format("Failed to write cache to '%s'", path)); + } + } + + public static void createDirectoryIfNonexistent(Path path) { + var cacheDirectory = path.toFile(); + if (!cacheDirectory.exists()) { + if (!cacheDirectory.mkdir()) { + throw new RuntimeException(String.format("Failed to create '%s'", cacheDirectory.toString())); + } + } + } + + public static String getFromCacheOrExecute(ConnectionInfo connectionInfo, String key, CacheCategory cacheCategory, Supplier f) { + var repo = connectionInfo.getPrefix(); + + if (SimpleCache.notInCache(key, repo, cacheCategory)) { + SimpleCache.putInCache(key, repo, cacheCategory, f.get()); + } + return SimpleCache.getFromCache(key, repo, cacheCategory); + } +} diff --git a/src/main/logo/NOTICE b/src/main/logo/NOTICE new file mode 100644 index 0000000..7401d4f --- /dev/null +++ b/src/main/logo/NOTICE @@ -0,0 +1 @@ +Artishock the Shocked Artichoke logo is based on the public domain work: https://commons.wikimedia.org/wiki/File:Artichoke.svg diff --git a/src/main/logo/artishock.svg b/src/main/logo/artishock.svg new file mode 100644 index 0000000..42abad8 --- /dev/null +++ b/src/main/logo/artishock.svg @@ -0,0 +1,2245 @@ + + + + + + + + image/svg+xml + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +