Skip to content

Commit

Permalink
Merge pull request #2 from sanori/dev
Browse files Browse the repository at this point in the history
Publish to SonaType repository
  • Loading branch information
sanori authored Nov 5, 2018
2 parents 7ddf6ac + c392972 commit 01596a0
Show file tree
Hide file tree
Showing 3 changed files with 57 additions and 6 deletions.
16 changes: 13 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ log formats are supported.
## How to use

### SQL (spark-sql)
When start spark-sql, include jar file of this project.
When start spark-sql:
```sh
spark-sql --jars accesslog_2.11-0.1.0-SNAPSHOT.jar
spark-sql --packages net.sanori.spark:access-log_2.11:0.1.0
```

In SQL, you can create user defined function and use it:
Expand Down Expand Up @@ -46,6 +46,16 @@ val logDs = lineDs
.as[CombinedLog]
```

When start spark-shell:
```sh
spark-shell --packages net.sanori.spark:access-log_2.11:0.1.0
```

In build.sbt:
```sbtshell
libraryDependencies += "net.sanori.spark" %% "access-log" % "0.1.0"
```

### RDD in Scala
```scala
import net.sanori.spark.accessLog.toCombinedLog
Expand Down Expand Up @@ -76,7 +86,7 @@ which has the following meaning:
```
sbt clean package
```
generates `accesslog_2.11-0.1.0-SNAPSHOT.jar` in `target/scala-2.11`.
generates `access-log_2.11-0.1.0.jar` in `target/scala-2.11`.

### Motivation
* To simplify analysis of web server logs
Expand Down
46 changes: 43 additions & 3 deletions build.sbt
Original file line number Diff line number Diff line change
@@ -1,9 +1,41 @@
ThisBuild / scalaVersion := "2.11.12"
ThisBuild / organization := "net.sanori"
ThisBuild / organization := "net.sanori.spark"
ThisBuild / organizationName := "SanoriNet"
ThisBuild / organizationHomepage := Some(url("http://sanori.github.io/"))

ThisBuild / scmInfo := Some(
ScmInfo(
url("https://github.com/sanori/spark-access-log"),
"scm:[email protected]:sanori/spark-access-log.git"
)
)
ThisBuild / developers := List(
Developer(
id = "sanori",
name = "Joo-Won Jung",
email = "[email protected]",
url = url("https://sanori.github.io/")
)
)
ThisBuild / licenses := Seq(
"Apache 2" -> new URL("http://www.apache.org/licenses/LICENSE-2.0.txt")
)
ThisBuild / homepage := Some(url("https://github.com/sanori/spark-access-log"))

ThisBuild / pomIncludeRepository := { _ => false }
ThisBuild / publishTo := {
val nexus = "https://oss.sonatype.org/"
if (isSnapshot.value)
Some("snapshots" at nexus + "content/repositories/snapshots")
else
Some("releases" at nexus + "service/local/staging/deploy/maven2")
}
ThisBuild / publishMavenStyle := true

lazy val root = (project in file("."))
.settings(
name := "accessLog",
name := "access-log",
version := "0.1.0",
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.2",
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.3.2",
libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.3.2",
Expand All @@ -12,4 +44,12 @@ lazy val root = (project in file("."))

// fork in Test := true
// javaOptions ++= Seq("-Xms512M", "-Xmx2048M", "-XX:MaxPermSize=2048M", "-XX:+CMSClassUnloadingEnabled")
// parallelExecution in Test := false
// parallelExecution in Test := false

credentials += Credentials(Path.userHome / ".sbt" / "sonatype_credential")

useGpg := true
// workaround for sbt/sbt-pgp#126
pgpSecretRing := pgpPublicRing.value
// To specify signing key
usePgpKeyHex("10CB79CA9DDE74FD")
1 change: 1 addition & 0 deletions project/plugins.sbt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
addSbtPlugin("com.jsuereth" % "sbt-pgp" % "2.0.0-M2")

0 comments on commit 01596a0

Please sign in to comment.