Scalatra tutorial part 4: Adding scalastyle

Scalastyle provides functionality to enforce style rules and best practices in Scala, similar to jslint, FXCop, or PMD. Setting it up is straightforward.

You can follow along here:
https://github.com/jieyangh/scalatra-sample-API with git hash 5d40618622e79835e870fac1533a90bbf9694dc3

First, we will modify plugins.sbt to add the scalastyle plugin and a resolver to help sbt find the plugin:

resolvers += "sonatype-releases" at "https://oss.sonatype.org/content/repositories/releases/"

addSbtPlugin("org.scalastyle" %% "scalastyle-sbt-plugin" % "0.6.0")

Next we add the scalastyle project settings to the project settings. The scalastyle default project settings will automatically cause the build task currently being executed to fail if scalastyle fails (scalastyleFailOnError):

  lazy val project = Project (
    "sampleAPI",
    file("."),
    settings = Defaults.defaultSettings
      ++ Seq(webSettings :_*)
      ++ Seq(org.scalastyle.sbt.ScalastylePlugin.projectSettings :_*)
      ++ Seq(
      libraryDependencies ++= Seq(
        "org.scalatra" %% "scalatra" % "2.2.2",
        "org.eclipse.jetty" % "jetty-webapp" % "8.1.7.v20120910" % "container,compile",
        "org.eclipse.jetty.orbit" % "javax.servlet" % "3.0.0.v201112011016",
        "ch.qos.logback" % "logback-classic" % "1.0.1",
        "org.scalatra" %% "scalatra-scalatest" % "2.2.2" % "test"
      )
    ) 
  )

For convenience, we can define our own custom build task “full” in build.sbt that does a clean, updates the packages, runs the tests and then runs scalastyle:

addCommandAlias("full", ";clean ;compile ;test ;scalastyle")

Before we run scalastyle we need to generate the config by first running:
./sbt scalastyleGenerateConfig

Now when we run ./sbt full we will see the scalastyle errors. You can see that it enforces some Scala best practices:

[info] scalastyle using config /Users/jiehu/scalatra-sample-API/scalastyle-config.xml
[warn] /Users/jiehu/scalatra-sample-API/src/main/scala/sampleApi/config/VersionInfo.scala:1: Header does not match expected text
[warn] /Users/jiehu/scalatra-sample-API/src/main/scala/sampleApi/config/VersionInfo.scala:37:0: Whitespace at end of line
[warn] /Users/jiehu/scalatra-sample-API/src/main/scala/sampleApi/config/VersionInfo.scala:27:7: Avoid using return
[warn] /Users/jiehu/scalatra-sample-API/src/main/scala/sampleApi/config/VersionInfo.scala:32:30: Avoid using return
[warn] /Users/jiehu/scalatra-sample-API/src/main/scala/sampleApi/config/VersionInfo.scala:16:28: Avoid using null
[warn] /Users/jiehu/scalatra-sample-API/src/main/scala/sampleApi/config/VersionInfo.scala: File must end with newline character
[warn] /Users/jiehu/scalatra-sample-API/src/main/scala/sampleApi/controllers/GreetingController.scala:1: Header does not match expected text
[warn] /Users/jiehu/scalatra-sample-API/src/main/scala/sampleApi/controllers/GreetingController.scala: File must end with newline character
[warn] /Users/jiehu/scalatra-sample-API/src/main/scala/sampleApi/controllers/HealthCheckController.scala:1: Header does not match expected text
[warn] /Users/jiehu/scalatra-sample-API/src/main/scala/ScalatraBootstrap.scala:1: Header does not match expected text
[warn] /Users/jiehu/scalatra-sample-API/src/main/scala/ScalatraBootstrap.scala:12:1: Whitespace at end of line
[warn] /Users/jiehu/scalatra-sample-API/src/main/scala/ScalatraBootstrap.scala: File must end with newline character

We can look up the rules here. I disabled the “Header does not match expected text” which is configured in scalastyle-config.xml file by setting enabled to false. This just checks if the predefined header appears at the start of every file. Since this is a simple tutorial project, there is no need for any such headers to exist. This makes the rule un-necessary.

 <check level="warning" class="org.scalastyle.file.HeaderMatchesChecker" enabled="false">

You can also turn off scalastyle in sections of code by putting them between //scalastyle:off and //scalastyle:on
It is best practice to fully specify the rule id being turned off //scalastyle:off magic.number This way it will only suppress that specific warning and not any others that may exist.

The rest of the checkin is just fixing the miscellaneous scalastyle errors. Once all of them have been fixed, the build succeeds!

NOTE: I had also tried using findbugs as well but found it didn’t play well with Scala. Findbugs works on Java bytecode. However when Scala compiles down to Java bytecode, it often results in Findbugs errors that aren’t really errors. For example, it will report that class names aren’t capitalized when in actuality they are, its just that traits apparently have lowercase class names in the java byte code.

Scalatra tutorial part 3

Part 1 and Part 2 are here. You can follow along on github at https://github.com/jieyangh/scalatra-sample-API. The git hash for part 3 is 92828543922dafb66b00bff03e9f115647747427 (simply type git checkout 92828543922dafb66b00bff03e9f115647747427 from the command line in your local repository). There’s a lot of code in this one, so I’ll just post mostly code snippets from the code on github.

Let’s make a simple health check API. To keep things simple, the API will consist of a single endpoint that reports the current application version. This can be used as a sanity check to quickly determine if the service is up and running and whether or not the right version was deployed. As more functionality is added to the API, the health check API can be expanded in scope to check connections to dependencies such as the database and other web services.

This feature can be broken down into three subtasks. First, we will modify the build script to store the version info inside a dynamically generated file. Second, we will create a singleton that is responsible for parsing this file and handling storage and retrieval of the version information. Once the first two steps are done, the third task is simple: We need to implement the actual API controller for the new endpoint.

1. Modifying the Build:
The build.sbt file we were using before allowed simple definitions of build tasks. By using a build.scala file, we have more freedom over what the build does, and use the full expressiveness of the scala programming language to specify build tasks. We will want the build to write out a text file with all the version information. The first difference between the previous build.sbt and build.scala is that we now define a project object, and set the library dependencies in there:

 lazy val project = Project (
    "sampleAPI",
    file("."),
    settings = Defaults.defaultSettings ++ Seq(webSettings :_*)++ Seq(
      libraryDependencies ++= Seq(
        "org.scalatra" %% "scalatra" % "2.2.2",
        "org.eclipse.jetty" % "jetty-webapp" % "8.1.7.v20120910" % "container,compile",
        "org.eclipse.jetty.orbit" % "javax.servlet" % "3.0.0.v201112011016",
        "ch.qos.logback" % "logback-classic" % "1.0.1",
        "org.scalatra" %% "scalatra-scalatest" % "2.2.2" % "test"
      )
    )
  )

The second difference is that we can now define various build tasks in the code. Here, we call

VersionHelper.BuildVersionsPlugin.buildVersionFile("src/main/resources/version.conf")

VersionHelper mixes in the sbt.Plugin trait, which basically is a way to create code modules that can be run during the build. VersionHelper will be responsible for creating the text file with all the version info. For simplicity sake, the version will simply be the major, minor, patch versions concatenated with the git sha. The major, minor, and patch versions will be hardcoded. The git sha is calculated in what is the least readable part of the code:

 private val sha1Matcher = """[0-9a-f]{5,40}""".r

  private val sha = sha1Matcher findFirstIn "git log --pretty=format:'%H' -1".!! 

  def gitSha: String = sha match {
    case Some(s) => s
    case None => "0000000000000000000000000000000000000000" //error
  }

The “.r” following the string creates a Regex using the magic of Scala implicits. This allows methods to be applied to objects that normally wouldn’t have them. I’m not a fan of them because it makes the code hard to read and follow, sacrificing maintainability for convenience. But this is the recommended way to create a Regex expression in Scala.

The regex is then run. findFirstIn is called as a postfix operator of the sha1Matcher Regex object. This is syntactic sugar. It matches with the string output by the expression: “git log –pretty=format:’%H’ -1″.!!

A note here: In Scala, operators are methods. To allow natural extensions to their language, Scala allows method names to take on various symbols reserved for operators such as “!”. What !! does is actually call scala.sys.process, which then runs the preceding string in the command line, returning the output. In our case, it gets the latest git sha. Again, this syntactic sugar sacrifices readability and allows people to abuse the language by making all sorts of ugly and unintuitive method names. However, this is part of the official Scala library. Finally, once we retrieve the gitSha, we do a pattern match on the result of the Regex match. This is an Option, which can either be Some or None. If there was no match, we just return an error string. The rest of the code in VersionHelper.scala writes out the version information to the version.conf file under /src/main/resources and is relatively straightforward in comparison with the snippet we just looked at. It can be found on github as linked in the start of this article.

The conf file will look something like the following:

 build {
   name = "Sample API"
   version = "1.1.0.92828543922dafb66b00bff03e9f115647747427"
   lastBuilt = "2015-01-08 05:37:22.401 UTC"
 }

Now that we have the configuration file, we will want to create a VersionInfo.scala class in our web application that reads from it. TypeSafe provides the config package that makes reading from .conf files easy. The VersionInfo.scala class can be found on github. To load the config, we use ConfigFactory.load as follows:

 val config = {
     try {
        ConfigFactory.load("version.conf")
     }
     catch {
       case ex:Exception => null
     }
   }

This will load version.conf from the classpath. Scala will be able to locate the version.conf underneath resources folder. The rest of the code in VersionInfo.scala then parses the file using a series of intuitive method calls that do exactly what you would expect them to:

   val name = loadConfig("build.name")
   val version = loadConfig("build.version")
   val lastBuilt = loadConfig("build.lastBuilt")

Each “.” in the string denotes a level of hierarchy in the conf file.

Finally, we get to the easy part. We create a controller that outputs the version info, and mount it to a heartbeat endpoint in the ScalatraBootstrap file. This is covered in part 1 of the Scalatra tutorial. All the code can be found on github as linked earlier. HealthCheckController is just a few lines and is trivial given that all the heavy lifting has been done.

Scalatra Tutorial (Part 2): Logging, debugging, and tests

Part 1 of this tutorial is here

You can follow along by getting the code from github here: https://github.com/jieyangh/scalatra-sample-API
The git hash for part 2 is 32190b1ae5eb685f6a06eaae6cd5fa15d5cf23bd

Now that we have a barebones implementation of a web API, let’s start adding on functionality and utility.

Debugging 

Remote debugging with IntelliJ makes life much easier. Here’s how to set it up. Open the project in IntelliJ and go to Run->Edit->Configurations. Choose Remote , hit the “+” button, and enter in a name for the configuration (I chose “ScalatraDebug” for simplicity). The default settings are fine and will set up a debug port on port 5005:

remote debug

We will need to modify the SBT parameters accordingly, since it defaults to a different port. Go to File->Settings->Other settings and select SBT from the side nav. Under IDE Settings, change the VM parameters so that its listening on port 5005:

-Xmx512M -XX:MaxPermSize=256M -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005

SBTDebug

Launch the application in the SBT console by typing “container:start” as we did before in part 1. Once the app is running, go to Run->Debug and select the remote debugging configuration that you previously created.

Logging

Any serious enterprise level application will need logging. Scalatra uses the logback framework by default, so that’s what we’ll use too. First we’ll need to add this dependency in the librarydependencies sequence in build.sbt:

"ch.qos.logback" % "logback-classic" % "1.0.1"

Next we will create a logback.xml under src/main/resources. Chances are if you’re reading this you’ve used logback before. If not, the file contents are fairly self explanatory. We can modify the pattern to suit our purposes, but including the date, thread, level, logger and actual message is a good a starting point as any:

<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>Captain's Log Stardate %date{ISO8601}. [%thread] %level %logger %msg
            </pattern>
        </encoder>
    </appender>

    <root level="debug">
        <appender-ref ref="STDOUT" />
    </root>
</configuration>

Finally, we’ll need to utilize the logger in our code. Let’s add some logging to the GreetingController scala class. First let’s create the logger:

val logger =  LoggerFactory.getLogger(getClass)

 

And now we just need to use our logger object to log things:

logger.info("We have entered a spectacular binary star system in the Kavis Alpha sector");

Again, pretty self explanatory. Here’s the full listing for the GreetingController:

package sampleApi.controllers

import org.scalatra.ScalatraServlet
import org.slf4j.{Logger, LoggerFactory}

class GreetingController extends ScalatraServlet {
  val logger =  LoggerFactory.getLogger(getClass)

  get("/") {
    logger.info("We have entered a spectacular binary star system in the Kavis Alpha sector");
    "Hello world"
  }

  get("/:name") {
    val name = params.getOrElse("name", "world")
    "Hello " + name
  }
}

Tests

Now we’ll want to add some automated testing. ScalaTest appears to be the go to test framework for Scala, and it provides support for Scalatra as well. This can be accomplished by adding a dependency for Scalatra ScalaTest in the libraryDependencies sequence in build.sbt:

"org.scalatra" %% "scalatra-scalatest" % "2.2.2" % "test"

Next we’ll want to create a test under src/test/scala. We’ll want our test class to extend the ScalatraSuite class, which allows us to use simple sentences to specify the expected behavior of our application. This is Behavior Driven Development (BDD). We also want our test class to have the “FunSuite” trait so that it can treat each test case as a function value. Note that in the official Scalatra testing example, the listed code didn’t compile for me, because it could not resolve “FunSuiteLike”. Replacing “FunSuiteLike” with “FunSuite” fixed the issue for me.

import org.scalatra.test.scalatest.ScalatraSuite
import org.scalatest.FunSuite
import sampleApi.controllers.GreetingController

class GreetingControllerTests extends ScalatraSuite  with FunSuite {
  addServlet(classOf[GreetingController], "/*")

  test("simple get") {
    get("/bob") {
      status should equal (200)
      body should include ("Hello bob")
    }
  }
} 

The test code is also pretty straightforward. An HTTP get request for “/bob” should return a status code of 200, and the response message should include “Hello bob”. You can run the test in IntelliJ by right clicking anywhere on the test code, or by creating a test configuration in the IDE and running it via the Run menu.

Scalatra tutorial (Part 1): Setting up a barebones “Hello world” REST API

I went through the exercise of setting up a Scalatra web API on my MacOS X for work. There are lots of guides out there, most of which use pre-existing projects on Github as a starting point. The official Scalatra documentation does this, although I ran into issues trying to follow their instructions (it seems to work fine when using Ubuntu but not with Mac). I used these guides as a reference, but I wanted to start from scratch and build out a simple GET endpoint that returned “Hello World”  using the least amount of code possible. The following is a guide on how to do exactly that.  The goal was to familiarize myself with how Scalatra APIs are structured and build upon that foundation.

I used Mac OS X (Mavericks) and IntelliJ (13.1) with the SBT and Scala plugins. The SBT plugin is optional, as you can use homebrew to install SBT and run the API from the command line. Most of the skeleton frameworks on github have an SBT bash script in the project root directory which unpacks and runs the SBT jar file (typically found under the project folder underneath the project root directory). Keep in mind that is why most documentation tells you to run the SBT commands using “./sbt”. The homebrew install adds SBT to your path, so you can simply type “sbt” instead.

If you want to follow along in github, you can check the sample project out at https://github.com/jieyangh/scalatra-sample-API. The hash for part 1 is 6c66a3df87ab321fa736f6ab0a819cb16d8e039e

Let’s get started!

Install Scala:

brew install scala

Install SBT (optional):

brew install sbt

SBT is a simple build tool, analagous to maven. Typically in most projects, there is a bash shell script with the same name in the project root that runs the sbt jar file (typically located in the project directory, and run via “./sbt”. If you install with brew or opt for a manual install, you can just type “sbt” instead). It is typically used  to generate the project idea file, update dependencies, and launch the application. However, the IntelliJ plugin makes this un-necessary though, so you can skip this step.

Install the Scala and SBT plugins for IntelliJ:

You’ll need to restart IntelliJ for the changes to take effect. Go to Preference->IDE Settings->Plugins->Browse Repositories. Search for “Scala” and “SBT” and install them both. The SBT plugin is optional, but it just makes life much easier.

scalaplugins

Create a new intelliJ project:

In the New Project dialog window, select Scala and choose “SBT”

newproj

Define the web.xml file:

Create a webapp/WEB-INF directory under src/main and add a web.xml file there. This file is a carry over from Java, which uses the web.xml file for servlet configuration. In Scalatra, web.xml is still utilized, but its role is much diminished. All that’s needed here is a definition for our listener class:

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns="http://java.sun.com/xml/ns/javaee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
version="3.0">
        <listener>
                <listener-class>org.scalatra.servlet.ScalatraListener</listener-class>
        </listener>
</web-app>

Configure the routes in ScalatraBootstrap.scala:

Create a ScalatraBootstrap.scala file under src/main/scala, which will contain the bulk of your application configuration code. The design philosophy seems to favor using code over XML, a pattern that we will see repeated later when we see how SBT handles plugins and dependencies. The ScalatraBootstrap file is where our route definitions will go. Since we’re keeping things simple, we will specify a “sample” endpoint underneath the website route and then map it the yet to be implemented servlet, GreetingController, that services it.

import javax.servlet.ServletContext
import org.scalatra.LifeCycle

import sampleApi.controllers.GreetingController

class ScalatraBootstrap extends LifeCycle {
  override def init(context: ServletContext) {
    // Mount servlets.
    context.mount(new GreetingController, "/sample/*")
  }
} 

Implement the GreetingController servlet:

Here is where we will implement our HelloWorld endpoints. Create a sampleApi.controllers package under src/main and add a GreetingController.scala. Note that its customary to put the API endpoints under a controllers package, but not necessary for the implementation.

package sampleApi.controllers
import org.scalatra.ScalatraServlet

class GreetingController extends ScalatraServlet {
  get("/") {
    "Hello world"
  }

  get("/:name") {
    val name = params.getOrElse("name", "world")
    "Hello " + name
  }
} 

Two HTTP GET routes are defined one that accepts a “name” query string parameter, and one that doesn’t. Both are mapped to the root of application (in our case this would be http://localhost:8080/)

Add dependencies in build.sbt:

Now edit the build.sbt file under the project root. Here is where we manage all our package dependencies. Modify build.sbt so that it contains the following code (you can safely override the default auto generated file provided by IntelliJ)

libraryDependencies ++= Seq(
  "org.scalatra" %% "scalatra" % "2.2.2",
  "org.eclipse.jetty" % "jetty-webapp" % "8.1.7.v20120910" % "container,compile",
  "org.eclipse.jetty.orbit" % "javax.servlet" % "3.0.0.v201112011016")

Seq(webSettings :_*)

libraryDependencies is a variable declared in keys.scala that stores all the managed dependencies. “++= Seq” appends our list of dependencies to the variable. Each dependency uses the same groupID % artifactID % revision format that Maven does. The main difference here is that instead of using XML to specify the dependencies in a pom file, Scala code is used instead. Again, the design philosophy seems to favor the expressiveness and flexibility of code over that of XML.

IntelliJ will prompt you to re-import your project. Enable auto-import to save you some clicks in the future:

reimport

Add Jetty plugin to SBT:
We are still missing one more dependency in order to get our application to run successfully. We will want to launch our application from SBT by running it inside a Jetty web container, which is basically just a wrapper for tomcat. SBT plugins are managed in the project/plugins.sbt file. Note that plugins.sbt manages all the SBT plugins, and build.sbt manages all the dependencies for the actual application itself. Add the following line:

addSbtPlugin("com.earldouglas" % "xsbt-web-plugin" % "0.7.0”)

Launch the application:

Your project structure should look like the following:
projectstructure

Go to View->Tool Windows->SBT Console. Hit the green arrow button to launch SBT. It will load the project definition and update the dependencies automatically. Once SBT has finished loading, you should see a “success” message followed by a command prompt. Type “container:start” to launch the jetty web server:

SBTConsole

You can now can open up a browser and point it to “localhost:8080/sample” and see “Hello world”. Congratulations!

References

http://ethanway.com/scalatra/

Great guide but it does some things that aren’t standard. Namely, it uses a pom file to manage dependencies and launches the application by manually spinning up a jetty server in code instead of using SBT.

https://github.com/laurilehmijoki/sbt-scalatra-skeleton

A simple HelloWorld API with JSON output, complete with a working test and instructions on how to launch the application.

http://www.scalatra.org/2.3/guides/deployment/configuration.html

Official documentation outlining how to setup your API