Taming the bits / Domando los bits

I present here a small tip for improving the IntellJ IDEA Terminal window.
Under Linux, it doesn't execute the ~/.bashrc file at startup. so it doesn't have the correct PATH and other important settings like environment variables.

My first attempt was to create a ~/.profile script that called my ~/.bashrc. It worked, but whenever I rebooted my computer, I couldn't log in. My KDE desktop complains. I have to open a console session in order to fix it.

Luckily, I've found a simple solution that works, and doesn't have this nasty secondary effect.

Open the IDEA settings, as shown in the following screenshot:

The important setting is to add the -i parameter to bash. In this way, the shell becomes interactive, and executes .bash_rc, like a regular console Window.

The problem

I need to select in a Jenkins job a branch of a GIT repository to work on.

Possible solutions

The git-parameter plugin of Jenkins

This standard Jenkins plugin called git-parameter does the job, but it has these limitations (at least in May 2015):

- If we clean the workspace, it is unable to find the branches.
- Branches aren't sorted

Let's see if we can see other solutions.

Solution based on Groovy scripting

This solution needs a slight modification to the server, supposing you are using gitolite for authorizing the access to your repository.

Modifying your gitolite installation

Following this guide, I have added a new command called branch.

For it to work, I log in to the computer that hosts the gitolite server and edit the .gitolite.rc file:

ssh root@myserver.com
su - git
vim ~/.gitolite.rc

and add this line:

LOCAL_CODE => "$ENV{HOME}/local",
ENABLE => [
  'branch',
  ...
]

then I create the 'branch' command:

mkdir -p local/commands
cd local/commands
vim branch

and place this content:

#!/bin/bash

cd ~/repositories/$1.git &&
git branch

Let's make it executable:

chmod +x branch

Now let's test it:

gitolite branch myrepo

and remotely:

ssh git@myserver.com branch myrepo

It is left as an improvement, to control if the user has access to the repository.

Create the parameter

Now let's configure our Jenkins job to include a parameter.

I use the Extensible choice plugin in order to have a dynamically filled combo box.

I configure to have a source of System Groovy Choice parameter, and enter this small Groovy script:

[ "ssh", "git@myserver.com", "branch myrepo"].execute().text.tokenize("\n").collect() {
  it.trim().replace("*", "")
}

git highlights the current repo with a star, and places some space at the left for aesthetical purposes.

If the parameter has a name of branch, then in the Branches to build field you can simply type origin/$branch.

This solution is simple, and doesn't have all the limitations of the previous solution.

The problem

In an app that uses JVM (Java Virtual Machine), it is normal to use 3rd party code, called dependencies.
Sometimes, some bad-behaved jars include 3rd party dependencies embedded, or the same jar can be found under different organizations. In this case, build tools like SBT, Gradle or Maven are not able to evict, and choose only the latest one.

If there are 2 different implementations of the same class in the same JVM, it is random which one will be used, and this can produce nasty bugs like it works in my computer but not in yours. It can be hard to diagnose this kind of problem.

Possible solutions

In Maven, I've used successfully the duplicate-finder-maven-plugin plugin. But after searching for SBT plugins, I've found none.

At first, I wanted to create a new plugin, but a significant part of the code is shared with the sbt-pack plugin. So, I've decided to fork it, and send a pull request to the original author.

sbt-pack plugin

For that purpose, I've modified the sbt-pack plugin, and added a new task called checkDuplicatedDependencies for detecting it.

The checkDuplicatedDependencies will fail and provide a list of all incompatibilites, in case it finds the same class in 2 different jars.

If the same class is duplicated, but it's the same implementation, it won't complain.

This is detected by computing a MD5 hash of the contents of the file.

If you consider the found conflicts are inofensive, in order to ignore them in a future run, use the checkDuplicateExclude setting. The value of this setting is automatically given.

Here is an example of report:

> checkDuplicatedDependencies
Conflict between org.slf4j:jcl-over-slf4j:1.7.10 and commons-logging:commons-logging:1.1.3:
  org/apache/commons/logging/impl/NoOpLog
  org/apache/commons/logging/impl/SimpleLog$1
  org/apache/commons/logging/impl/SimpleLog
  org/apache/commons/logging/Log
Conflict between commons-collections:commons-collections:3.2.1 and commons-beanutils:commons-beanutils-core:1.7.0:
  org/apache/commons/collections/ArrayStack
  org/apache/commons/collections/BufferUnderflowException
  org/apache/commons/collections/FastHashMap$1
Conflict between org.eclipse.birt.runtime:org.eclipse.birt.runtime:4.3.1 and commons-codec:commons-codec:1.6:
  org/apache/commons/codec/binary/Base32
  org/apache/commons/codec/binary/Base32InputStream
  org/apache/commons/codec/binary/Base32OutputStream
  org/apache/commons/codec/binary/Base64
  org/apache/commons/codec/binary/Base64InputStream
  org/apache/commons/codec/binary/Base64OutputStream
  org/apache/commons/codec/binary/BaseNCodec
Conflict between xerces:xercesImpl:2.9.1 and org.python:jython-standalone:2.5.2:
  org/w3c/dom/html/HTMLDOMImplementation
Conflict between commons-collections:commons-collections:3.2.1 and commons-beanutils:commons-beanutils:1.8.3:
  org/apache/commons/collections/ArrayStack
  org/apache/commons/collections/Buffer
  org/apache/commons/collections/BufferUnderflowException
  org/apache/commons/collections/FastHashMap$1
  org/apache/commons/collections/FastHashMap$CollectionView$CollectionViewIterator
Conflict between org.python:jython-standalone:2.5.2 and com.google.guava:guava:15.0:
  com/google/common/base/package-info
  com/google/common/collect/package-info
  com/google/common/io/package-info
  com/google/common/net/package-info
  com/google/common/primitives/package-info
  com/google/common/util/concurrent/package-info
Conflict between org.eclipse.birt.runtime:org.eclipse.osgi:3.9.1.v20130814-1242 and org.eclipse.birt.runtime:org.eclipse.osgi.services:3.3.100.v20130513-1956:
  org/osgi/service/log/LogService
  org/osgi/service/log/LogListener
  org/osgi/service/log/LogEntry
  org/osgi/service/log/LogReaderService
Conflict between commons-beanutils:commons-beanutils:1.8.3 and commons-beanutils:commons-beanutils-core:1.7.0:
  org/apache/commons/beanutils/BasicDynaBean
  org/apache/commons/beanutils/BasicDynaClass
  org/apache/commons/beanutils/BeanAccessLanguageException
Conflict between javax.mail:mail:1.4.1 and com.sun.mail:javax.mail:1.5.1:
  com/sun/mail/handlers/image_gif
  com/sun/mail/handlers/image_jpeg
  com/sun/mail/handlers/message_rfc822
  javax/mail/Address
  javax/mail/AuthenticationFailedException
  javax/mail/Authenticator
  javax/mail/BodyPart
  ...

If you consider these conflicts are inofensive, in order to ignore them, use:
set checkDuplicatedExclude := Seq(
  "org.slf4j" % "jcl-over-slf4j" % "1.7.10" -> "commons-logging" % "commons-logging" % "1.1.3",
  "commons-collections" % "commons-collections" % "3.2.1" -> "commons-beanutils" % "commons-beanutils-core" % "1.7.0",
  "org.eclipse.birt.runtime" % "org.eclipse.birt.runtime" % "4.3.1" -> "commons-codec" % "commons-codec" % "1.6",
  "xerces" % "xercesImpl" % "2.9.1" -> "org.python" % "jython-standalone" % "2.5.2",
  "commons-collections" % "commons-collections" % "3.2.1" -> "commons-beanutils" % "commons-beanutils" % "1.8.3",
  "org.python" % "jython-standalone" % "2.5.2" -> "com.google.guava" % "guava" % "15.0",
  "org.eclipse.birt.runtime" % "org.eclipse.osgi" % "3.9.1.v20130814-1242" -> "org.eclipse.birt.runtime" % "org.eclipse.osgi.services" % "3.3.100.v20130513-1956",
  "commons-beanutils" % "commons-beanutils" % "1.8.3" -> "commons-beanutils" % "commons-beanutils-core" % "1.7.0",
  "javax.mail" % "mail" % "1.4.1" -> "com.sun.mail" % "javax.mail" % "1.5.1"
)

[error] (checkDuplicatedDependencies) Detected 424 conflict(s)

As you can see, clashes are common for big projects like mine that has about 300 dependencies.

The problem

If you're working in Linux, with IntelliJ IDEA embedded Terminal, .bashrc isn't called.

The solution

Create in your home directory this simple script, and it must be named .profile:

if [ "$SHELL" = "/bin/bash" ];then
    . ~/.bashrc
fi   

The problem

We have a Java or Scala app that has some dependencies and one entry point.
We would like to:

• have all needed jars in a directory call i.e. lib.
• be able to start our app by simply executing "java -jar myapp.jar".

Suppose that:

• we have the a multiproject build.sbt and
• the project myapp holds the entry point mycompany.myapp.MainClass

then this would the our build.sbt would be something like this:

lazy val myapp = project.settings(Seq(
    mainClass in (Compile, packageBin) := Some("mycompany.myapp.MainClass"),
  // Remove version from the artifact
 artifactName in (Compile, packageBin) := { (sv: ScalaVersion, module: ModuleID, artifact: Artifact) =>
        "myapp.jar"
    },
    packageOptions in (Compile, packageBin) +=
        Package.ManifestAttributes(java.util.jar.Attributes.Name.CLASS_PATH -> 
            ((dependencyClasspath in Compile).value
                .map{_.data.getName}
                .filter(!_.startsWith("scala-"))
                .mkString(" "))
        ),
    projectDependencies ++= Seq(
     "com.somecompany" % "dependency1" % "1.0"
  )
): _*)

We are filtering scala-library.jar as this app is Java-based. SBT always adds scala-library.jar.

In this way, all the needed classpath is already coded in the MANIFEST.MF and it's easier to launch our app.

This has been tested with SBT 0.13.7.

If your use recent versions of Java, another alternative is to use:

java -cp '*' mycompany.myapp.MainClass

Suppose you want to provide externally some setting in SBT.

You could create an input task or you can set setting from the command line.

Now, let's explain how to do the 2nd choice.

Suppose you have this in your build.sbt file:

val buildNumber = settingKey[Int]("Build number")
// Provide some default value
buildNumber := 0

Then from e.g. Jenkins, you could set the build number in this way:

sbt "set buildNumber:=5" "show buildNumber"

In this example we have shown its current value, but we could you something more practical like package.

My requirements are:
- Low cost or free, as my purpose is earning no money with it.
- Easy to learn and use
- Easy to insert snippets of code

My starting point for choosing has been this article.

Some candidates:

• Free Wordpress doesn't allow any plugins. You need some hosting if you want enough power.
• I've tried a little Weebly, but it isn't suitable for sharing code.
• Jekyll is powerful enough, it's integrated with github, and has a great community, but you cannot use any computer. You need to have installed git, Ruby. It uses familiar MarkDown syntax.
• Finally I've chosen Logdown, as:
  • it provides more visual feedback,
  • it also uses MarkDown syntax.
    
  • the free edition covers my needs.
  • I have always the choice of exporting my blog elsewhere.

I'm in the process of migrating a complex build process involving C++, Python, Java, Javascript and Scala code from Ant, Maven and shell scripts to SBT.

What I like of SBT is:

• its simplicity combined with power. I don't have to resort to write code in external fashion.
• interactive shell and tab completion for free
• much more compact than Maven.
• Maven doesn't know much about dependencies. If I run project A that depends on library B, and I've changed some source file in B, Maven doesn't recompile B.
• Good integration with Eclipse and IDEA, our standard IDE's.

I've used successfully before SBT in the Android world, even though Gradle is more used.

I find plugins for nearly all my needs.