Local Spark Development With NetBeans or Eclipse

One of the features of Spark is that you can develop software on your local machine, and then later run it on a distributed system. This page describes how to set up Spark so that you can build and run on your laptop.

Java

For compatibility, make sure you have the Java 8 JDK installed http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

You can also download a combined package with both NetBeans and the JDK http://www.oracle.com/technetwork/java/javase/downloads/jdk-netbeans-jsp-142931.html

Spark Installation

Download Spark from http://spark.apache.org/downloads.html (Links to an external site.) Pick a directory where you want to keep your Spark installation, and unpack the downloaded .tgz file.

NetBeans

  • Create your project with Category set to "Java", and "Project" set to "Java Application".
  • After you finish creating your project, right click on the project name in the left-side Projects panel, and select "Properties" from the menu.
  • In the Properties dialog:
  • Select the Category "Libraries". Make sure that Java Platform is set to "JDK 1.8". Select "Compile" if not already selected.
    Click on "Add JAR/Folder". In the dialog, navigate to the "jars" folder in your Spark installation directory. Select all the .jar files (click on the first one, shift-click on the last one to select all). Make sure that "Absolute Path" is selected.
    Click "Choose" and then "OK".
    Your project is now ready to compile and run Spark Java programs.
  • To set the program arguments for running your Spark program, go back to the project Properties dialog, select "Run",

Eclipse

  • Set up for Eclipse is very similar as NetBeans.
  • Create a new project and give it a name and select "JAVA SE 1.8" or "JDK 1.8" for "Execution Environment JRE"
  • Right click on the project name, select "Build Path" and then "Add External Archives"
  • In the dialog, navigate to the "jars" folder in your Spark installation directory
  • Select all the .jar files (click on the first one, shift-click on the last one to select all).
  • Your project is now ready to run Spark Java programs
  • To set the program arguments, right click on project's name and go to "Properties" -> "Run/Debug Settings"
  • Select your main class and click "edit". In the dialog, click the "Arguments" tab and type your arguments separated by space in the "Arguments" box

Have Jenkins Run A Shell Script In A Remote Machine Every Time There Is A Change In The Git Repository

From Jenkins home page, click on "New Item" and select a "Freestyle Project" and fill in the general details such as "Project Name and Description"

Under the Source Code Management section, add your bitbucket url and set up the ssh credential so that Jenkins will be able to poll bitbucket after certain intervals.

To set up the ssh credential, generate a pair of public and private RSA key and put the public key in your bitbucket repo

and the private key in your Jenkins Source Code Management section by clicking on the "add" button which opens up a new window as shown below where you add in the ssh credential.

Then, we set up the Build Trigger section where we have Jenkins poll the repository we added in the earlier step. As shown in the figure, I am having it poll every 15 minutes.

Next thing we want to do is to step the Build Environment. In this step, we can set things that we want to do before and after the actual build step. I decided to run my test script before the build and the screenshot below shows the set up.

As you can see, you need to select the ssh server which is the remote machine where you want your script to run. To set up the ssh server, you need to go to the home page of Jenkins and select "Manage Jenkins" and select "Configure Jenkins" and set up your ssh server.

When you come back to this section after you set up your ssh server, you will see the name of your ssh server in the pull-down list. Here you can set up the ssh credential as password or key. I have set it as a password.


Since we are only running a shell script, there is nothing to build and that is the reason why my Build Section is empty.


The last step "Post-Build Section" is optional but you can use it to set up your email notification. But that's it. Following the above steps will run your bash script every time there is a new code in your bitbucket repository. Since github is also a git repository the steps should be exactly the same I think.