Skip to main content

Posts

Showing posts from 2013

Spark Tutorial and Cheatsheet

Main resources:
Scala Cheat Sheet
Reactive Cheat Sheet
Spark Cheat sheet
Spark Quick start
Spark programming guide
Spark Streaming: processing real-time data streams
Spark SQL and DataFrames: support for structured data and relational queries
MLlib: built-in machine learning library
GraphX: Spark’s new API for graph processing

Scala programming examples:

Define a object with main function -- Helloworld.
object HelloWorld { def main(args: Array[String]) { println("Hello, world!") } } Execute main function:
scala> HelloWorld.main(null) Hello, world! Creating RDDs
Parallelized Collections:
val data = Array(1, 2, 3, 4, 5) val distData = sc.parallelize(data) External Datasets:
val distFile = sc.textFile("data.txt") Above command returns the content of the file:
scala> distFile.collect() res16: Array[String] = Array(1,2,3, 4,5,6) SparkContext.wholeTextFiles can return (filename, content).
val distFile = sc.wholeTextFiles("/tmp/tmpdir") scala> distFile.collect…

Making Spray based ReST API publicly / remotely accessible

Recently I developed a demo Spray based ReST API and ran it on Windows.
While I was able to access it locally using http://localhost:port/service link, but remotely I was not.
I made following two changes to fix the issue:
1. In your application's application.conf file make hostname as 0.0.0.0
- 127.0.0.1 or localhost is foe listening only on the loopback interface.
- 0.0.0.0 for listening on all available network interfaces. This works in Java 7 and 8 fine. In Java 6, only IPv4 address works.
My application.conf looks like:
akka { loglevel = DEBUG event-handlers = ["akka.event.slf4j.Slf4jEventHandler"] } spray.can.server { request-timeout = 10s } service { host = 0.0.0.0 port = 8090 }
2. Open 8090 (or whatever you want to use) port in Windows Firewall: http://windows.microsoft.com/en-gb/windows/open-port-windows-firewall#1TC=windows-7

SecurityException with Bouncy Castle

I am using BouncyCastle library to encrypt the data transfer with Kakfa.
While my program run fine from IntelliJ, but when I package a fat JAR and run, it throws following exception:

java.lang.SecurityException: JCE cannot authenticate the provider BC at javax.crypto.Cipher.getInstance(Cipher.java:657) at javax.crypto.Cipher.getInstance(Cipher.java:596) at com.xyz.abc.fusioncell.utils.CryptoService$class.decrypt(CryptoService.scala:63) at com.xyz.abc.fusioncell.Boot$.decrypt(Boot.scala:13) at com.xyz.abc.fusioncell.conf.Config$$anonfun$userId$1.apply(Config.scala:28) at com.xyz.abc.fusioncell.conf.Config$$anonfun$userId$1.apply(Config.scala:28) at scala.util.Try$.apply(Try.scala:192) at com.xyz.abc.fusioncell.conf.Config$class.userId(Config.scala:28) at com.xyz.abc.fusioncell.Boot$.userId$lzycompute(Boot.scala:13) at com.xyz.abc.fusioncell.Boot$.userId(Boot.scala:13) at com.xyz.abc.fusioncell.Boot$.main(Boot.scala:24) at com.xyz.abc.fusio…