Skip to main content


Kafka MirrorMaker in Kafka

Check MirrorMaker.scala for more details.
Target cluster setupDownload and install Kafka (target cluster). Select appropriate version and download its tgz from Kafka Downloads page.tar -zxf kafka_2.11- cd kafka_2.11- Configure Target Kafka cluster's ZooKeepervim ./config/ # the directory where the snapshot is stored. dataDir=/work/kafka_2.11- # the port at which the clients will connect clientPort=2181 # disable the per-ip limit on thseparatedof connections since this is a non-production config maxClientCnxns=0 Start Target Kafka cluster's ZooKeeper./bin/ config/ Configure Target Kafka cluster's Servervim ./config/ # The id of the broker. This must be set to a unique integer for each broker. # The number of threads handling network requests # The number of threads doing disk I/O # The send buffer (SO_…

SecurityException with Bouncy Castle

I am using BouncyCastle library to encrypt the data transfer with Kakfa.
While my program run fine from IntelliJ, but when I package a fat JAR and run, it throws following exception:

java.lang.SecurityException: JCE cannot authenticate the provider BC
    at javax.crypto.Cipher.getInstance(
    at javax.crypto.Cipher.getInstance(
    at scala.util.Try$.apply(Try.scala:192)
Caused by: java.util.jar.JarException: file:/work/data_extractor/src/main/sh/data_extractor.jar has unsigned entries - scala/Array$$anonfun$apply$3.class
    at javax.crypto.JarVerifier.verifySingleJar(
    at javax.crypto.JarVerifier.verifyJars(
    at javax.crypto.JarVerifier.verify(
    at javax.crypto.JceSecurity.verifyProviderJar(
    at javax.crypto.JceSecurity.getVerificationResult(
    at javax.crypto.Cipher.getInstance(
    ... 11 more

First part of solution is to register the cryptographic service provider at runtime to ensure the configuration will work for everyone. You can use either of the Security.addProvider() or Security.insertProviderAt() methods:

if (Security.getProvider(BouncyCastleProvider.PROVIDER_NAME) == null)
    Security.addProvider(new BouncyCastleProvider)

Next, you need to add BouncyCastle provider to security providers in your java platform in two steps:
1. Copy BouncyCastle library (currently bcprov-jdk16-146.jar) to directory $JAVA_HOME/jre/lib/ext/
2. Register BouncyCastle provider: edit file $JAVA_HOME/jre/lib/security/ and add following line in list of providers:

Final list looks like:
# List of providers and their preference orders (see above):
Recompile and package the fat jar.