How to connect to Cassandra using WSO2 DSS

WSO2 Data Service Server supports exporting data as a web service, called a data service. It can connect to various data sources and expose those data to do operations on them. This post will guide you to use a Cassandra data source using WSO2 DSS.

1. Download Cassandra from here and install (You may have to change some configurations. Please refer this).
2. Create a keyspace, table and store some data
Now we will look into how to configure DSS
3.You can download DSS from here
4. Extract the pack and goto $DSS_HOME/bin folder and run sh wso2Server.sh (if you are a Windows user run wso2Server.bat file)
5. Add a data source. This step has been described in details here
6. You can test connection and save it
7. Deploy data service

The important thing that I would like to address in here is the versions compatibility of WSO2DSS and Cassandra jdbc drivers.

• The WSO2 DSS version older than 3.1.0 compatible with Cassandra JDBC 1.1.1. By default the Cassandra 1.1.x supports CQL (Common Query Language) 2. CQL3 is released with much more functionalities than CQL2 such as schema-free feature. 
• But you can connect to CQL3 by starting the Cassandra server with the following command ./cqlsh -3
• From WSO2 DSS (older version starting from 3.0.1) you have to specify the connection url as below. jdbc:cassandra://localhost:9160/keyspace_name?version=3.0.0
• If you do not specify the version in the URL it will connect to CQL2 by default
• But from DSS version 3.1.0 onward it supports Cassandra JDBC 1.2.x and because of that, it will connect to CQL3 by default.

Escape non printable characters in WSO2 DSS

WSO2 Dss can be used to expose data from different data sources (ex: MYSQL, ORACLE) as a web service. Escaping non printable characters is a feature introduced from WSO2 DSS 3.1.0 onward.

Lets say in your database there are characters which cannot be printed. But you cannot replace or delete them as those are the data you have. But if you want to retrieve those data through a web service it will not allow to do so as they cannot be represented. For an example when you are invoking a data service (using soap-ui or try-it) and do a select query for a data contains characters which are not in printable format it will throw the following exception.

ERROR {org.wso2.carbon.dataservices.core.description.query.SQLQuery} -  DS Fault Message: Error in XML generation at StaticOutputElement.execute
DS Code: UNKNOWN_ERROR
Nested Exception:-
com.ctc.wstx.exc.WstxIOException: Invalid white space character (0x5) in text to output
 {org.wso2.carbon.dataservices.core.description.query.SQLQuery}
DS Fault Message: Error in XML generation at StaticOutputElement.execute
DS Code: UNKNOWN_ERROR
Nested Exception:-
com.ctc.wstx.exc.WstxIOException: Invalid white space character (0x5) in text to output

 at org.wso2.carbon.dataservices.core.engine.StaticOutputElement.execute(StaticOutputElement.java:232)
 at org.wso2.carbon.dataservices.core.engine.OutputElementGroup.execute(OutputElementGroup.java:121)
 at org.wso2.carbon.dataservices.core.description.query.Query.writeResultEntry(Query.java:339)
 at org.wso2.carbon.dataservices.core.description.query.SQLQuery.processNormalQuery(SQLQuery.java:822)
 at org.wso2.carbon.dataservices.core.description.query.SQLQuery.runQuery(SQLQuery.java:2034)
 at org.wso2.carbon.dataservices.core.description.query.Query.execute(Query.java:254)
 at org.wso2.carbon.dataservices.core.engine.CallQuery.execute(CallQuery.java:161)
 at org.wso2.carbon.dataservices.core.engine.CallQueryGroup.execute(CallQueryGroup.java:118)
 at org.wso2.carbon.dataservices.core.description.operation.Operation.execute(Operation.java:73)
 at org.wso2.carbon.dataservices.core.engine.DataService.invoke(DataService.java:598)
 at org.wso2.carbon.dataservices.core.engine.DSOMDataSource.execute(DSOMDataSource.java:135)
 at org.wso2.carbon.dataservices.core.engine.DSOMDataSource.serialize(DSOMDataSource.java:147)
 at org.apache.axiom.om.impl.llom.OMSourcedElementImpl.internalSerialize(OMSourcedElementImpl.java:691)

This happened as those characters cannot be serialized to xml when they are going to be printed. But with this newly added feature you can define to escape the those characters with '?' when you define output mapping in the process of creating the data service. You can select "Escape Non Printable Characters" from the UI.

If it is selected the corresponding characters will be replaced by ? and if it is not, then it will throw the above exception. Find more from here. 

Analyze your IBM thread dump in Linux

Last week I had to analyze a heap dump taken from IBM VM. That heap dump was in .txt file and therefore it is readable. Part from the heap dump is shown below.

NULL           ------------------------------------------------------------------------
0SECTION       TITLE subcomponent dump routine
NULL           ===============================
1TISIGINFO     Dump Event "user" (00004000) received
1TIDATETIME    Date:                 2004/08/23 at 10:20:50
1TIFILENAME    Javacore filename:    /u/mcculls/test/javacore.20040823.102050.16908401.txt
NULL           ------------------------------------------------------------------------
0SECTION       GPINFO subcomponent dump routine
NULL           ================================
2XHOSLEVEL     OS Level         : z/OS 06.00
2XHCPUS        Processors -
3XHCPUARCH       Architecture   : s390x
3XHNUMCPUS       How Many       : 2

As the format of this heap dump is different from hprof file format this cannot be analyzed using either MAT analyzer (the eclipse plugin) or JProfiler because those prefer binary data to analyze.

There is a heap dump analyzer specially to the IBM VM called 'IBM Thread and Monitor Dump Analyzer for Java'. You can download the jar from here. (Please note that this version works with only JDK 7). Then you can simply run the jar file using this command

chanika@chanika-ThinkPad-T530:/apps/jdk1.7.0/bin$ ./java -jar /home/chanika/Downloads/jca445.jar

Then you will get a window as following

Click on File -> Open Thread Dumps -> Brows the thread dump file and select that.

Then that file will come under thread dump list and the when you select the dump file from the list some information in the header will appear as following screenshot

Then you can analyze the thread dump using options in 'Analyze' tab.  This is a sample thread dump analyze I have done.

Simultaneous read and write in Custom TCPMon

TCP Monitors are used to monitor messages passed through a TCP based communication. Apache TCPMon is one of the open source TCP Montor which is developed under Apache Software Licence. You can download TCPMon from here.

But when I used TCPMon to listen two way TCP communication by configuring the listen port and the  target port, I observed there is no simultaneous reading and writing for the two paths (from source to target and target to source).

Therefore a simple custom TCPMon is written to facilitate simultaneous read and write. Here is the approach I followed.

To listen from a port we can create a new Socket connection, configure the port and accept data. Here I have taken the listening port as 2389.

public static void main(String[] args) throws Exception {
        ServerSocket srcSocket = new ServerSocket(2389);
        while (true) {
            new Thread(new TCPServer(srcSocket.accept())).start();
        }
    }

It requires what is the target port that data should be sent to. Here the target port is 3456.

Socket targetSock = new Socket("localhost", 3456);

Then we can have input stream and the output stream from both socket connections.

        srcIn = srcSocket.getInputStream();
        srcOut = srcSocket.getOutputStream();
        targetIn = targetSock.getInputStream();
        targetOut = targetSock.getOutputStream();

Then we should pipe data from the input stream to output stream to have no data loss during communication.

private class Pipe extends Thread {

        private InputStream in;

        private OutputStream out;

        public Pipe(InputStream in, OutputStream out) {
            this.in = in;
            this.out = out;
        }

        public void run() {
            try {
                byte[] buff = new byte[512];
                int i;
                while (true) {
                    i = in.read(buff);
                    if (i > 0) {
                        out.write(buff, 0, i);
                        System.out.print(".");
                    }
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
        }

    }

In order to have simultaneous data writing and reading operations two threads should be started.

        new Pipe(srcIn, targetOut).start();
        new Pipe(targetIn, srcOut).start();

Now our TCPMon works fine with simultaneous data passing.

The full code is shown below.

import java.io.*;
import java.net.ServerSocket;
import java.net.Socket;

public class TCPServer implements Runnable {

    private InputStream srcIn, targetIn;

    private OutputStream srcOut, targetOut;

    public TCPServer(Socket srcSocket) throws Exception {
        Socket targetSock = new Socket("localhost", 3456);
        srcIn = srcSocket.getInputStream();
        srcOut = srcSocket.getOutputStream();
        targetIn = targetSock.getInputStream();
        targetOut = targetSock.getOutputStream();
    }

    public void run() {
        new Pipe(srcIn, targetOut).start();
        new Pipe(targetIn, srcOut).start();
    }

    private class Pipe extends Thread {

        private InputStream in;

        private OutputStream out;


        public Pipe(InputStream in, OutputStream out) {
            this.in = in;
            this.out = out;
        }

        public void run() {
            try {
                byte[] buff = new byte[512];
                int i;
                while (true) {
                    i = in.read(buff);
                    if (i > 0) {
                        out.write(buff, 0, i);
                        System.out.print(".");
                    }
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
        }

    }

    public static void main(String[] args) throws Exception {
        ServerSocket srcSocket = new ServerSocket(2389);
        while (true) {
            new Thread(new TCPServer(srcSocket.accept())).start();
        }
    }
}

How to add log files using log4net

log4net is an open source library which is developed under Apache License. If you are developing an .NET web application and if you want to keep a log file, og4net is the best way to do so. You can simply follow following steps.

1. First you have to download Log4Net. You can download it from here.
2. Open the visual studio create a new .net web application or open an existing one.
3. Add log4net.dll to reference in your web application. You can find dll file from ~\log4net-1.2.11\bin\net\4.0\release\log4net.dll in the extracted folder.
4. Edit the web.config file in order to configure log4net by putting this code.

<configuration>
<configSections>
<section name=”log4net” type=”log4net.Config.Log4NetConfigurationSectionHandler,Log4net”/>
</configSections>

5. Add an xml file to do some configurations regarding log4net. (Including the maximum size of the log file, rolling style etc). The code is shown below.Save it as log4net.xml

<?xml version=”1.0″ encoding=”utf-8″ ?>

<log4net>
<root>
<level value=”DEBUG” />
<appender-ref ref=”LogFileAppender” />
</root>
<appender name=”LogFileAppender” type=”log4net.Appender.RollingFileAppender” >
<param name=”File” value=”.\Logs\log.txt” />
<param name=”AppendToFile” value=”true” />
<rollingStyle value=”Size” />
<maxSizeRollBackups value=”10″ />
<maximumFileSize value=”10MB” />
<staticLogFileName value=”true” />
<layout type=”log4net.Layout.PatternLayout”>
<param name=”ConversionPattern” value=”%-5p%d{yyyy-MM-dd hh:mm:ss} – %m%n” />
</layout>
</appender>
</log4net>

6. Then add the following code into Global.asax, to configure log4net.xml file within the Application_Start() method.

log4net.Config.XmlConfigurator.Configure(new System.IO.FileInfo(Server.MapPath(“log4net.xml”)));

7. After that you can update log files by enabling which actions should be written. It can be any action in the code. For an example you can add following code to the Index action in Default controller.

public class DefaultController : BaseController

{
protected static readonly ILog log = LogManager.GetLogger(“root”);
//
// GET: /Default1/

public ActionResult Index(string fmt)
{
log4net.Config.XmlConfigurator.Configure();
log.Debug(“Action:- ” + this.ControllerContext.Controller.ToString() + “”);

return GetFormatView(“Home/Home.aspx”);
}
}