Riak NoSQL database on Mac OS-X 10.9

   

In world lot of databases available in nowadays, Riak is most power full opensource database from NoSQL database category.In web based application I have tried to implement Riak database. It is hard to install and configure  at first time in my linux system, because that time I am new to this DB. Riak is written in Erlang, known for its ability to distribute data across nodes using consistent hashing in a simple key/value scheme in namespaces called buckets.

Main Advantages of Riak

Fault-Tolerant Availability
    Riak replicates key/value stores across a cluster of nodes with a default n_val of three. In the case of node outages due to network partition or hardware failures, data can still be written to a neighboring node beyond the initial three, and read-back due to its "masterless" peer-to-peer architecture.

Queries
    Riak supports a REST API through HTTP and Protocol Buffers for basic PUT, GET, POST, and DELETE functions. Additional query choices are offered including secondary indices, Riak Search leveraging the Apache Solr Engine with Solr client query APIs and MapReduce. MapReduce has native support for both JavaScript (using the SpiderMonkey runtime) and Erlang.

Predictable Latency
    Riak evenly distributes data across nodes with consistent hashing and can provide an excellent latency profile, even in the case of multiple node failures. Key/Values can be stored in memory, disk, or a combination depending on which pluggable backend one chooses.

Multi-Datacenter Replication
    In multi-datacenter replication, one cluster acts as a "primary cluster". The primary cluster handles replication requests from one or more "secondary clusters" (generally located in other regions or countries). If the datacenter with the primary cluster goes down, a secondary cluster can take over as the primary cluster.
    There are two primary modes of operation: fullsync and realtime. In fullsync mode, a complete synchronization occurs between primary and secondary cluster(s), by default every 360 minutes. In realtime mode, continual, incremental synchronization occurs - replication is triggered by new updates.

Install Riak from Source

    Let's install Riak on Mac OS-X 10.9 Maverick and build a five-node cluster running on your local machine.
 Instructions for downloading and installing Riak from source.

Impotent thing is, You must have Xcode tools installed from Apple's Developer website

Riak requires Erlang R15B01. Installation of Erlang R15B01 from source..

First download and unpack the source:

curl -O http://erlang.org/download/otp_src_R15B01.tar.gz
tar zxvf otp_src_R15B01.tar.gz
cd otp_src_R15B01

Then configure Elang using LLVM (the default):

 CFLAGS=-O0 ./configure --disable-hipe --enable-smp-support --enable-threads \
--enable-kernel-poll --enable-darwin-64bit

Now build and install, you will prompted for your sudo password.

make && sudo make install

Download and unpack Riak-1.4.8

After installing Erlang, We are going to download and unpack the source distribution of Riak-1.4.8.

wget http://s3.amazonaws.com/downloads.basho.com/riak/1.4/1.4.8/riak-1.4.8.tar.gz
tar zxvf riak-1.4.8.tar.gz
cd riak-1.4.8 
make all

Setup and startup Five nodes

After successful built,  we are going to setup five self-contained nodes on Riak. 

First, make five nodes:

make devrel DEVNODES=5

You have just generated a dev directory. Let's go into that directory to check out its these directories dev1, dev2, dev3, dev4:

cd dev; ls

Each directory starting with dev is a complete, self-contained package containing a Riak node. We need to start all nodes,  you can use a for loop to iterate through and start the available nodes::

 for node in `ls`; do $node/bin/riak start; done

--ulimit warning--

At this point you may receive a warning message to increase the number of open file handles (ulimit). 

To check the current limits:

launchctl limit maxfiles

 To adjust the maximum open file limits,   edit(create) /etc/launchd.conf (using any editor with sudo previllage)and increase the limits for soft and hard as appropriate. add the lines as follows:

limit maxfiles 16384 32768

Save the file, and restart the system for the new limits to take effect. After restarting,
verify current limits:

launchctl limit

    cpu         unlimited      unlimited
    filesize    unlimited      unlimited
    data        unlimited      unlimited
    stack       8388608        67104768
    core        0              unlimited
    rss         unlimited      unlimited
    memlock     unlimited      unlimited
    maxproc     709            1064
    maxfiles    16384          32768

After that start the all five nodes again using above for loop command and check running nodes, to do this run : 

ps aux | grep beam

This should give you granular details on the five running Riak nodes.  If you'd like to simply check which nodes are running and which are not, you can run the riak ping command on for loop:

 for node in `ls`; do $node/bin/riak ping; done

If the response is PONG, then the node is up and running. Otherwise, the node is currently stopped

Cluster joining

The next step is to join these five nodes together to form a cluster. You can do this using the Riak Admin tool. Specifically, what we want to do is join dev2,dev3, dev4, and dev5 to dev1: , to do this run:

 for n in {2..5}; do dev$n/bin/riak-admin cluster join dev1@127.0.0.1; done

To make the above joins take effect, you first must review the plan:

 dev1/bin/riak-admin cluster plan

The plan will print out a synopsis of what it plans to do, and how the cluster will look after this is completed.

 =============================== Staged Changes ================================
Action         Nodes(s)
-------------------------------------------------------------------------------
join           'dev2@127.0.0.1'
join           'dev3@127.0.0.1'
join           'dev4@127.0.0.1'
join           'dev5@127.0.0.1'
-------------------------------------------------------------------------------


NOTE: Applying these changes will result in 1 cluster transition

###############################################################################
                         After cluster transition 1/1
###############################################################################

================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
valid     100.0%     20.3%    'dev1@127.0.0.1'
valid       0.0%     20.3%    'dev2@127.0.0.1'
valid       0.0%     20.3%    'dev3@127.0.0.1'
valid       0.0%     20.3%    'dev4@127.0.0.1'
valid       0.0%     18.8%    'dev5@127.0.0.1'
-------------------------------------------------------------------------------
Valid:5 / Leaving:0 / Exiting:0 / Joining:0 / Down:0

Transfers resulting from cluster changes: 51
  12 transfers from 'dev1@127.0.0.1' to 'dev5@127.0.0.1'
  13 transfers from 'dev1@127.0.0.1' to 'dev4@127.0.0.1'
  13 transfers from 'dev1@127.0.0.1' to 'dev3@127.0.0.1'
  13 transfers from 'dev1@127.0.0.1' to 'dev2@127.0.0.1'


Finally, you can commit the batch:

 dev2/bin/riak-admin cluster commit

Test the cluster 

Now we now a have a running five-node Riak cluster. Let's make sure it's working properly. For this we have a couple of options. A simple option is to run the member-status command on one of our nodes:

dev1/bin/riak-admin member-status

This will give us a high-level view of our cluster and tell us the percentage of the ring that each node manages:

 ================================= Membership ==================================
Status     Ring    Pending    Node
-------------------------------------------------------------------------------
valid      20.3%      --      'dev1@127.0.0.1'
valid      20.3%      --      'dev2@127.0.0.1'
valid      20.3%      --      'dev3@127.0.0.1'
valid      20.3%      --      'dev4@127.0.0.1'
valid      18.8%      --      'dev5@127.0.0.1'
-------------------------------------------------------------------------------
Valid:5 / Leaving:0 / Exiting:0 / Joining:0 / Down:0

You should now have a five-node Riak cluster up and running. 

Shortcut For running Riak node

I have written two shell script to ease for running Riak nodes.It is very ease to use in every day.

First, create riak_run.sh using text editor and add following lines and save:

#!/bin/bash
#script for initiating riak database.

echo "Initiating Riak DB..."

cd riak-1.3.2/dev/

for node in `ls`; do $node/bin/riak start; done

echo "Riak DB initiated."

echo "-----------------"

echo "Checking Riak....."
ps aux | grep beam


echo "Riak checking completed."


Second, create riak_stop.sh file, add and save below lines:

#!/bin/bash
#script for initiating riak database.

echo "Stoping Riak DB..."

cd riak-1.3.2/dev/

for node in `ls`; do $node/bin/riak stop; done

echo "Riak DB stoped."

echo "-----------------"

echo "Checking Riak....."
ps aux | grep beam


echo "Riak checking completed."

must save this two files in home directory and run the below command in your Teminal:

make it as executable
chmod +x riak_run.sh
chmod +x riak_stop.sh
  For run and check the status of Riak nodes:

./riak_run.sh

For stop Riak nodes:

./riak_stop.sh

Congratulations!....Cheers!!!!

Reference: http://docs.basho.com/riak

Web based AVR Project

          This AVR Project is derived from the idea to control home appliances through web. It is a combination of web application and avr circuit. This project consist  python(flask frame work ) as server and JavaScript with HTML5 as client. The User Interface and avr microcontroller is communicate through serial port of PC.

CIRCUIT DIAGRAM

Circuit is divided into two parts, microcontroller-circuit and serial port interface.
The main circuit consist of atmega8 microcontroller, four leds connected to PORTD (pins-6,11,12,13) and a potentiometer in pin-23(ADC0).The MAX232 used as level converter for communicating the USART of atmega8 with serial port(PC).USART is simply a form of serial data communication.

main circuit

Microcontroller and PC communication using RS232: RS232 is the encoded version of USART.

serial port interface

  This blog post updating now....

USBasp programmer with avrdude(Linux)

    After success full  implementation of  serial programmer, I turned to make a USBasp programmer .The firmware is downloaded from USBasp official site.In this tutorial we will see how to use AVRdude for burning hex files into AVR microcontroller using USBasp. USBasp is a USB in-circuit programmer for Atmel AVR controllers. It simply consists of an ATMega8 and a couple of passive components. The programmer uses a firmware-only USB driver, no special USB controller is needed. In my circuit only use the six pins(MOSI,MISO,SCK,RESET,GND,+5V) from the usbasp to target microcontroller, It is little different from actual circuit in official site. No need of external +5V power supply to target microcontroller. The circuit is given below

USBasp programmer

NOTE:The fuse bits for 12Mhz crystal  HFUSE=0xc9 and  LFUSE=0xef  burn explicitly to  atmega8 using serial programmer(any programmer).It is very important, the USBasp programmer cannot work with default fuse bits. The command for setting the fuse bit:

avrdude -c ponyser -p m8 -P /dev/ttyS0 -U lfuse:w:0xc9:m -U hfuse:w:0xef:m

For reading the current fuse bit in atmega8:

avrdude -c ponyser -p m8 -P /dev/ttyS0 -U hfuse:r:high.bin:b -U lfuse:r:low.bin:b

AVR Serial Programmer

First I think, how to build a USBasp programmer.But I have n't a usb programmer to flash the firmware to it. Then decided to create a serial programmer and flash the firmware through this programmer to  atmega8. My  AVR serial programmer will transform hex file to most AVR microcontroller. It can build using few readily  available components. This programmer is compatible with popular avrdude software or PonyProg software.

We will be using the serial port for burning. First, we have to develop a burning circuit for it.

REQUIRED COMPONENTS 

Resistors :             10k     - 2
                           4.7k    - 2
                           15k     - 1

Zener diode :         5.1v   - 2

Diode :               1N4148 - 1

Transistor :      BC549/48 -  1

9-PIN RS232 female connector

serial programmer

Unit testing with nose Framework

         Unit testing is a software development process in which the smallest testable parts of an application, called units, are individually and independently tested for proper operation. Unit testing of software applications is done during the development (coding) of an application. Proper unit testing done during the development stage saves both time and money in the end. Unit tests find problems early in the development cycle. Some programming languages directly support unit testing, like Python, java etc. There are a couple of ways to integrate unit tests into your development style. These include Test Driven Development, where unit tests are written prior to the functionality they're testing; during refactoring, where existing code -- sometimes code without any automated tests to start with -- is retrofitted with unit tests as part of the refactoring process; bug fix testing, where bugs are first pinpointed by a targetted test and then fixed; and straight test enhanced development, where tests are written organically as the code evolves. In the end, I think it matters more that you're writing unit tests than it does exactly how you write them.For me, the most important part of having unit tests is that they can be run quickly, easily, and without any thought by developers.

nose

There are many unit test frameworks in Python, and more arise every day. I personally use nose, and it fits my needs fairly well. nose is simpler Unit Testing Framework for python. nose extends unit test to make testing easier. nose comes with a number of builtin plugins to help you with output capture, error introspection, code coverage, doctests, and more. It also comes with plugin hooks for loading, running, watching and reporting on tests and test runs.

Installation
Please refer the nose 1.3.0 documentation for installation of nose on Unix like systems.

simple examples:

Now let's start with a few examples. Here's the simplest nose test you can write:

def test_a():
      assert 'a' == 'a'

Put this in a file called 'test_me.py', and then run nosetest. You will see this output:

.
----------------------------------------------------------------------
Ran 1 test in 0.002s

OK

If you want to see exactly what test was run, you can use nosetest -v

test_me.test_a ... ok

----------------------------------------------------------------------
Ran 1 test in 0.002s

OK

A fairly common pattern for unit tests is something like this:

def test():
   setup_test()
   try:
      do_test()
      make_test_assertions()
   finally:
      cleanup_after_test()

Here, setup_test is a function that creates necessary objects, opens database connections, finds files, etc.Then do_test and make_test_assertions acually run the test code and check to see that the test completed successfully.Finally, the preconditions are cleaned up.

test discovery and execution

nose is a unit test discovery and execution package. Before it can execute any tests, it needs to discover them. nose has a set of rules for discovering tests, and then a fixed protocol for running them. While both can be modified by plugins, for the moment let's consider only the default rules.nose only looks for tests under the working directory, unless you specify one with the -w command line option.nose can only execute tests that it finds. If you're creating a new test suite, it's relatively easy to make sure that nose finds all your tests.

nose command line

Apart from the plugins, there are only a few options that I use regularly.

-w: Specifying the working directory
nose only looks for tests in one place. The -w flag lets you specify that location; e.g. nosetests -w my_nose_test/.  will run only those tests in the directory "/my_nose_test/"
As of the latest development version you can specify multiple working directories on the command line:
nosetests -w nose_test/ -w sample/

-s: Not capturing stdout
By default, nose captures all output and only presents stdout from tests that fail. By specifying '-s', you can turn this behavior off.

-v: Info and debugging output
nose is intentionally pretty terse. If you want to see what tests are being run, use '-v'.

-p: plugins
Output list of available plugins and exit. Combine with higher verbosity for greater detail

Tools for testing

           nose.tools provides a few convenience functions to make writing tests easier. You don’t have to use them; nothing in the rest of nose depends on any of these methods.

nose.tools.ok_(expr, msg=None)
Shorthand for assert. Saves 3 whole characters!

nose.tools.eq_(a, b, msg=None)
Shorthand for ‘assert a == b, “%r != %r” % (a, b)

nose.tools.make_decorator(func)
Wraps a test decorator so as to properly replicate metadata of the decorated function, including nose’s additional stuff (namely, setup and teardown).

nose.tools.raises(*exceptions)
Test must raise one of expected exceptions to pass.

nose.tools.set_trace()
Call pdb.set_trace in the calling frame, first restoring sys.stdout to the real output stream. Note that sys.stdout is NOT reset to whatever it was before the call once pdb is done!

nose.tools.timed(limit)
Test must finish within specified time limit to pass.

nose.tools.with_setup(setup=None, teardown=None)
Decorator to add setup and/or teardown methods to a test function:

nose.tools.istest(func)
Decorator to mark a function or method as a test

nose.tools.nottest(func)
Decorator to mark a function or method as not a test

Running nose programmatically

nose has a friendly top-level API which makes it accessible to Python programs. You can run nose inside your own code by doing this:

import nose

### configure paths, etc here

nose.run()

### do other stuff here

By default nose will pick up on sys.argv; if you want to pass in your own arguments, use nose.run(argv=args). You can also override the default test collector, test runner, test loader, and environment settings at this level. This makes it convenient to add in certain types of new behavior

For more details please refer the official site.

Graph Colouring

       Now, I am creating a web application, Graph Coloring.The graphs are drowned using nodes and lines. In this Graph Coloring application, the nodes of the graph are colored using different colors for adjacent nodes. My application has two parts, one python script that runs in server and the other, an html file which act as user interface for the users.The html file provides the graphical view representation of my application.This application will be explained in detail, later on.

Graph and Graph Coloring

        Mathematically, a graph is a representation of a set of objects where some pairs of the objects are connected by links. The interconnected objects are represented by mathematical abstractions called vertices, and the links that connect some pairs of vertices are called edges.Vertices are also called nodes or points, and edges are also called lines or arcs.
        In Graph theory, graph coloring is a special case of graph labeling. In its simplest form, it is a way of coloring the vertices of a graph such that no two adjacent vertices share the same color; this is called a vertex coloring.

        Now let's going through the codes.In my application, i have used canvas in html file.Canvas is a rectangular portion or area within the html page, and you can control every pixel in it. The canvas element has several methods for drawing paths, boxes, circles, characters, and adding images. For drawing within the canvas, you should need Javascript. The codes are below.

<canvas id="graph" width="700" height="440" style="border:2px solid #000000;"></canvas>

The above code is used for drawing a canvas in html5. I have choose JavaScript for drawing on canvas.More details on canvas and Javascript are available at: http://www.w3schools.com/html5/html5_canvas.asp. 

       Here in the graph application my aim is use javascript for detecting the mouse movements so that i can get the position of each mouse click. Thus getting the points where mouse is clicked, so that i can draw the vertex and lines. I have assigned different clicks for application, double click for creating a node and two single clicks for drawing a line. My codes for mouse actions are below.

function Point(x,y){
    this.x = x;
    this.y = y;
}

function getMousePos(e){
    var point = new Point(0,0);
    if (e.pageX != undefined && e.pageY != undefined) {
point.x = e.pageX;
point.y = e.pageY;
}
    else{
point.x = e.clientX + document.body.scrollLeft +
document.documentElement.scrollLeft;
point.y = e.clientY + document.body.scrollTop +
document.documentElement.scrollTop;
}

    point.x -= canvas.offsetLeft;
    point.y -= canvas.offsetTop;
    return point;      
}

In the above code, mouse points are returned.the vertex and line are drowned using this points.

function drawcircle(e){
    point=getMousePos(e);
    if(checkNode(point,40)==-1){
Vertex(point,"black");
context.font = "10pt Courier New";
context.fillText(v,point.x+10,point.y-20);
v++;
vertex_list.push(point);
}
    else{ 
        alert("node overlapped");
}
}

function drawline(e){
    var point = getMousePos(e);
    var vertex_no=checkNode(point,40);
    if(vertex_no>=0){
if(start_edge==0){
 start_edge=1;
 selected_vertex=vertex_no;
 context.beginPath();
       context.moveTo(vertex_list[vertex_no].x, vertex_list[vertex_no].y);
}
else{
 start_edge=0;
          if(adj_list[selected_vertex]==undefined)
adj_list[selected_vertex]= new Array();
 if(adj_list[vertex_no]==undefined)
adj_list[vertex_no]= new Array();
 if(selected_vertex!=vertex_no){
adj_list[selected_vertex].push(vertex_no);
adj_list[vertex_no].push(selected_vertex);
 }
 context.lineTo(vertex_list[vertex_no].x,vertex_list[vertex_no].y)
 context.stroke();
 context.closePath(); 
}
}
    else
start_edge=0;
    }

       The functions ‘drawline’ and ‘drawcircle’ gets called when the user clicks, i.e. double or single click, anywhere within the canvas. Its arguments is a MouseEvent object that contains information about where the user clicked. Each functions calls getMousePos(e), where the getMousePos(e) returns the values of mouse coordinates where ever the user clicks within the canvas. The values are returned to a vertex_list.After the values are returned, they are used to draw nodes and lines within the canvas.
       Now for the coloring part, an adjacency list containing the list of nodes is needed for coloring.Since the html file is the client and the adjacency list are sent to the server. For communicating with server, i.e. the python script, jQuery is used.i have used jQuery for sending the adjacent list to the server running the python script where the coloring algorithm is implemented.For more details on jQuery visit http://www.w3schools.com/jquery/default.asp.
The complete the code of Graph coloring application is available at: https://github.com/jaseemkp/Graph-Coloring-Django
You can try the graph coloring application at : http://jastech-graph.herokuapp.com/

MongoDB

           Now Let's going through Simple NoSQL database MongoDB. MongoDB is leading NoSQL database,designed for how we build and run application today. NoSQL is designed for distributed data stores where very large scale of data storing is needed. It is a database system that provides simple lightweight mechanism for storage and retrieval of data.NoSQL has become much popular in the last few years.MongoDB is a general purpose, open-source NoSQL database.
Features of MongoDB:

• Document data model with dynamic schemas
• Full, flexible index support and rich queries
• Auto-Sharding for horizontal scalability
• Built-in replication for high availability
• Text search
• Advanced security
• Aggregation Framework and MapReduce
• Large media storage with GridFS

Organizations of all sizes use MongoDB to quickly and easily develop, scale and operate applications. Instead of storing data in rows and columns as one would with a relational database, MongoDB stores a binary form of JSON documents (BSON).

 MongoDB Installation:
      Now let's see how to get started  with MongoDB.This tutorial provide basic installation steps for installing MongoDB on Ubuntu.12.04 Linux system, where we uses  .deb package as the basis of the installation.
Enter the following command to import the 10gen public GPG Key:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10

Create a /etc/apt/sources.list.d/10gen.list file and include the following line for the 10gen repository.

deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen

Now Enter the following command to reload your repository:

sudo apt-get update

Issue the following command to install MongoDB:

sudo apt-get install mongodb-10gen

When this command completes, you have successfully installed MongoDB!
You can start the mongod process by issuing the following command:

sudo service mongodb start

As needed, you may stop the mongod process by issuing the following command:

sudo service mongodb stop

You can connect to your MongoDB instance by issuing the following command at the system prompt:

mongo

This will connect to the database running on the localhost interface by default. For more information please refer the link:http://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/

pymongo

pymongo is python package index. We prefer pip to install pymongo on python:

pip install pymongo

In Python shell, type the following command to import pymongo package.

import pymongo

To connect to the database, we create a database instance- a connection object by including the following in our python shell.

from pymongo import Connection
connection = Connection()
collection = connection.DB_NAME.COLLECTION_NAME

now we are ready to insert database.

collection.insert({"fruit": "Mango", "color": "Orange"})

The above code snippet is enough for inserting a row(document or BSON document in terms of MongoDB) in the table(collection) collection_name.
For retrieving all the data entered into the collection collection_name,

collection.find()

It's also possible to get the specific item in the collection.

collection.find_one({"color": "Orange",})

Reading all the documents of the specified kind, is by,

collection.find({"color": "Orange", })

for more details please click Here.


for sql to mongoDB convertion also refer the link http://docs.mongodb.org/manual/reference/sql-comparison/

Paint application with MongoDB

Now, I have replaced database of My Paint Application from Sqlite3 to MongoDB. The html front end and javascripts are same.Please refer my earlier post about paint application for more info.I will explain how to connect to database and retrive data from it.

from flask import Flask, render_template, request, redirect
from pymongo import Connection

app = Flask(__name__)

connection = Connection()
collection = connection.paintdb.drawings

@app.route('/', methods=['GET', 'POST'])
def paint():
    if request.method == 'GET':
        all_data = {}
        datas = collection.find()
        for data in datas:
            all_data[data["fname"]] = data["image_data"]
        return render_template('paint.html', py_all= all_data)
    elif request.method == 'POST':
        filename = request.form['fname']
        data = request.form['whole_data']
        collection.insert({"fname":filename, "image_data":data})
        return redirect('/')

if __name__ == '__main__':
   app.run()

from pymango module import class "Connection".It is used to connect with database.Then form a collection(table),named as drawings. Datas are inserted directly into collection. 

The complete code is Here