Using LinkedIN API with Python to retrieve Connections’ Details

Ever faced a problem extracting your LinkedIN connections’ details one by one, don’t worry after reading this post you will able to scrape all your connections’ details in one go. To follow what is next you need to have a basic understanding of Python and an account on LinkedIN.

• Step 1: Sign up at  https://www.linkedin.com/secure/developer and click on “Add New Application”
  
• Step 2: Fill the compulsory fields, accept the terms & conditions and click on “Add Application”
  
• Step 3: On the next screen you will receive some keys and ids that we will we using in next steps. These keys are specific to you and are like “passwords” to access your account, so keep them safe
  
• Step 4: Open a text editor, copy the code below and paste it. Update the consumer_key, consumer_secret, user_token and user_secret as you have received in step 3

#Including the libraries that needed for this program
import oauth2 as oauth
import httplib2
import time, os, simplejson, csv
import demjson

#Authentication and connecting with the server
consumer_key = 'XXXXXXXXX'
consumer_secret = 'XXXXXXXXXXXXXXX'
user_token = 'XXXXXXXXXXXXXXXXXXX'
user_secret = 'XXXXXXXXXXXXXXXXXXXXX'
consumer = oauth.Consumer(consumer_key, consumer_secret)
client = oauth.Client(consumer)
access_token = oauth.Token(
key=user_token,
secret=user_secret)
client = oauth.Client(consumer, access_token)

#Retrieving connections' your connection data. Update the consumer_key, consumer_secret, user_token and user_secret#as yours
resp,content = client.request("https://api.linkedin.com/v1/people/~/connections?format=json", "GET", "")

#Cleaning the data received
data =demjson.decode(content)
v = data.values()[3][0:]
k = v[1].keys()[0:]

#Writing the data to a CSV file 
f = csv.writer(open("connection_detail.csv", "wb+"))
f.writerow(v[1].keys()[0:])
for i in range(len(v)):
    f.writerow(v[i].values()[0:])

• Step 5: Execute the above code in Python, it will generate a CSV file in the working directory named “connection_detail.csv” that has all your connections’ data.

Neo4j: Basics of Cypher

Cypher :  Neo4j’s Graph Query Language

Cypher is a declarative, SQL inspired language for Creating(inserting student data into the graph), Finding (retrieving individual student and school), Querying(discovering related students and schools based on relationships, properties etc.) and much more in graphs. Cypher allows us to perform actions on the graph database without requiring us to describe exactly how to do it.

Nodes

Cypher uses ASCII-art to represent patterns. In Cypher we need to surround nodes with parentheses which look like circles, e.g. (node). To refer node after defining it, we’ll give it an identifier like (school) for school or (student) for students. So for example, if we want to find all the students and the school they have studied in, the query will include the identifiers student and school, e.g. in a pattern like (student)-->(school) so we can refer to them later, e.g. to access properties like student.name and school.name.

The more general structure is:

MATCH (node) RETURN node.property

MATCH (node1)-->(node2) 
RETURN node2.propertyA, node2.propertyB

Relationships

The problem with the Cypher query we saw above is that it didn’t mention anything about the relationship between the nodes, so even though we used the identifier student, we may well have gotten back teachers of our schools. So we need to be able to describe the types of relationships in our Cypher queries.

If we wanted to retrieve every student who had studied in a school, we would describe the pattern(student)-[:Student_Of]->(school) to retrieve only nodes that had a relationship typed Student_Of  with other nodes (schools) those nodes would the be students as stated by the Student_Of relationship.

Or generally:

MATCH (node1)-[:REL_TYPE]->(node2)

Sometimes we need access to information about a relationship (e.g. its type or properties). For example, we might want to output the year the when the student joined that particular school and that year would probably be a property of the Student_Of relationship. As with nodes, we can use identifiers for relationships (in front of the :TYPE). If we tried to match (student)-[year_join:Student_Of]->(school) we would be able to output the year_join.year for each of the student in all of the schools that they studied in.

MATCH (node1)-[rel:TYPE]->(node2) 
RETURN rel.property

 

Introduction to Graph Database with Neo4j

Conventional databases(Relational Database) typically consist of many tables. A table is much like a spreadsheet, in that it’s made up of rows and columns. All rows have the same columns, and each column contains the data itself, think of tables in the same way like a table in Excel. The definition of those tables and their columns determine the storage capabilities of the database, whereas the relations between the columns define the kinds of facts that can be stored in such a database.

What is Graph Database ?

We live in a world, where there are different pieces of information which are connected to each other in way or another. A Graph database take this connectivity or relationship between different information as a core aspect of its data model, which helps in efficient storage, processing and querying. While other databases computes relationship at query time, a graph database stores relationship as its primary element. Accessing those already persistent connections is an efficient, constant-time operation and allows you to quickly traverse millions of connections per second. Graph databases performs well in managing highly connected data and complex queries.

• A Graph Database stores data in a Graph, representing any kind of data
• The records in a graph database are called Nodes
• Nodes are connected to each other via directed Relationships
• Each single Node and Relationship can have named attributes referred to as Properties
• A Label is used to organizes nodes into groups
  .

Image Source: Neo4j.com

Sponsored by Neo Technology, Neo4j is an open-source NoSQL graph database implemented in Java and Scala. Neo4j is a Database – use it to reliably store information and find it later. Neo4j’s data model is a Graph, in particular a Property Graph. Cypher is Neo4j’s graph query language(declarative). You describe what you you are interested in, not how it is acquired. Cypher is meant to be very readable and expressive

You can download Neo4j from http://neo4j.com/download and install it as a server on all operating systems. By default, the Neo4j Server is bundled with a web interface bound tohttp://localhost:7474.

Creative Commons: Protect Your Creative Work

Image Source : http://www.creativecommons.org

Being a creative artist, blogger, innovator many people face this problems like how they can protect there work so that it is not being copied or used by someone else; want people to share their work or even modify it as long as they give credit; don’t want companies to use it for commercial purpose.

Creative Commons is a nonprofit organization that enables the sharing and use of creativity and knowledge through free legal tools. They provide free, easy-to-use copyright licenses provide a simple, standardized way to give the public permission to share and use your creative work — on conditions of your choice. CC licenses let you easily change your copyright terms from the default of “all rights reserved” to “some rights reserved.”

Licenses provided by Creative Commons are not an alternative to copyright, instead they work alongside copyright and enable you to modify your copyright terms to best suit your needs.

If you want to give people the right to share, use, and even build upon a work you’ve created, you should consider publishing it under a Creative Commons(CC) license. CC gives you flexibility (for example, you can choose to allow only non-commercial uses) and protects the people who use your work, so they don’t have to worry about copyright infringement, as long as they abide by the conditions you have specified.

There are various conditions that you can include in your license, some of them are:

Attribution- ShareAlike: Including this condition in your license allows someone to remix, tweak, and build upon your work even for commercial purposes but they must give you credit and license their new creations under the same terms.

Attribution- Noderivs: This term allows redistribution, commercial and non-commercial, as long as it is passed along unchanged and in whole, with credit to you.

Attribution- Non Commercial: This license lets others use, change and build upon your work as long as they are doing it for non-commercial purposes and their new works must acknowledge your contribution but they don’t necessarily have to be license their derivative works on these same terms.

You can see various licences available by visiting this link : http://creativecommons.org/licenses

Image Source :http://creativecommons.org/licenses/

 

Choose a Creative Commons license : http://creativecommons.org/choose/

 

Analyze Network Data with Wireshark

Image Source : http://www.wireshark.org

 

Wireshark is a free and open-source packet analyzer. It is used for network troubleshooting, analysis, software and communications protocol development and education.  A network packet analyzer is a tool used to analyze what’s going on inside a network cable.

Wireshark capture network packets in real time and tries to display that packet data as detailed as possible in human-readable format.

How to use Wireshark to capture network data ?

• Step 1: Download and install Wireshark from http://www.wireshark.org/download.html
  
  
• Step 2: Open Wireshark from your computer. Then, select a network from left side of the window and press start to start capturing network data from it.
  
• Step 3: As soon as you click start, Wireshark starts capturing data packets for that Network. Press “Stop”, whenever you want  Wireshark to stop caputring packets.
• You will see packets highlighted in green, blue and black. Wireshark uses colors to help you identify the types of traffic at a glance. By default, green is TCP traffic, dark blue is DNS traffic, light blue is UDP traffic.
• You can save this data in CSV(Comma Separted) format to analyze it later for other purposes.

Infogr.am: Infographics Made Easy

Image Source : http://infogr.am/

Infographics are one of the most efficient and best way of to represent a conclusion, result or facts. In form of text, images and design infographics represent complex data and results. Infographics have already established themselves as a popular marketing tools.

Infographics are graphic visual representations to present complex information quickly and clearly. And, of course to make a good one you need to have decent knowledge of colors, font(type), sketching and softwares such as Photoshop, Illustrator etc. which certainly you can’t gain overnight. So what if you need a infographic, hiring a freelance graphic designer can cost you much.

There are a few websites available where you can design your infographic easily like infogr.am ,piktochart.com etc.. These websites allow you to select from 100 of already existing templates.

Image Source : infogr.am

 

Then you can upload you own dataset in tabular form or import a spreadsheet.

And can generate your own infographic.

SQL And SQL Injection(A Web Attack Technique)

What is SQL ?
SQL stands for Structured Query Language, it is a set of instructions used to interact with a database. It is the standard language for relational database management systems( According to ANSI ). SQL commands are used to perform operations on a database such as update data on it or retrieve data from it. Some common relational database management systems in the market that use SQL are: Oracle, Sybase, Microsoft SQL Server, Access, Ingres, etc. Some of the SQL commands that are common to all relational database management systems are Select, Insert, Update, Delete, Create, and Drop, these commands can be used to do almost everything that one needs to do with a database.

What is SQL Injection ?
Almost every website you visit, has its very own database where it stores important information such as your user name, password and other useful things that neither you nor the website wants to reveal to anyone else. Not just websites, every organisation manages its own database to keep its data. SQL Injection is one of the many web attack technique used by hackers to steal data from organizations. It is one of the most common application layer attack techniques used today. It takes advantage of improper coding of your website/web applications that allows a hacker to inject SQL commands in form of user input in a web form. In simple words, it arises because the fields available for user input allow SQL statements to pass through and query the database directly, through which a hacker can retrieve, update or even delete data.

A Simple Example
Here is what a HTML Code for a form(log in panel) looks like:

<form>
User Name: <input type="text" name="username"><br>
Password : <input type="password" name="pwd">
</form>

So whenever you enter a “User Name” and “Password”  it assigns its value to “username”  and “pwd” respectively. Now the system checks its database to check whether this username exists or not and if it exists then the password entered by you matches with the one it has in its database, a sample SQL query to do the same is:

SELECT * FROM userdatabase WHERE username = '$username' AND password = '$pwd’

Now, what a hacker do is, he/she injects a SQL statement in the login panel like for both username and password he/she enters ” anything' or 'x'='x .” In this way our SQL query now reads like this:

SELECT id FROM userdatabase WHERE username = 'anything' OR 'x'='x' AND password = 'anything' OR 'x'='x'

which is a valid query because 'x'='x' is true irrespective of anything.

This will allow the hacker to bypass the login form without actually knowing a valid username/password combination!

 

Web Scraping(Data Extraction from Web) Using iMacros

What is Web Scraping ?
Web scraping (web harvesting or web data extraction) is a computer software technique of extracting information or data from websites. These programs usually simulate user exploration of the World Wide Web by either implementing low-level Hypertext Transfer Protocol (HTTP), or embedding in to a web browser, such as Internet Explorer, Google Chrome or Mozilla Firefox. Sometimes websites set up barriers to prevent scrapping or browser automation, in those cases human examination and copy & paste proves to be the best web scraping technique.

In this blogpost, I am going to show how to extract data from a website using iMacros. I am going to extract a table consisting of population count of all the countries of  the world from this website: http://data.worldbank.org/indicator/SP.POP.TOTL(Note:We use this site for demonstrational purposes, only) using iMacros addon for Mozilla Firefox.

Image source: http://www.imacros.net

• Install iMacros add-on for Mozilla Firefox(https://addons.mozilla.org/en-US/firefox/addon/imacros-for-firefox/)
• After installing you can see iMacros icon on the top bar, click on it to open the iMacros Panel.
  
•  Recording a Macro: The very first step for data extraction is to record a macro(which in simple words contains a set of commands which the browser performs). Select “Rec” tab in the iMacros Panel, then push the “Record” button to start recording the macro. Your macro has started recording, enter this url: http://data.worldbank.org/indicator/SP.POP.TOTL  in the url panel of the browser. After the page successfully opens, push the “Stop” button in the iMacros Panel to stop recording the macro.
• Playing the Macro: You can see the macro you have just made in iMacros Panel, double click on it to start playing.
  You can see as soon as you click the macro it automatically opens the web page (http://data.worldbank.org/indicator/SP.POP.TOTL)
  
• Editing the Macro: To edit a macro right click on it and select “Edit Macro“. It will open the macro in iMacros Editor.
  Add these lines of code after the existing code in the macro:

  TAG POS=2 TYPE=TABLE ATTR=TXT:* EXTRACT=TXT
  SAVEAS TYPE=EXTRACT FOLDER=* FILE=population.csv
  

  Lets analyse the code we have right now, it looks something like this
  

• Finally, “Save & Close” the editor and play the edited macro. After it being played open the default iMacros folder(Usually C:Usersuser_nameDocumentsiMacrosDownloads), you can see a file named “population.csv“. Open it with a text editor,  you will find all the extracted data from the table in that separated by commas.

 

You can do a lot of cool stuffs using iMacros, find the full documentation at http://wiki.imacros.net/Main_Page

Make Responsive Websites With Bootstrap

Image Source :- http://getbootstrap.com/

What is Bootstrap ?
Bootstrap is the most popular HTML, CSS, and JS framework for developing responsive, mobile first projects on the web. Bootstrap is a free collection of tools for creating websites and web applications. It contains HTML and CSS-based design templates for typography, forms, buttons, navigation and other interface components, as well as optional JavaScript extensions. In June 2014 it was the No.1 project on GitHub with 69,000+ stars and 25,000+ forks(Source: Wikipedia).

How to download and use Bootstrap?
To download the latest Bootstrap available visit http://getbootstrap.com/, after that unzip the compressed folder. It usually contains three folders inside it namely css, js and fonts. The folder includes the following content :

bootstrap/
├── css/
│   ├── bootstrap.css
│   ├── bootstrap.min.css
│   ├── bootstrap-theme.css
│   └── bootstrap-theme.min.css
├── js/
│   ├── bootstrap.js
│   └── bootstrap.min.js
└── fonts/
    ├── glyphicons-halflings-regular.eot
    ├── glyphicons-halflings-regular.svg
    ├── glyphicons-halflings-regular.ttf
    └── glyphicons-halflings-regular.woff

(Source: www.getbootstrap.com)

To create your first bootstrap website, include the above files in the head of your HTML document.

A sample Bootstrap HTML document:
Here is what a normal Bootstrap HTML document looks like.

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Bootstrap 101 Template</title>

<!-- Bootstrap -->
<link href="css/bootstrap.min.css" rel="stylesheet">

<!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
<!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
<!--[if lt IE 9]>
https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js
https://oss.maxcdn.com/respond/1.4.2/respond.min.js
<![endif]-->
</head>
<body>
<h1>Hello, world!</h1>

<!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js
<!-- Include all compiled plugins (below), or include individual files as needed -->
http://js/bootstrap.min.js
</body>
</html>

(Source: www.getbootstrap.com)