Python Tag

How to use Pandas Scatter Matrix to visualize trends in data

In this post, we’ll cover scatter matrices (pair plots) using Pandas. Now, Pandas is using Matplotlib to make the scatter matrix.

pic courtesy : marsja.se

A scatter matrix (pairs plot) compactly plots all the numeric variables we have in a dataset against each other one. In Python, this data visualization technique can be carried out with many libraries but if we are using Pandas to load the data, we can use the base scatter_matrix method to visualize the dataset.

Prerequisites

Python data visualization will require that we have Pandas and its dependencies installed. It’s very easy to install Pandas. Either we use pip to install Python packages, such as Pandas, or we install a Python distribution (e.g., Anaconda, ActivePython). Here’s how to install Pandas with pip: pip install pandas.

Pandas scatter_matrix Syntax

In general, to create a scatter plot matrix with Pandas using the following syntax:

pandas.plotting.scatter_matrix(dataframe)

pic courtesy : marsja.se

A number of parameters we can use (see above image). Now we are going to use hist_kwds, diagonal, and marker to create pair plots in Python. In the first example, however, we use the simple syntax of the scatter_matrix method.

Data Simulation using Numpy

Now, we are going to create fake data to visualize. Here we will use Numpy to create 3 variables (x1, x2, and x3). Specifically, we use the normal method from random.

import numpy as np
import pandas as pd

np.random.seed(134)                     
N = 1000                              
 
x1 = np.random.normal(0, 1, N)                        
x2 = x1 + np.random.normal(0, 3, N)              
x3 = 2 * x1 - x2 + np.random.normal(0, 2, N)

Next step, before visualizing the data we create a Pandas dataframe from a dictionary.

df = pd.DataFrame({'x1':x1,
                   'x2':x2,
                   'x3':x3})

df.head()

pic courtesy : marsja.se

Now, you can see that we have variables x1, x2, and x3 as columns. Normally, we would import data using Pandas read_csv or Pandas read_excel methods, for instance.

Pandas scatter_matrix (pair plot) Example 1:

In the first Pandas scatter_matrix example, we will only use the created dataframe as input. Now, this will create the following pair plot:

pd.plotting.scatter_matrix(df)

pic courtesy : marsja.se

As evident in the scatter matrix above, we are able to produce a relatively complex matrix of scatterplots and histograms using only one single line of code Now, what does this pairs plot actually contain?

• The diagonal shows the distribution of the three numeric variables of our example data.
• In the other cells of the plot matrix, we have the scatterplots (i.e. correlation plot) of each variable combination of our dataframe. In the middle graphic in the first row we can see the correlation between x1 & x2. Furthermore, in the right graph in the first row we can see the correlation between x1 & x3; and finally, in the left cell in the second row, we can see the correlation between x1 & x2.

In this first example, we just went through the most basic usage of Pandas scatter_matrix method. In the following examples, we are going to modify the pair plot (scatter matrix) a bit.

Pandas scatter_matrix (pair plot) Example 2:

In the second example, on how to use Pandas scatter_matrix method to create a pair plot, we will use the hist_kwd parameter. Now, this parameter takes a Python dictionary as input. For instance, we can change the number of bins, in the histogram, by adding this to the code:

pd.plotting.scatter_matrix(df, hist_kwds={'bins':30})

pic courtesy : marsja.se

Pandas scatter_matrix (pair plot) Example 3:

Now, in the third Pandas scatter matrix example, we are going to plot a density plot instead of a histogram. This is, also, very easy to accomplish. In the code chunk below, we added the diagonal parameter:

pd.plotting.scatter_matrix(df, diagonal='kde')

pic courtesy : marsja.se

That produced a nice scatter matrix (pair plot) with density plots on the diagonal instead of a histogram. Note, that the diagonal parameter takes either “hist” or “kde” as an argument. Thus, if we wanted to have both density and histograms in our scatter matrix, we cannot.

Pandas scatter_matrix (pair plot) Example 4:

In the fourth Pandas scatter_matrix example, we are going to change the marker. This is accomplished by using the marker parameter:

pd.plotting.scatter_matrix(df, marker='+')

Scatter Matrix (pair plot) using other Python Packages

Now, there are some limitations to Pandas scatter_method. One limitation, for instance, is that we cannot plot both a histogram and the density of our data in the same plot. Another limitation is that we cannot group the data. Furthermore, we cannot plot the regression line in the scatter plot. However, if we use the Seaborn and the pairplot() method we can have more control over the scatter matrix. For instance, we can, using Seaborn pairplot() group the data, among other things. Another option is to use Plotly, to create the scatter matrix.

Summary: 3 Simple Steps to Create a Scatter Matrix with Pandas

In this post, we have learned how to create a scatter matrix (pair plot) with Pandas. It was super simple and here are three simple steps to use Pandas scatter_matrix method to create a pair plot:

Step 1: Load the Needed Libraries

In the first step, we will load pandas: import pandas as pd

Step 2: Import the Data to Visualize

In the second step, we will import data from a CSV file using Pandas read_csv method:

csv_file = 'https://vincentarelbundock.github.io/Rdatasets/csv/MASS/survey.csv'
df_s = pd.read_csv(csv_file, index_col=0)

df_s.head()

pic courtesy : marsja.se

Step 3: Use Pandas scatter_matrix Method to Create the Pair Plot

In the final step, we create the pair plot using Pandas scatter_matrix method. Note, however, that we use Pandas iloc to select certain columns.

pd.plotting.scatter_matrix(df_s.iloc[:, 1:9])

pic courtesy : marsja.se

Note, that in the pair plot above, Pandas scatter_matrix only chose the columns that have numerical values.

For more information about to Hire Python Developer from us, E-mail us at – hello@hkinfosoft.com or Skype: hkinfosoft.

Introducing Sketch2Code – Turn Sketches into Working HTML in Seconds

The user interface design process involves lots of creativity and iteration. The process often starts with drawings on a whiteboard or a blank sheet of paper, with designers and engineers sharing ideas and trying their best to represent the underlying customer scenario or workflow. Once a candidate design is arrived at, it’s usually captured via a photograph and then translated manually into a working HTML wireframe that works in a web browser. Such translation takes time and effort and it often slows down the design process.

What if the design could instead be captured from a whiteboard and be instantly reflected in a browser? If user could do that, at the end of a design brainstorm session user would have a readymade prototype that’s already been validated by the designer, developer and perhaps even the customer.

Introducing Sketch2Code – a web based solution that uses AI to transform a picture of a hand-drawn user interface into working HTML code.

pics courtesy: blogs.technet.microsoft.com

Let’s take a closer look at the process of transforming hand-drawn images into HTML using Sketch2Code:

• The user first uploads an image using our website.
• A custom vision model predicts what HTML elements are present in the image and also pins their location.
• A handwriting text recognition service reads the text inside the predicted elements.
• A layout algorithm uses the spatial information from the bounding boxes of the predicted elements to generate a grid structure that accommodates all these components.
• An HTML generation engine uses the above pieces of information to generate HTML markup code reflecting the end result.

The application workflow looks something like this:

pics courtesy: blogs.technet.microsoft.com

Sketch2Code uses the following elements:

• Microsoft Custom Vision Model: This model has been trained with images of different handwritten designs, tagging the information associated with common HTML elements including text boxes, buttons, images, etc.
• Microsoft Computer Vision Service: This is used to identify the text within a design element.
• Azure Blob Storage: The information associated with each step of the HTML generation process is stored, including the original image, predicted results, the layout and grouping information, etc.
• Azure Function: This serves as the backend entry point that coordinates the generation process by interacting with all the services.
• Azure Website: The user interface front-end that enables uploading a new design and seeing the generated HTML results.

The above elements combine together via the following architecture:

pics courtesy: blogs.technet.microsoft.com

Intrigued? You can find the code, solution development process and all other details associated with Sketch2Code on GitHub. Sketch2Code was developed by Microsoft in collaboration with Kabel and Spike Techniques.

We find the range of scenarios in which AI can be applied to be truly amazing, and this is one simple but powerful example of how AI can augment human ingenuity. If you get a chance to play with Sketch2Code, please do share your experiences and thoughts with us below.

For more Information and to build website/System using Python, Hire Python Developer from us as we give you high quality product by utilizing all the latest tools and advanced technology. E-mail us any clock at – hello@hkinfosoft.com or Skype us: “hkinfosoft“.

To develop custom web app using Python, please visit our technology page.

Python’s Fake Increment and Decrement Operators

In Python, you can increase the value of a variable by 1 or reduce it by 1 using the augmented assignment operators. The code spam += 1 and spam -= 1 increments and decrements the numeric values in spam by 1, respectively.

Other languages such as C++ and Java have the ++ and -- operators for incrementing and decrementing variables. (The name of C++ itself reflects this; the name is a tongue-in-cheek joke that indicates it’s an enhanced form of the C language.) Code in C++ and Java could have ++spam or spam++. Python wisely doesn’t include these operators; they are notoriously susceptible to subtle bugs.

However, it is perfectly legal to have the following Python code:

 
>>> spam = 42
>>> spam = ++spam
>>> spam
42
>>> spam = --spam
>>> spam
42

The first thing you notice is that the ++ and -- “operators” in Python don’t actually increment or decrement the value in spam. Rather, the leading - is Python’s unary negation operator. It allows you to have code like this:

>>> spam = 42
>>> print(-spam)
-42

It’s legal to have multiple unary negative operators in front of a value. With two of them, you’d get the negative of the negative of the value, which for integer values just evaluates to the original value:

>>> print(--spam)
42
>>> --42
42

This is a quite silly thing to do, and you won’t ever see a unary negation operator used twice in real-world code. (Though if you did, it’s probably because the programmer learned to program in another language has just written buggy Python code!) There is also a + unary operator. It evaluates an integer value to the same sign as the original value, which is to say, it does absolutely nothing:

>>> +42
42
>>> spam = -42
>>> print(+spam) # +spam is not the same as abs(spam)
-42

Being able to write +42 (or ++42) seems just as silly as –42, so why does Python even have this unary operator? It exists only to complement the – operator if you need to overload these operators for your own classes. (See the documentation for the __pos__ special method.)

The + and – unary operators are only valid when in front of a Python value, not after it. While spam++ and spam– might be legal code in C++ or Java, they produce syntax errors in Python:

>>> spam++
  File "<stdin>", line 1
    spam++
         ^
SyntaxError: invalid syntax

Python doesn’t have increment and decrement operators, it’s only a quirk of the langauge syntax that can make it seem like it does.

For more Information and to build website/System using Python, Hire Python Developer from us as we give you high quality product by utilizing all the latest tools and advanced technology. E-mail us any clock at – hello@hkinfosoft.com or Skype us: “hkinfosoft“.

To develop custom web app using Python, please visit our technology page.

Creating Dynamic Forms with Django

What is a dynamic form and why would you want one?

Usually, you know what a form is going to look like when you build it. You know how many fields it has, what types they are, and how they’re going to be laid out on the page. Most forms you create in a web app are fixed and static, except for the data within the fields.

A dynamic form doesn’t always have a fixed number of fields and you don’t know them when you build the form. The user might be adding multiple lines to a form, or even multiple complex parts like a series of dates for an event. These are forms that need to change the number of fields they have at runtime, and they’re harder to build. But the process of making them can be pretty straightforward if you use Django’s form system properly.

Django does have a formsets feature to handle multiple forms combined on one page, but that isn’t always a great match and they can be difficult to use at times. We’re going to look at a more straightforward approach here.

Creating a dynamic form

For our examples, we’re going to let the user create a profile including a number of interests listed. They can add any number of interests, and we’ll make sure they don’t repeat themselves by verifying there are no duplicates. They’ll be able to add new ones, remove old ones, and rename the interests they’ve already added to tell other users of the site about themselves.

Start with the basic static profile form.

class Profile(models.Model):
    first_name = models.CharField()
    last_name = models.CharField()
    interest = models.CharField()

class ProfileForm(forms.ModelForm):
    first_name = forms.CharField(required=True)
    last_name = forms.CharField(required=True)
    interest = forms.CharField(required=True)

class Meta:
    model = Profile

Create a fixed number of interest fields for the user to enter.

class Profile(models.Model):
    first_name = forms.CharField()
    last_name = forms.CharField()

class ProfileInterest(models.Model):
    profile = models.ForeignKey(Profile)
    interest = models.CharField()

class ProfileForm(forms.ModelForm):
    first_name = forms.CharField(required=True)
    last_name = forms.CharField(required=True)
    interest_0 = forms.CharField(required=True)
    interest_1 = forms.CharField(required=True)
    interest_2 = forms.CharField(required=True)

    def save(self):
        Profile = self.instance
        Profile.first_name = self.cleaned_data[“first_name”]
        Profile.last_name = self.cleaned_data[“last_name”]

        profile.interest_set.all().delete()
        For i in range(3):
           interest = self.cleaned_data[“interest_{}”.format(i]
           ProfileInterest.objects.create(
               profile=profile, interest=interest)

But since our model can handle any number of interests, we want our form to do so as well.

class ProfileForm(forms.ModelForm):
    first_name = forms.CharField(required=True)
    last_name = forms.CharField(required=True)

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        interests = ProfileInterest.objects.filter(
            profile=self.instance
        )
        for i in range(len(interests) + 1):
            field_name = 'interest_%s' % (i,)
            self.fields[field_name] = forms.CharField(required=False)
            try:
                self.initial[field_name] = interests[i].interest
            Except IndexError:
                self.initial[field_name] = “”
        field_name = 'interest_%s' % (i + 1,)
        self.fields[field_name] = forms.CharField(required=False)
        self.fields[field_name] = “”

    def clean(self):
        interests = set()
        i = 0
        field_name = 'interest_%s' % (i,)
        while self.cleaned_data.get(field_name):
           interest = self.cleaned_data[field_name]
           if interest in interests:
               self.add_error(field_name, 'Duplicate')
           else:
               interests.add(interest)
           i += 1
           field_name = 'interest_%s' % (i,)
       self.cleaned_data[“interests”] = interests

    def save(self):
        profile = self.instance
        profile.first_name = self.cleaned_data[“first_name”]
        profile.last_name = self.cleaned_data[“last_name”]

        profile.interest_set.all().delete()
        for interest in self.cleaned_data[“interests”]:
           ProfileInterest.objects.create(
               profile=profile,
               interest=interest,
           )

Rendering the dynamic fields together

You won’t know how many fields you have when rendering your template now. So how do you render a dynamic form?

def get_interest_fields(self):
    for field_name in self.fields:
        if field_name.startswith('interest_'):
            yield self[field_name]

The last line is the most important. Looking up the field by name on the form object itself (using bracket syntax) will give you bound form fields, which you need to render the fields associated with the form and any current data.

{% for interest_field in form.get_interest_fields %}
    {{ interest_field }}
{% endfor %}

Reducing round trips to the server

It’s great that the user can add any number of interests to their profile now, but kind of tedious that we make them save the form for every one they add. We can improve the form in a final step by making it as dynamic on the client-side as our server-side.

We can also let the user enter many more entries at one time. We can remove the inputs from entries they’re deleting, too. Both changes make this form much easier to use on top of the existing functionality.

Adding fields on the fly

To add fields spontaneously, clone the current field when it gets used, appending a new one to the end of your list of inputs.

$('.interest-list-new').on('input', function() {
    let $this = $(this)
    let $clone = $this.clone()

You’ll need to increment the numbering in the name, so the new field has the next correct number in the list of inputs.

    let name = $clone.attr('name')
    let n = parseInt(name.split('_')[1]) + 1
    name = 'interest_' + n

The cloned field needs to be cleared and renamed, and the event listeners for this whole behavior rewired to the clone instead of the original last field in the list.

$clone.val('')
    $clone.attr('name', name)
    $clone.appendTo($this.parent())
    $this.removeclass('interest-list-new')
    $this.off('input', arguments.callee)
    $clone.on('input', arguments.callee)
})

Removing fields on the fly

Simply hide empty fields when the user leaves them, so they still submit but don’t show to the user. On submit, handle them the same but only use those which were initially filled.

$form.find(“input[name^=interest_]:not(.interest-list-new)”)
    .on(“blur”, function() {
        var value = $(this).val();
        if (value === “”) {
            $(this).hide();
        }
    })

Why dynamic forms matter

An unsatisfying user experience that takes up valuable time may convince users to leave your site and go somewhere else. Using dynamic forms can be a great way to improve user experiences through response time to keep your users engaged.

For more Information and to build website/System using Python, Hire Python Developer from us as we give you high quality product by utilizing all the latest tools and advanced technology. E-mail us any clock at – hello@hkinfosoft.com or Skype us: “hkinfosoft“.

To develop custom web app using Python, please visit our technology page.