The most popular libraries for data analysis in Python are NumPy, Pandas and Matplotlib. NumPy is a library for working with numerical data, while Pandas is built on top of NumPy and provides more advanced functionality for data manipulation and analysis and Matplotlib is used to plot the data in user-friendly charts and graphs. In this article, we’ll explore how to use these libraries to analyze data in Python.

P.S.: This tutorial assumes that you have a basic understanding of how Python works and how to set up virtual environments to separate environments for different projects.

Installing NumPy and Pandas

Before we can start analyzing data with NumPy, Pandas and Matplotlib, we need to install them, which is a simple process.

pip install numpy pandas matplotlib

This will download and install NumPy, Pandas and Matplotlib on your system. With these packages installed, we're ready to start working with data.

NumPy Basics

NumPy provides a powerful set of tools for working with numerical data in Python. One of the most important features of NumPy is the ndarray object, which provides a fast and efficient way to store and manipulate arrays of numerical data.

To get started with NumPy, we first need to import the library:

import numpy as np

This imports NumPy and renames it to "np" for our convenience. Creating an array in NumPy is simple. We can create a one-dimensional array (also called a vector) by passing a Python list to the np.array() function:

import numpy as np
x = np.array([1,2,3])
print(x)

Output:

[1 2 3]

We can also create two-dimensional arrays (matrices) by passing a list of lists to the np.array() function:

y = np.array([[1,2,3], [4,5,6]])
print(y)

Output:

[[1 2 3]
 [4 5 6]]

NumPy provides a number of functions for creating arrays with specific values, such as zeros() and ones(). We can create an array of zeros with the np.zeros() and an array of ones with the np.ones() functions respectively:

z = np.zeros((3,3))
print(z)

Output:

[[0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]]

a = np.ones((4,4))
print(a)

Output:

[[1. 1. 1. 1.]
 [1. 1. 1. 1.]
 [1. 1. 1. 1.]
 [1. 1. 1. 1.]]

NumPy provides a lot of functions for manipulating arrays. For example, we can reshape an array with the reshape() function:

a = np.array([1,2,3,4,5,6])
b = a.reshape((2,3))
print(b)

Output:

[[1 2 3]
 [4 5 6]]

We can also perform mathematical operations on arrays. For example, we can add two arrays together:

c = np.array([1,2,3])
d = np.array([4,5,6])
​
e = c + d
print(e)

Output:

[5 7 9]

Similarly, we can multiply two arrays together:

f = np.array([1,2,3])
g = np.array([4,5,6])
h = f * g
print(h)

Output:

[ 4 10 18]

NumPy also provides functions for computing basic strategies on arrays. For example, we can compute the mean, median, and standard deviation of an array:

i = np.array([1,2,3,4,5,6])
print(np.mean(i))
print(np.median(i))
print(np.std(i))

Output:

3.5
3.5
1.707825127659933

Pandas Basics

While NumPy provides a powerful set of tools for working with numerical data, it doesn't provide much in the way of data manipulation or analysis. This is where Pandas come in.

Pandas provides two main classes for working with data: Series and DataFrame. A Series is a one-dimensional array-like object, while a DataFrame is a two-dimensional table-like object. Both of these classes provide a rich set of functions for manipulating and analyzing data.

Creating a Series in Pandas is simple. We can create a Series from a Python list:

import pandas as pd
s = pd.Series([1,2,3])
print(s)

Output:

0    1
1    2
2    3
dtype: int64

The first column of the output shows the index of the Series, which is automatically generated by Pandas. We can also specify our own index:

t = pd.Series([1,2,3], index=['a', 'b', 'c'])
print(t)

Output:

a    1
b    2
c    3
dtype: int64

Creating a DataFrame in Pandas is also simple. We can create a DataFrame from a Python dictionary:

u = pd.DataFrame({'Name': ['Shady', 'Tony', 'Amanda'], 'Age': [25, 30, 35]})
print(u)

Output:

     Name  Age
0   Shady   25
1    Tony   30
2  Amanda   35

Like Series, DataFrames also have an index, which is generated automatically if not specified.

Pandas provides a rich set of functions for manipulating and analyzing data. For example, we can filter a DataFrame to include only rows that meet a certain condition:

In [16]:

v = pd.DataFrame({'Name': ['Shady', 'Tony', 'Amanda'], 'Age': [25, 30, 35]})
w = v[v['Age'] > 30]
print(w)

Output:

     Name  Age
2  Amanda   35

We can also sort a DataFrame by a certain Column:

x = pd.DataFrame({'Name': ['Shady', 'Tony', 'Amanda'], 'Age': [25, 30, 35]})
y = x.sort_values(by='Age')
print(y)

Output:

     Name  Age
0   Shady   25
1    Tony   30
2  Amanda   35

Another useful function is groupby(), which allows us to group rows of a DataFrame by a certain column:

z = pd.DataFrame({'Name': ['Alice', 'Bob', 'Charlie'], 'Gender': ['F', 'M', 'M'], 'Age': [25, 30, 35]})
a = z.groupby('Gender')['Age'].mean()
print(a)

Output:

Gender
F    25.0
M    32.5
Name: Age, dtype: float64

This groups the rows of the DataFrame by the "Gender" column and computes the mean of the "Age" column for each group.

Now that we know some basic functionalities of NumPy and Pandas, let's move on to see how we can use these libraries for data analysis.

Data Analysis with NumPy and Pandas

Suppose we have a CSV File, containing some data we want to analyze. We can read the data into a pandas DataFrame using the read_csv() function:

sales = pd.read_csv('sales_data.csv')
sales.head(10)

Output:

This assumes that the CSV file is located in the same directory as our Python script. If the file is present somewhere else, we should give the exact path to that location. For example, 'home/user/my_project/data.csv'

Once we have the data in the DataFrame, we can use Pandas functions to analyze it. We have a CSV file that contains information about different stores' sales data including their product group, product code, stock quantity, cost, price, and sales. We can compute some statistics based on this data using Pandas.

average_cost = sales['cost'].mean()
print("Average Cost: ", average_cost)

total_sales_week = sales.groupby('store')['last_week_sales'].sum()
print("Total sales last week: ", total_sales_week)

Output:

Average Cost:  502.79470000000003
Total sales last week:  store
Alpino      915
Mary       1024
Paladin    1200
Soner      1066
Uler       1222
Name: last_week_sales, dtype: int64

We can also filter the data based on certain conditions. For example, if we wanted to only see the data of one store:

ulers_data = sales[sales.store=="Uler"]
ulers_data

Output:

This will only output a DataFrame containing only the rows where the store column is Uler.

We can also use Pandas to plot the data. For example, let's say we want to plot a histogram of last week's sales of different products.

import matplotlib.pyplot as plt
plt.bar(ulers_data['last_week_sales'], ulers_data['product_code'])
plt.xlabel('Product Code')
plt.ylabel('Last week sales')
plt.title('Last week sales of products')
plt.show()

Output:

We can get the average sales of a specific product group and plot them as well:

avg_sales = sales.groupby(['store', 'product_group'], as_index=False).agg(avg_sales = ("last_week_sales", "mean"))
avg_sales

Output:

avg_sales.plot.bar(x="store", y="avg_sales", color=['r'])

Output:

Conclusion:

In this section, we have shown how to use NumPy and Pandas for data analysis. We have demonstrated how to read data from a CSV file into a Pandas DataFrame, compute statistics on the data, filter the data based on certain conditions, and plot the data using Matplotlib.

These are just a few examples of the many things you can do with NumPy and Pandas for data analysis. With these tools, you can quickly and easily analyze large amounts of data, allowing you to gain insights and make informed decisions.

If you like what you are reading, Buy me a coffee, to support me.

print("hello world")

You can also save the script in a file and run it through the command line to execute that code.

$ python3 hello.py

These are just a few demos of what you can do in python.

Scripts in Django

After you set up a project in Django, It's quite a hassle to manually check your models and objects through the Django shell command. Let's create a simple Django app (considering you have already installed and set up Django).

class Students(models.Model):
    first_name = models.CharField(max_length=200, null=True, blank=True)
    last_name = models.CharField(max_length=200, null=True, blank=True)
    address = models.CharField(max_length=200, null=True, blank=True)
    dob = models.DateField(auto_now_add=False, null=True, blank=True)

    def __str__(self):
        return self.first_name + self.last_name

After creating the model run makemigrations and migrate command in the command line.

python manage.py makemigrations && migrate

Now we can add some students through the Django shell.

python manage.py shell

An interactive console will open in your terminal.

Now we can create our first student.

from students.models import Students
Students.objects.create(first_name='John', last_name='Doe', address='12th street, Washington')

This process can be tedious if we have to repeat it for many students and will consume a lot of time. But wait there's an easy trick to do this. This is where Django's runscript command comes in.

In your command line write the following line:

pip install django-extensions

Add it to the INSTALLED_APPS on your settings file:

INSTALLED_APPS = [
    ...,
    'django_extensions',
]

Now we just have to create a folder called scripts inside our main project folder (where our manage.py exists).

mkdir scripts
touch scripts/__init__.py

Now we can write our scripts to add, remove or modify our students. Let's create a script to delete all students in our database.

touch scripts/delete_all_students.py

from students.models import Students

def run():
    #fetch all students
    students = Students.objects.all()
    #delete all students
    students.delete()

Now that our script is ready, we can just run our script through the command line as follows:

python manage.py runscript delete_all_students

If you now check the student's model you'll find that all students have been removed.

python manage.py shell

from students.models import Students
Students.objects.all()

We have an empty Queryset here, meaning all the students have been removed. This is just the tip of the iceberg and you can do almost every shell command from your runscript command. The only limitation is your imagination.

TLDR;

You can use python manage.py runscript <script> to run various time-consuming shell commands directly in an instant. If you want to know more about the runscript command visit here.

Benefits of super() function:

1. We don't need to remember the name of the parent class. Calling the super() method will directly access the parent class without specifying the name of the parent class.
2. We can use it with single or multiple inheritances.
3. It is better for code reusability and modularity, which helps to keep the DRY(Do not Repeat Yourself) approach.
4. The super() function is a dynamic function and goes along with the concept of the dynamically typed programming language of python.

Limiting factors of super() function:

1. The arguments passed in the super() function and the arguments we have in the super function should match.
2. Every time we call the super method the keyword super() should be used.
3. We have to specify the exact class and methods present, which are referred to by the super() function.

Using super() function in inheritance

Let us have a rectangle and a square function that is used to calculate the area and perimeter of the rectangle and the square. Normally what we do is the following:

class Rectangle:
    def __init__(self, length, width):
        self.length = length
        self.width = width

    def area(self):
        return self.length*self.width

    def perimeter(self):
        return 2*self.length + 2*self.width


class Square:
    def __init__(self, length):
        self.length = length

    def area(self):
        return self.length**2

    def perimeter(self):
       return 4*self.length()

These are two simple classes Rectangle and Square. You can use these as:

>>> square = Square(7)
>>> square.area()
49
>>> rectangle = Rectangle(5,2)
>>> rectangle.perimeter()
14

A square is a special type of rectangle which have both of its sides equal to each other. So we can use inheritance in the Square class to reduce the code.

class Rectangle:
    def __init__(self, length, width):
        self.length = length
        self.width = width

    def area(self):
        return self.length * self.width

    def perimeter(self):
        return 2 * self.length + 2 * self.width

#We'll declare that the Square class inherits from the Rectangle class
class Square(Rectangle):
    def __init__(self, length):
        super().__init__(length, length)

Here we've used the super() function to call the __init__() method of the Rectangle class, allowing us to use it in the Square class without repeating the same code twice. The core functionality remains the same after using the super() method:

>>> square = Square(5)
>>> square.area()
25
>>> square.perimeter()
20

Behind the scenes, when we call the Square(5) method, it goes on to the __init__() method and see the super() method which then calls the __init__() method of the Rectangle class and pass the length twice as arguments for length and width. After the Square is initialized, all other functions of the Rectangle can be used by it, allowing us to use them wherever we like.

Conclusion

The super() method is a very useful method in python that is used while applying the inheritance of classes. It can be used with multiple inheritances as well.