Objective: -
Renting, also known as hiring or letting, is an agreement where a payment is made for the temporary use of a good, service, or property owned by another. A gross lease is when the tenant pays a flat rental amount and the landlord pays for all property charges regularly incurred by the ownership. Renting can be an example of the sharing economy.
Rental housing refers to a property occupied by someone other than the owner, for which the tenant pays a periodic mutually agreed rent to the owner.The most unaffordable basic necessity is undoubtedly a roof over the head. Leave aside owning a house, even taking a small house on rent in most metros is unaffordable.
The rent of a house depends on a lot of factors. With appropriate data and Machine Learning techniques, many real estate platforms find the housing options according to the customer’s budget.
The goal is to build a machine learning model that predicts the the rent of a house.
Dataset: -
We have information on almost 4700+ Houses/Apartments/Flats Available for Rent with different parameters like BHK, Rent, Size, No. of Floors, Area Type, Area Locality, City, Furnishing Status, Type of Tenant Preferred, No. of Bathrooms, Point of Contact.
Attribute Information:
BHK: Number of Bedrooms, Hall, Kitchen.
Rent: Rent of the Houses/Apartments/Flats.
Size: Size of the Houses/Apartments/Flats in Square Feet.
Floor: Houses/Apartments/Flats situated in which Floor and Total Number of Floors (Example: Ground out of 2, 3 out of 5, etc.)
Area Type: Size of the Houses/Apartments/Flats calculated on either Super Area or Carpet Area or Build Area.
Area Locality: Locality of the Houses/Apartments/Flats.
City: City where the Houses/Apartments/Flats are Located.
Furnishing Status: Furnishing Status of the Houses/Apartments/Flats, either it is Furnished or Semi-Furnished or Unfurnished.
Tenant Preferred: Type of Tenant Preferred by the Owner or Agent.
Bathroom: Number of Bathrooms.
Point of Contact: Whom should you contact for more information regarding the Houses/Apartments/Flats.
Step 1: Import all the required libraries
Pandas : In computer programming, pandas is a software library written for the Python programming language for data manipulation and analysis and storing in a proper way. In particular, it offers data structures and operations for manipulating numerical tables and time series
Sklearn : Scikit-learn (formerly scikits.learn) is a free software machine learning library for the Python programming language. It features various classification, regression and clustering algorithms including support vector machines, random forests, gradient boosting, k-means and DBSCAN, and is designed to interoperate with the Python numerical and scientific libraries NumPy and SciPy. The library is built upon the SciPy (Scientific Python) that must be installed before you can use scikit-learn.
Pickle : Python pickle module is used for serializing and de-serializing a Python object structure. Pickling is a way to convert a python object (list, dict, etc.) into a character stream. The idea is that this character stream contains all the information necessary to reconstruct the object in another python script.
Seaborn : Seaborn is a Python data visualization library based on matplotlib. It provides a high-level interface for drawing attractive and informative statistical graphics.
Matplotlib : Matplotlib is a plotting library for the Python programming language and its numerical mathematics extension NumPy. It provides an object-oriented API for embedding plots into applications using general-purpose GUI toolkits like Tkinter, wxPython, Qt, or GTK.
#Loading libraries
import pandas as pd
import seaborn as sns
import pickle
import numpy as np
import plotly.graph_objects as go
import matplotlib.pyplot as plt
import plotly.express as px
from sklearn.metrics import r2_score
from sklearn.linear_model import LinearRegression, Ridge, RidgeCV, Lasso, LassoCV
from sklearn.model_selection import KFold, cross_val_score, train_test_split
import warnings
warnings.filterwarnings('ignore')
Step 2 : Read dataset and basic details of dataset
Goal:- In this step we are going to read the dataset, view the dataset and analysis the basic details like total number of rows and columns, what are the column data types and see to need to create new column or not.
In this stage we are going to read our problem dataset and have a look on it.
#loading the dataset
try:
df = pd.read_csv('C:/Users/YAJENDRA/Documents/final notebooks/House Rent Prediction/data/House_Rent_Dataset.csv') #Path for the file
print('Data read done successfully...')
except (FileNotFoundError, IOError):
print("Wrong file or file path")
Data read done successfully...
# To view the content inside the dataset we can use the head() method that returns a specified number of rows, string from the top.
# The head() method returns the first 5 rows if a number is not specified.
df.head()
Step3: Data Preprocessing
Why need of Data Preprocessing?
Preprocessing data is an important step for data analysis. The following are some benefits of preprocessing data:
It improves accuracy and reliability. Preprocessing data removes missing or inconsistent data values resulting from human or computer error, which can improve the accuracy and quality of a dataset, making it more reliable.
It makes data consistent. When collecting data, it’s possible to have data duplicates, and discarding them during preprocessing can ensure the data values for analysis are consistent, which helps produce accurate results.
It increases the data’s algorithm readability. Preprocessing enhances the data’s quality and makes it easier for machine learning algorithms to read, use, and interpret it.
Why we drop column?
By analysing the first five rows we found that there are columns that aren’t good for our model, se we gonna drop it using the below method:
df = df.drop(['Posted On'], axis=1)
Axis are defined for arrays with more than one dimension. A 2-dimensional array has two corresponding axes: the first running vertically downwards across rows (axis 0) and the second running horizontally across columns (axis 1).
After we read the data, we can look at the data using:
# count the total number of rows and columns.
print ('The train data has {0} rows and {1} columns'.format(df.shape[0],df.shape[1]))
The train data has 4746 rows and 11 columns
Count of unique occurences of every value in all categorical value
df["Floor"].value_counts()
1 out of 2 379
Ground out of 2 350
2 out of 3 312
2 out of 4 308
1 out of 3 293
...
11 out of 31 1
50 out of 75 1
18 out of 26 1
12 out of 27 1
23 out of 34 1
Name: Floor, Length: 480, dtype: int64
df["Area Locality"].value_counts()
Bandra West 37
Gachibowli 29
Electronic City 24
Velachery 22
Miyapur, NH 9 22
..
Kengeri Upanagara 1
Ittamadu, Banashankari, Outer Ring Road 1
Rmv Extension, Armane Nagar 1
snv la 1
Manikonda, Hyderabad 1
Name: Area Locality, Length: 2235, dtype: int64
df["Area Type"].value_counts()
Super Area 2446
Carpet Area 2298
Built Area 2
Name: Area Type, dtype: int64
df["City"].value_counts()
Mumbai 972
Chennai 891
Bangalore 886
Hyderabad 868
Delhi 605
Kolkata 524
Name: City, dtype: int64
df["Tenant Preferred"].value_counts()
Bachelors/Family 3444
Bachelors 830
Family 472
Name: Tenant Preferred, dtype: int64
df["Furnishing Status"].value_counts()
Semi-Furnished 2251
Unfurnished 1815
Furnished 680
Name: Furnishing Status, dtype: int64
df["Point of Contact"].value_counts()
Contact Owner 3216
Contact Agent 1529
Contact Builder 1
Name: Point of Contact, dtype: int64
df["Area Type"].value_counts()
Super Area 2446
Carpet Area 2298
Built Area 2
Name: Area Type, dtype: int64
The df.value_counts() method counts the number of types of values a particular column contains.
df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 4746 entries, 0 to 4745
Data columns (total 11 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 BHK 4746 non-null int64
1 Rent 4746 non-null int64
2 Size 4746 non-null int64
3 Floor 4746 non-null object
4 Area Type 4746 non-null object
5 Area Locality 4746 non-null object
6 City 4746 non-null object
7 Furnishing Status 4746 non-null object
8 Tenant Preferred 4746 non-null object
9 Bathroom 4746 non-null int64
10 Point of Contact 4746 non-null object
dtypes: int64(4), object(7)
memory usage: 408.0+ KB
The df.info() method prints information about a DataFrame including the index dtype and columns, non-null values and memory usage.
df.iloc[1]
BHK 2
Rent 20000
Size 800
Floor 1 out of 3
Area Type Super Area
Area Locality Phool Bagan, Kankurgachi
City Kolkata
Furnishing Status Semi-Furnished
Tenant Preferred Bachelors/Family
Bathroom 1
Point of Contact Contact Owner
Name: 1, dtype: object
df.iloc[ ] is primarily integer position based (from 0 to length-1 of the axis), but may also be used with a boolean array. The iloc property gets, or sets, the value(s) of the specified indexes.
Data Type Check for every column
Why data type check is required?
Data type check helps us with understanding what type of variables our dataset contains. It helps us with identifying whether to keep that variable or not. If the dataset contains contiguous data, then only float and integer type variables will be beneficial and if we have to classify any value then categorical variables will be beneficial.
objects_cols = ['object']
objects_lst = list(df.select_dtypes(include=objects_cols).columns)
print("Total number of categorical columns are ", len(objects_lst))
print("There names are as follows: ", objects_lst)
Total number of categorical columns are 7
There names are as follows: ['Floor', 'Area Type', 'Area Locality', 'City', 'Furnishing Status', 'Tenant Preferred', 'Point of Contact']
int64_cols = ['int64']
int64_lst = list(df.select_dtypes(include=int64_cols).columns)
print("Total number of numerical columns are ", len(int64_lst))
print("There names are as follows: ", int64_lst)
Total number of numerical columns are 4
There names are as follows: ['BHK', 'Rent', 'Size', 'Bathroom']
float64_cols = ['float64']
float64_lst = list(df.select_dtypes(include=float64_cols).columns)
print("Total number of float64 columns are ", len(float64_lst))
print("There name are as follow: ", float64_lst)
Total number of float64 columns are 0
There name are as follow: []
#count the total number of rows and columns.
print ('The new dataset has {0} rows and {1} columns'.format(df.shape[0],df.shape[1]))
The new dataset has 4746 rows and 11 columns
Step 2 Insights: -
Here, 7 columns are of categorial type and 4 columns are of integer type.
After this step we have to calculate various evaluation parameters which will help us in cleaning and analysing the data more accurately.
Step 3: Descriptive Analysis
Goal/Purpose: Finding the data distribution of the features. Visualization helps to understand data and also to explain the data to another person.
df.describe()
The df.describe() method returns description of the data in the DataFrame. If the DataFrame contains numerical data, the description contains these information for each column: count — The number of not-empty values. mean — The average (mean) value.
Measure the variability of data of the dataset
Variability describes how far apart data points lie from each other and from the center of a distribution.
Standard Deviation
The standard deviation is the average amount of variability in your dataset.
It tells you, on average, how far each data point lies from the mean. The larger the standard deviation, the more variable the data set is and if zero variance then there is no variability in the dataset that means there no use of that dataset.
So, it helps in understanding the measurements when the data is distributed. The more the data is distributed, the greater will be the standard deviation of that data.Here, you as an individual can determine which company is beneficial in long term. But, if you didn’t know the SD you would have choosen a wrong compnay for you.
df.std()
BHK 0.832256
Rent 78106.412937
Size 634.202328
Bathroom 0.884532
dtype: float64
We can also understand the standard deviation using the below function.
def std_cal(df,float64_lst):
cols = ['normal_value', 'zero_value']
zero_value = 0
normal_value = 0
for value in float64_lst:
rs = round(df[value].std(),6)
if rs > 0:
normal_value = normal_value + 1
elif rs == 0:
zero_value = zero_value + 1
std_total_df = pd.DataFrame([[normal_value, zero_value]], columns=cols)
return std_total_df
std_cal(df, float64_lst)
int64_cols = ['int64']
int64_lst = list(df.select_dtypes(include=int64_cols).columns)
std_cal(df,int64_lst)
zero_value -> is the zero variance and when then there is no variability in the dataset that means there no use of that dataset.
Variance
The variance is the average of squared deviations from the mean. A deviation from the mean is how far a score lies from the mean.
Variance is the square of the standard deviation. This means that the units of variance are much larger than those of a typical value of a data set.
Why do we used Variance ?
By Squairng the number we get non-negative computation i.e. Disperson cannot be negative. The presence of variance is very important in your dataset because this will allow the model to learn about the different patterns hidden in the data
df.var()
BHK 6.926499e-01
Rent 6.100612e+09
Size 4.022126e+05
Bathroom 7.823963e-01
dtype: float64
We can also understand the Variance using the below function.
zero_cols = []
def var_cal(df,float64_lst):
cols = ['normal_value', 'zero_value']
zero_value = 0
normal_value = 0
for value in float64_lst:
rs = round(df[value].var(),6)
if rs > 0:
normal_value = normal_value + 1
elif rs == 0:
zero_value = zero_value + 1
zero_cols.append(value)
var_total_df = pd.DataFrame([[normal_value, zero_value]], columns=cols)
return var_total_df
var_cal(df, float64_lst)
var_cal(df, int64_lst)
zero_value -> Zero variance means that there is no difference in the data values, which means that they are all the same.
Measure central tendency
A measure of central tendency is a single value that attempts to describe a set of data by identifying the central position within that set of data. As such, measures of central tendency are sometimes called measures of central location. They are also classed as summary statistics.
Mean
The mean is the arithmetic average, and it is probably the measure of central tendency that you are most familiar.
Why do we calculate mean?
The mean is used to summarize a data set. It is a measure of the center of a data set.
df.mean()
BHK 2.083860
Rent 34993.451327
Size 967.490729
Bathroom 1.965866
dtype: float64
We can also understand the mean using the below function.
def mean_cal(df,int64_lst):
cols = ['normal_value', 'zero_value']
zero_value = 0
normal_value = 0
for value in int64_lst:
rs = round(df[value].mean(),6)
if rs > 0:
normal_value = normal_value + 1
elif rs == 0:
zero_value = zero_value + 1
mean_total_df = pd.DataFrame([[normal_value, zero_value]], columns=cols)
return mean_total_df
mean_cal(df, int64_lst)
mean_cal(df,float64_lst)
zero_value -> that the mean of a paticular column is zero, which isn’t usefull in anyway and need to be drop.
Median
The median is the middle value. It is the value that splits the dataset in half.The median of a dataset is the value that, assuming the dataset is ordered from smallest to largest, falls in the middle. If there are an even number of values in a dataset, the middle two values are the median.
Why do we calculate median ?
By comparing the median to the mean, you can get an idea of the distribution of a dataset. When the mean and the median are the same, the dataset is more or less evenly distributed from the lowest to highest values.The median will depict that the patient below median is Malignent and above that are Benign.
df.median()
BHK 2.0
Rent 16000.0
Size 850.0
Bathroom 2.0
dtype: float64
We can also understand the median using the below function.
def median_cal(df,int64_lst):
cols = ['normal_value', 'zero_value']
zero_value = 0
normal_value = 0
for value in float64_lst:
rs = round(df[value].mean(),6)
if rs > 0:
normal_value = normal_value + 1
elif rs == 0:
zero_value = zero_value + 1
median_total_df = pd.DataFrame([[normal_value, zero_value]], columns=cols)
return median_total_df
median_cal(df, float64_lst)
zero_value -> that the median of a paticular column is zero which isn’t usefull in anyway and need to be drop.
Mode
The mode is the value that occurs the most frequently in your data set. On a bar chart, the mode is the highest bar. If the data have multiple values that are tied for occurring the most frequently, you have a multimodal distribution. If no value repeats, the data do not have a mode.
Why do we calculate mode ?
The mode can be used to summarize categorical variables, while the mean and median can be calculated only for numeric variables. This is the main advantage of the mode as a measure of central tendency. It’s also useful for discrete variables and for continuous variables when they are expressed as intervals.
df.mode()
def mode_cal(df,int64_lst):
cols = ['normal_value', 'zero_value', 'string_value']
zero_value = 0
normal_value = 0
string_value = 0
for value in float64_lst:
rs = df[value].mode()[0]
if isinstance(rs, str):
string_value = string_value + 1
else:
if rs > 0:
normal_value = normal_value + 1
elif rs == 0:
zero_value = zero_value + 1
mode_total_df = pd.DataFrame([[normal_value, zero_value, string_value]], columns=cols)
return mode_total_df
mode_cal(df, list(df.columns))
zero_value -> that the mode of a paticular column is zero which isn’t usefull in anyway and need to be drop.
Null and Nan values
- Null Values
A null value in a relational database is used when the value in a column is unknown or missing. A null is neither an empty string (for character or datetime data types) nor a zero value (for numeric data types).
df.isnull().sum()
BHK 0
Rent 0
Size 0
Floor 0
Area Type 0
Area Locality 0
City 0
Furnishing Status 0
Tenant Preferred 0
Bathroom 0
Point of Contact 0
dtype: int64
As we notice that there are no null values in our dataset.
- Nan Values
NaN, standing for Not a Number, is a member of a numeric data type that can be interpreted as a value that is undefined or unrepresentable, especially in floating-point arithmetic.
df.isna().sum()
BHK 0
Rent 0
Size 0
Floor 0
Area Type 0
Area Locality 0
City 0
Furnishing Status 0
Tenant Preferred 0
Bathroom 0
Point of Contact 0
dtype: int64
As we notice that there are no nan values in our dataset.
for value in objects_lst:
print(f"{value:{10}} {df[value].value_counts()}")
Floor 1 out of 2 379
Ground out of 2 350
2 out of 3 312
2 out of 4 308
1 out of 3 293
...
11 out of 31 1
50 out of 75 1
18 out of 26 1
12 out of 27 1
23 out of 34 1
Name: Floor, Length: 480, dtype: int64
Area Type Super Area 2446
Carpet Area 2298
Built Area 2
Name: Area Type, dtype: int64
Area Locality Bandra West 37
Gachibowli 29
Electronic City 24
Velachery 22
Miyapur, NH 9 22
..
Kengeri Upanagara 1
Ittamadu, Banashankari, Outer Ring Road 1
Rmv Extension, Armane Nagar 1
snv la 1
Manikonda, Hyderabad 1
Name: Area Locality, Length: 2235, dtype: int64
City Mumbai 972
Chennai 891
Bangalore 886
Hyderabad 868
Delhi 605
Kolkata 524
Name: City, dtype: int64
Furnishing Status Semi-Furnished 2251
Unfurnished 1815
Furnished 680
Name: Furnishing Status, dtype: int64
Tenant Preferred Bachelors/Family 3444
Bachelors 830
Family 472
Name: Tenant Preferred, dtype: int64
Point of Contact Contact Owner 3216
Contact Agent 1529
Contact Builder 1
Name: Point of Contact, dtype: int64
Categorical data are variables that contain label values rather than numeric values.The number of possible values is often limited to a fixed set.
Use Label Encoder to label the categorical data. Label Encoder is the part of SciKit Learn library in Python and used to convert categorical data, or text data, into numbers, which our predictive models can better understand.
Label Encoding refers to converting the labels into a numeric form so as to convert them into the machine-readable form. Machine learning algorithms can then decide in a better way how those labels must be operated. It is an important pre-processing step for the structured dataset in supervised learning.
df.head()
df.rename(columns = {'Area Type':'Area_Type','Area Locality':'Area_Locality','Furnishing Status':'Furnishing_Status','Tenant Preferred':'Tenant_Preferred','Point of Contact':'Point_of_Contact'}, inplace = True)
#from sklearn.preprocessing import LabelEncoder
df["Area_Type"] = df["Area_Type"].map({"Super Area": 1,
"Carpet Area": 2,
"Built Area": 3})
df["City"] = df["City"].map({"Mumbai": 4000, "Chennai": 6000,
"Bangalore": 5600, "Hyderabad": 5000,
"Delhi": 1100, "Kolkata": 7000})
df["Furnishing_Status"] = df["Furnishing_Status"].map({"Unfurnished": 0,
"Semi-Furnished": 1,
"Furnished": 2})
df["Tenant_Preferred"] = df["Tenant_Preferred"].map({"Bachelors/Family": 2,
"Bachelors": 1,
"Family": 3})
df["Point_of_Contact"] = df["Point_of_Contact"].map({"Contact Owner": 1,
"Contact Agent": 2,
"Contact Builder": 3})
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
df.Area_Locality = le.fit_transform(df.Area_Locality)
df = df.drop(['Floor'],axis = 1)
#After Encoding
df.head()
Skewness
Skewness is a measure of the asymmetry of a distribution. A distribution is asymmetrical when its left and right side are not mirror images. A distribution can have right (or positive), left (or negative), or zero skewness
Why do we calculate Skewness ?
Skewness gives the direction of the outliers if it is right-skewed, most of the outliers are present on the right side of the distribution while if it is left-skewed, most of the outliers will present on the left side of the distribution
Below is the function to calculate skewness.
def right_nor_left(df, int64_lst):
temp_skewness = ['column', 'skewness_value', 'skewness (+ve or -ve)']
temp_skewness_values = []
temp_total = ["positive (+ve) skewed", "normal distrbution" , "negative (-ve) skewed"]
positive = 0
negative = 0
normal = 0
for value in float64_lst:
rs = round(df[value].skew(),4)
if rs > 0:
temp_skewness_values.append([value,rs , "positive (+ve) skewed"])
positive = positive + 1
elif rs == 0:
temp_skewness_values.append([value,rs,"normal distrbution"])
normal = normal + 1
elif rs < 0:
temp_skewness_values.append([value,rs, "negative (-ve) skewed"])
negative = negative + 1
skewness_df = pd.DataFrame(temp_skewness_values, columns=temp_skewness)
skewness_total_df = pd.DataFrame([[positive, normal, negative]], columns=temp_total)
return skewness_df, skewness_total_df
float64_cols = ['int64']
float64_lst_col = list(df.select_dtypes(include=float64_cols).columns)
skew_df,skew_total_df = right_nor_left(df, float64_lst_col)
skew_df
skew_total_df
for i in df.columns:
print(df[i].skew())
0.5992157733648072
21.409942283288803
2.2998924373541834
0.06996486058904668
0.026910410775186515
-1.0891356651118271
0.3451856309998541
-0.10207880292378843
1.272951426254645
0.7654497697609112
We notice with the above results that we have following details:
8 columns are positive skewed
2 columns are negative skewed
Step 3 Insights: -
With the statistical analysis we have found that the data have a lot of skewness in them all the columns are positively skewed with mostly zero variance.
Statistical analysis is little difficult to understand at one glance so to make it more understandable we will perform visulatization on the data which will help us to understand the process easily.
Why we are calculating all these metrics?
Mean / Median /Mode/ Variance /Standard Deviation are all very basic but very important concept of statistics used in data science. Almost all the machine learning algorithm uses these concepts in data preprocessing steps. These concepts are part of descriptive statistics where we basically used to describe and understand the data for features in Machine learning
Step 4: Data Exploration
Goal/Purpose:
Graphs we are going to develop in this step:
Histogram
A histogram is a bar graph-like representation of data that buckets a range of classes into columns along the horizontal x-axis.The vertical y-axis represents the number count or percentage of occurrences in the data for each column
# Distribution in attributes
%matplotlib inline
import matplotlib.pyplot as plt
df.hist(bins=50, figsize=(30,30))
plt.show()
Histogram Insight: -
Histogram helps in identifying the following:
View the shape of your data set’s distribution to look for outliers or other significant data points.
Determine whether something significant has boccurred from one time period to another.
Why Histogram?
It is used to illustrate the major features of the distribution of the data in a convenient form. It is also useful when dealing with large data sets (greater than 100 observations). It can help detect any unusual observations (outliers) or any gaps in the data.
From the above graphical representation we can identify that the highest bar represents the outliers which is above the maximum range.
We can also identify that the values are moving on the right side, which determines positive and the centered values determines normal skewness.
#+ve skewed
df['Rent'].skew()
21.409942283288803
The target variable is positively skewed.A normally distributed (or close to normal) target variable helps in better modeling the relationship between target and independent variables.
Heatmap
A heatmap (or heat map) is a graphical representation of data where values are depicted by color.Heatmaps make it easy to visualize complex data and understand it at a glance
Correlation — A positive correlation is a relationship between two variables in which both variables move in the same direction. Therefore, when one variable increases as the other variable increases, or one variable decreases while the other decreases.
Correlation can have a value:
1 is a perfect positive correlation
0 is no correlation (the values don’t seem linked at all)
-1 is a perfect negative correlation
#correlation plot
sns.set(rc = {'figure.figsize':(15,20)})
corr = df.corr().abs()
sns.heatmap(corr,annot=True)
plt.show()
Notice the last column from right side of this map. We can see the correlation of all variables against diagnosis. As you can see, some variables seem to be strongly correlated with the target variable. Here, a numeric correlation score will help us understand the graph better.
print (corr['Rent'].sort_values(ascending=False)[:15], '\n') #top 15 values
print ('----------------------')
print (corr['Rent'].sort_values(ascending=False)[-5:]) #last 5 values`
Rent 1.000000
Bathroom 0.441215
Size 0.413551
BHK 0.369718
Point_of_Contact 0.338966
Area_Type 0.214867
Furnishing_Status 0.146251
City 0.124932
Area_Locality 0.018849
Tenant_Preferred 0.006027
Name: Rent, dtype: float64
----------------------
Area_Type 0.214867
Furnishing_Status 0.146251
City 0.124932
Area_Locality 0.018849
Tenant_Preferred 0.006027
Name: Rent, dtype: float64
corr
Heatmap insights: -
As we know, it is recommended to avoid correlated features in your dataset. Indeed, a group of highly correlated features will not bring additional information (or just very few), but will increase the complexity of the algorithm, hence increasing the risk of errors.
Why Heatmap?
Heatmaps are used to show relationships between two variables, one plotted on each axis. By observing how cell colors change across each axis, you can observe if there are any patterns in value for one or both variables.
Boxplot
A boxplot is a standardized way of displaying the distribution of data based on a five number summary (“minimum”, first quartile [Q1], median, third quartile [Q3] and “maximum”).
Basically, to find the outlier in a dataset/column.
features = df.columns.tolist()
sns.boxplot(data=df)
<Axes: >
The dark points are known as Outliers. Outliers are those data points that are significantly different from the rest of the dataset. They are often abnormal observations that skew the data distribution, and arise due to inconsistent data entry, or erroneous observations.
Boxplot Insights: -
Sometimes outliers may be an error in the data and should be removed. In this case these points are correct readings yet they are different from the other points that they appear to be incorrect.
The best way to decide wether to remove them or not is to train models with and without these data points and compare their validation accuracy.
So we will keep it unchanged as it won’t affect our model.
Here, we can see that most of the variables possess outlier values. It would take us days if we start treating these outlier values one by one. Hence, for now we’ll leave them as is and let our algorithm deal with them. As we know, tree-based algorithms are usually robust to outliers.
Why Boxplot?
Box plots are used to show distributions of numeric data values, especially when you want to compare them between multiple groups. They are built to provide high-level information at a glance, offering general information about a group of data’s symmetry, skew, variance, and outliers.
In the next step we will divide our cleaned data into training data and testing data.
Step 2: Data Preparation
Goal:-
Tasks we are going to in this step:
Now we will spearate the target variable and feature columns in two different dataframe and will check the shape of the dataset for validation purpose.
Split dataset into train and test dataset.
Scaling on train dataset.
1. Now we spearate the target variable and feature columns in two different dataframe and will check the shape of the dataset for validation purpose.
# Separate target and feature column in X and y variable
target = 'Rent'
# X will be the features
X = df.drop(target,axis=1)
#y will be the target variable
y = df[target]
X.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 4746 entries, 0 to 4745
Data columns (total 9 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 BHK 4746 non-null int64
1 Size 4746 non-null int64
2 Area_Type 4746 non-null int64
3 Area_Locality 4746 non-null int32
4 City 4746 non-null int64
5 Furnishing_Status 4746 non-null int64
6 Tenant_Preferred 4746 non-null int64
7 Bathroom 4746 non-null int64
8 Point_of_Contact 4746 non-null int64
dtypes: int32(1), int64(8)
memory usage: 315.3 KB
y
0 10000
1 20000
2 17000
3 10000
4 7500
...
4741 15000
4742 29000
4743 35000
4744 45000
4745 15000
Name: Rent, Length: 4746, dtype: int64
# Check the shape of X and y variable
X.shape, y.shape
((4746, 9), (4746,))
# Reshape the y variable
y = y.values.reshape(-1,1)
# Again check the shape of X and y variable
X.shape, y.shape
((4746, 9), (4746, 1))
2. Spliting the dataset in training and testing data.
Here we are spliting our dataset into 80/20 percentage where 80% dataset goes into the training part and 20% goes into testing part.
# Split the X and y into X_train, X_test, y_train, y_test variables with 80-20% split.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Check shape of the splitted variables
X_train.shape, X_test.shape, y_train.shape, y_test.shape
((3796, 9), (950, 9), (3796, 1), (950, 1))
df = df.drop(['Rent'],axis = 1)
Insights: -
Train test split technique is used to estimate the performance of machine learning algorithms which are used to make predictions on data not used to train the model.It is a fast and easy procedure to perform, the results of which allow you to compare the performance of machine learning algorithms for your predictive modeling problem. Although simple to use and interpret, there are times when the procedure should not be used, such as when you have a small dataset and situations where additional configuration is required, such as when it is used for classification and the dataset is not balanced.
In the next step we will train our model on the basis of our training and testing data.
Step 3: Model Training
Goal:
In this step we are going to train our dataset on different regression algorithms. As we know that our target variable is in continous format so we have to apply regression. In our dataset we have the outcome variable or Dependent variable.
Algorithms we are going to use in this step
Linear Regression
Lasso Regression
Ridge Regression
K-fold cross validation is a procedure used to estimate the skill of the model on new data. There are common tactics that you can use to select the value of k for your dataset. There are commonly used variations on cross-validation, such as stratified and repeated, that are available in scikit-learn
# Define kfold with 10 split
cv = KFold(n_splits=10, shuffle=True, random_state=42)
The goal of cross-validation is to test the model’s ability to predict new data that was not used in estimating it, in order to flag problems like overfitting or selection bias and to give an insight on how the model will generalize to an independent dataset (i.e., an unknown dataset, for instance from a real problem).
1. Linear Regression
Linear regression analysis is used to predict the value of a variable based on the value of another variable. The variable you want to predict is called the dependent variable. The variable you are using to predict the other variable’s value is called the independent variable.
This form of analysis estimates the coefficients of the linear equation, involving one or more independent variables that best predict the value of the dependent variable. Linear regression fits a straight line or surface that minimizes the discrepancies between predicted and actual output values. There are simple linear regression calculators that use a “least squares” method to discover the best-fit line for a set of paired data. You then estimate the value of X (dependent variable) from Y (independent variable).
Train set cross-validation
#Using Logistic Regression Algorithm to the Training Set
log_R = LinearRegression() #Object Creation
log_R.fit(X_train, y_train)
LinearRegression()
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
LinearRegression
LinearRegression()
#Accuracy check of trainig data
#Get R2 score
log_R.score(X_train, y_train)
0.23749058020975422
#Accuracy of test data
log_R.score(X_test, y_test)
0.4358371980771356
# Getting kfold values
lg_scores = -1 * cross_val_score(log_R,
X_train,
y_train,
cv=cv,
scoring='neg_root_mean_squared_error')
lg_scores
array([ 59535.71402494, 49818.53133825, 48165.28092165, 40426.12710301,
24435.25547094, 49105.9350403 , 36860.56935647, 179840.95746263,
49170.89477276, 42618.6216595 ])
# Mean of the train kfold scores
lg_score_train = np.mean(lg_scores)
lg_score_train
57997.78871504405
Prediction
Now we will perform prediction on the dataset using Logistic Regression.
# Predict the values on X_test_scaled dataset
y_predicted = log_R.predict(X_test)
# Evaluating the classifier
# printing every score of the classifier
# scoring in anything
from sklearn.metrics import classification_report
from sklearn.metrics import f1_score, accuracy_score, precision_score,recall_score
from sklearn.metrics import confusion_matrix
print("The model used is Linear Regression")
l_acc = r2_score(y_test, y_predicted)
print("\nThe accuracy is: {}".format(l_acc))
The model used is Linear Regression
The accuracy is: 0.4358371980771356
2. Lasso Regression
Lasso regression is a regularization technique. It is used over regression methods for a more accurate prediction. This model uses shrinkage. Shrinkage is where data values are shrunk towards a central point as the mean. The lasso procedure encourages simple, sparse models (i.e. models with fewer parameters). This particular type of regression is well-suited for models showing high levels of multicollinearity or when you want to automate certain parts of model selection, like variable selection/parameter elimination.
#Using Lasso Regression
from sklearn import linear_model
clf = linear_model.Lasso(alpha=0.1)
#looking for training data
clf.fit(X_train,y_train)
Lasso(alpha=0.1)
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
Lasso
Lasso(alpha=0.1)
#Accuracy check for training data
clf.score(X_train,y_train)
0.23749058018843472
#Get kfold values
Nn_scores = -1 * cross_val_score(clf,
X_train,
y_train,
cv=cv,
scoring='neg_root_mean_squared_error')
Nn_scores
array([ 59535.65825618, 49818.49364033, 48165.27582718, 40426.15535952,
24435.2109466 , 49105.9337378 , 36860.47727419, 179840.95340027,
49170.88331078, 42618.59970382])
# Mean of the train kfold scores
Nn_score_train = np.mean(Nn_scores)
Nn_score_train
57997.764145667585
Prediction
# Predict the values on X_test_scaled dataset
y_predicted = clf.predict(X_test)
Evaluating all kinds of evaluating parameters.
# Evaluating the classifier
# printing every score of the classifier
# scoring in anything
from sklearn.metrics import classification_report
from sklearn.metrics import f1_score, accuracy_score, precision_score,recall_score
from sklearn.metrics import confusion_matrix
print("The model used is Lasso Regression")
k_acc = r2_score(y_test, y_predicted)
print("\nThe accuracy is: {}".format(k_acc))
The model used is Lasso Regression
The accuracy is: 0.4358381054916356
3. Ridge Regression
Ridge regression is a model tuning method that is used to analyse any data that suffers from multicollinearity. This method performs L2 regularization. When the issue of multicollinearity occurs, least-squares are unbiased, and variances are large, this results in predicted values being far away from the actual values. Ridge regression is a method of estimating the coefficients of multiple-regression models in scenarios where the independent variables are highly correlated. It has been used in many fields including econometrics, chemistry, and engineering.
#Using Ridge Regression
from sklearn.linear_model import Ridge
rig = Ridge(alpha=1.0)
rig.fit(X_train, y_train)
Ridge()
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
Ridge
Ridge()
#Accuracy check of trainig data
#Get R2 score
rig.score(X_train, y_train)
0.23749051901110307
#Accuracy of test data
rig.score(X_test, y_test)
0.43585433531880213
# Get kfold values
Dta_scores = -1 * cross_val_score(rig,
X_train,
y_train,
cv=cv,
scoring='neg_root_mean_squared_error')
Dta_scores
array([ 59534.17744696, 49817.38451987, 48164.36783672, 40427.33250514,
24432.82381413, 49105.41340154, 36857.60095862, 179842.19904198,
49170.92087562, 42618.79979513])
# Mean of the train kfold scores
Dta_score_train = np.mean(Dta_scores)
Dta_score_train
57997.10201957321
Prediction
# predict the values on X_test_scaled dataset
y_predicted = rig.predict(X_test)
Evaluating all kinds of evaluating parameters.
from sklearn.metrics import classification_report
from sklearn.metrics import f1_score, accuracy_score, precision_score,recall_score
from sklearn.metrics import confusion_matrix
print("The model used is Ridge Regression")
r_acc = r2_score(y_test, y_predicted)
print("\nThe accuracy is {}".format(r_acc))
The model used is Ridge Regression
The accuracy is 0.43585433531880213
Insight: -
cal_metric=pd.DataFrame([l_acc,k_acc,r_acc],columns=["Score in percentage"])
cal_metric.index=['Linear Regression',
'Lasso Regression',
'Ridge Regression']
cal_metric
As you can see with Linear Regression Model we are getting a better result even for the recall which is the most tricky part.
So we gonna save our model now.
Step 4: Save Model
Goal:- In this step we are going to save our model in pickel format file.
import pickle
pickle.dump(log_R , open('House_rent_prediction_linearregression.pkl', 'wb'))
pickle.dump(clf, open('House_rent_prediction_lasssoregression.pkl', 'wb'))
pickle.dump(rig , open('House_rent_prediction_ridgeregression.pkl', 'wb'))
import pickle
def model_prediction(features):
pickled_model = pickle.load(open('House_rent_prediction_linearregression.pkl', 'rb'))
Rent = str(list(pickled_model.predict(features)))
return str(f'The Rent is {Rent}')
df.head()
We can test our model by giving our own parameters or features to predict.
BHK = 2
Size = 1000
Area_Type = 2
Area_Locality = 221
City = 7000
Furnishing_Status = 0
Tenant_Preferred = 2
Bathroom = 1
Point_of_Contact = 1
model_prediction([[BHK,Size,Area_Type,Area_Locality,City,Furnishing_Status,Tenant_Preferred,Bathroom,Point_of_Contact]])
'The Rent is [array([4596.93615518])]'
Conclusion
After observing the problem statement we have build an efficient model to overcome it but the accuracy is very less because of thr correlation between the columns is too close. Hence, the accuracy for the prediction is 43.35% for the rent.
Checkout whole project codehere(github repo).
🚀 Unlock Your Dream Job with HiDevs Community!
🔍 Seeking the perfect job?HiDevs Community is your gateway to career success in the tech industry. Explore free expert courses, job-seeking support, and career transformation tips.
💼 We offer an upskill program in Gen AI, Data Science, Machine Learning, and assist startups in adopting Gen AI at minimal development costs.
🆓 Best of all, everything we offer is completely free! We are dedicated to helping society.
Book free of cost 1:1 mentorship on any topic of your choice —topmate
✨ We dedicate over 30 minutes to each applicant’s resume, LinkedIn profile, mock interview, and upskill program. If you’d like our guidance, check out our services here
💡 Join us now, and turbocharge your career!
Deepak Chawla LinkedInVijendra Singh LinkedInYajendra Prajapati LinkedInYouTube ChannelInstagram PageHiDevs LinkedInProject Youtube LinkStreamlit Application Link