Data transformation is an integral part to data analysis and computation. Transformations can improve the quality of your data by standardizing formats and minimizing the effects of outliers or other factors that may influence the dataset’s normality. In statistical analysis, transformations help normalize the distribution of the dataset, allowing you perform parametric tests.
Below are a few different data transformation techniques, and their purposes:
-
Normalization: Scales data to a specific range (usually between 0-1). It is typically used when comparing data with different units, such as in machine learning algorithms.
-
Standardization (Z-score normalization): Centers the dataset around the mean with a standard deviation of 1. This is typically used to compare data points across different distributions or prepare dataset for machine learning algorithms that assume normal distribution.
-
Log Transformation: Reduces skewness and manages exponential growth in the dataset. It is typically used when data spans several orders of magnitude or has a long tail.
-
Square Root Transformation: Stabilizes the variance and normalizes the data distribution. It is typically used with count data or with a Poisson distribution (e.g., number of occurrences of an event).
-
Box-Cox Transformation: Makes data more normally distributed, often used to meet certain assumptions of linear regression or parametric tests.
-
Imputation: Fills missing data with substituted values, often used to handle datasets with missing values to ensure completeness.
When performing data transformation, you should be aware of the potential biases. Incorrect transformation techniques, such as using log transformation on a dataset that includes zeros or negative values, can distort the results. Additionally, removing or improperly transforming outliers can affect the validity of your results if those outliers carry significant information. Always carefully inspect how these transformations may impact your dataset and the results they convey.
An advantage of using Julius is its ability to help identify the best transformation technique based on the characteristics of your dataset. In this guide, we will learn how to use Julius to transform our dataset from non-normal distribution to a normal one. The dataset we will be using can be found here..
There are two sheets within this set: the first one contains the ‘rawdata’ (without the log-transformed income), and the second sheet contains ‘logtrans’ (with the transformed values from the ‘income’ column).
Prompt for Data Collection and Analysis
Objective: Using linear regression, analyze the relationship between years in business and income to investigate if length of stay at a company impacts income.
Dataset Description:
- Income: Represents the annual income for an individual.
- Years_in_Business: Represents the number of years with the business they are currently employed at.
- Sample Size: The dataset includes a diverse sample of individuals from various educational backgrounds and income levels (N= 1000).
Below is a quick overview of the dataset we will be using:
| Income | Years_in_Business |
|---|---|
| 99.26 | 12.99 |
| 53.48 | 11.72 |
| 55.61 | 13.3 |
| 66.05 | 15.05 |
| 45.09 | 11.53 |
| 38.72 | 11.53 |
| 147.69 | 15.16 |
| 79.68 | 13.53 |
| 46.67 | 11.06 |
| 38.78 | 13.09 |
| 53.55 | 11.07 |
| 30.55 | 11.07 |
| 119.44 | 12.48 |
| 8.23 | 8.17 |
| 30.97 | 8.55 |
| 28.83 | 10.88 |
| 14.39 | 9.97 |
| 34.7 | 12.63 |
| 18.08 | 10.18 |
Step 1: Data Visualization
Let’s first visualize our dataset with a histogram to understand the distribution.
Prompt: Please create a histogram on ‘income’.
In the image above, we can see that our dataset has a very long tail (trailing end of data on the right side) indicating that it is a non-normal distribution. To confirm this, let’s ask Julius to list the assumptions of the linear regression model and test them on this dataset.
Prompt: Please list the assumptions of a linear regression model and check the dataset to see if it passes them.
The following assumptions are as follows:
- Linearity and homoscedasticity
- Independence
- Normality
- No Multicollinearity (doesn’t apply to this dataset as we only have one independent variable).
The Durbin_Watson statistic tests for autocorrelation (if the first value is correlated with the immediate previous value). The result for this test was 2.070, which is above the 2.0 threshold, so it passed.
However, when looking at the q-q plot, we can see that our points do not align (approximately) along the reference line (red line), which indicates that our residuals are not normally distributed. Therefore, based off of all of these tests, we can say that ‘income’ is not normally distributed.
Now what? Let’s ask Julius to run a log transformation on our dataset to see if changes the normality.
Step 2: Log transformation
Below is the new table with an additional column with the log transformed values:
| Income | Log_Income | Years_in_Business |
|---|---|---|
| 99.26 | 4.60 | 12.99 |
| 53.48 | 3.98 | 11.72 |
| 55.61 | 4.02 | 13.3 |
| 66.05 | 4.19 | 15.05 |
| 45.09 | 3.81 | 11.53 |
| 38.72 | 3.66 | 11.53 |
| 147.69 | 5.00 | 15.16 |
| 79.68 | 4.38 | 13.53 |
| 46.67 | 3.84 | 11.06 |
| 38.78 | 3.66 | 13.09 |
| 53.55 | 3.98 | 11.07 |
| 30.55 | 3.42 | 11.07 |
| 119.44 | 4.78 | 12.48 |
| 8.23 | 2.11 | 8.17 |
| 30.97 | 3.43 | 8.55 |
| 28.83 | 3.36 | 10.88 |
| 14.39 | 2.67 | 9.97 |
| 34.7 | 3.55 | 12.63 |
| 18.08 | 2.90 | 10.18 |
Let’s see how the histogram looks with this new log_income variable:
In the image above, we can now see that our dataset follows more of a ‘bell-curve’ distribution. We can further check to see if our dataset is ready for linear regression by asking Julius to test the assumptions:
In the image above, we see that the ‘red line’ approximately follows the horizontal line.
In the q-q plot above, we can also see that the points lie approximately along the red line. This indicates that our dataset is normally distributed. Additionally, we can see that Julius has calculated the Durbin-Watson statistic, which was found to be 2.029. This is close to 2, and thus no autocorrelation is present.
Our log transformed dataset has passed all assumptions. We can now run a linear regression model.
Step 3: Running a linear regression model
Prompt: can we run a linear regression on the log_income versus years_in_business to check for a relationship?
From the image above, Julius has given us a comprehensive output of the regression results. The images below give us a rundown of the outcome:
Putting this all together, we can conclude that the following:
"The regression analysis indicates a significant strong positive relationship between ‘years_in_business’ and ‘log_income’ (t = 35.971, p < 0.001). The R² value was found to be 0.565, meaning that approximately 56.6% of the variance in ‘log_income’ is explained by ‘years_in_business’.
It was found for each additional year of business, the log transformed income increases by approximately 0.2897 units. In terms of actual income, this corresponds to an approximate increase of 33.6% for each additional year of business."
I hope you found this guide helpful in choosing the right transformation for your dataset. Thanks for joining me on performing log transformation on this dataset!
Keywords: AI Statistics, AI Statistical analysis, GPT, GPT-4o, Data transformation, Statistical analysis, Linear regression, normalization, income analysis, assumptions of linear regression, box-cox, log transformation, imputation.