The full dataset is about 80 GB, my laptop ram is just 16 gb. The good thing is i have already separated the data into separate feather files, and now i have files of around 500 mb each.
Other than the huge file size, i have huge number of features ( around 1500 ) and it's a complex problem, where i know linear regression is not a great choice, but to start with and establish some initial bounds / baselines i am trying linear regression.
I read up on how i can reduce features, and something like co variance matrix, pca would help me reduce co related features, but calculating that itself is a big challenge. I read up on stream, map, reduce which i might be able to use in python but it is still very slow. How can i do this faster ?
Basically, how do i load the data faster and perform operations on it ?