Menu

How do i start with regression on huge dataset with huge number of features ?

Reddit r/learnpython·u/Virtual-Current6295·6 months ago
#PR4veNGz
#huge#features#regression#reduce#start
Reading 0:00
15s threshold

The full dataset is about 80 GB, my laptop ram is just 16 gb. The good thing is i have already separated the data into separate feather files, and now i have files of around 500 mb each.
Other than the huge file size, i have huge number of features ( around 1500 ) and it's a complex problem, where i know linear regression is not a great choice, but to start with and establish some initial bounds / baselines i am trying linear regression.

I read up on how i can reduce features, and something like co variance matrix, pca would help me reduce co related features, but calculating that itself is a big challenge. I read up on stream, map, reduce which i might be able to use in python but it is still very slow. How can i do this faster ?

Basically, how do i load the data faster and perform operations on it ?