pandas 常用函数

怼烎@ 2021-11-22 09:42 426阅读 0赞

一、常用功能及函数简介

包导入

一般我们需要做如下导入，numpy和pandas一般需要联合使用：

import pandas as pd  
import numpy as np  
本文采用如下缩写：

df：Pandas DataFrame对象  
s： Pandas Series对象  
数据导入

pd.read\_csv(filename)：从CSV文件导入数据  
pd.read\_table(filename)：从限定分隔符的文本文件导入数据  
pd.read\_excel(filename)：从Excel文件导入数据  
pd.read\_sql(query, connection\_object)：从SQL表/库导入数据  
pd.read\_json(json\_string)：从JSON格式的字符串导入数据  
pd.read\_html(url)：解析URL、字符串或者HTML文件  
pd.read\_clipboard()：从粘贴板获取内容  
pd.DataFrame(dict)：从字典对象导入数据  
数据导出

df.to\_csv(filename)：导出数据到CSV文件  
df.to\_excel(filename)：导出数据到Excel文件  
df.to\_sql(table\_name, connection\_object)：导出数据到SQL表  
df.to\_json(filename)：以Json格式导出数据到文本文件  
创建对象

pd.DataFrame(np.random.rand(20,5))：创建20行5列的随机数组成的DataFrame对象  
pd.Series(my\_list)：从可迭代对象my\_list创建一个Series对象  
df.index = pd.date\_range('1900/1/30', periods=df.shape\[0\])：增加一个日期索引  
index和reindex联合使用很有用处，index可作为索引并且元素乱排序之后，所以跟着元素保持不变，因此，当重拍元素时，只需要对index进行才重排即可:reindex。

另外， reindex时，还可以增加新的标为NaN的元素。

数据查看

df.head(n)：查看DataFrame对象的前n行  
df.tail(n)：查看DataFrame对象的最后n行  
df.shape()：查看行数和列数  
df.info()：查看索引、数据类型和内存信息  
df.describe()：查看数值型列的汇总统计  
s.value\_counts(dropna=False)：查看Series对象的唯一值和计数  
df.apply(pd.Series.value\_counts)：查看DataFrame对象中每一列的唯一值和计数  
apply的用处很多，比如可以通过跟lambda函数联合，完成很多功能：将包含某个部分的元素挑出来等等。

cities\['Is wide and has saint name'\] = (cities\['Area square miles'\] > 50) & cities\['City name'\].apply(lambda name: name.startswith('San'))

数据选取

df\[col\]：根据列名，并以Series的形式返回列  
df\[\[col1, col2\]\]：以DataFrame形式返回多列  
s.iloc\[0\]：按位置选取数据  
s.loc\['index\_one'\]：按索引选取数据  
df.iloc\[0,:\]：返回第一行  
数据清洗

df.columns = \['a','b','c'\]：重命名列名  
pd.isnull()：检查DataFrame对象中的空值，并返回一个Boolean数组  
pd.notnull()：检查DataFrame对象中的非空值，并返回一个Boolean数组  
df.dropna()：删除所有包含空值的行  
df.fillna(x)：用x替换DataFrame对象中所有的空值  
s.astype(float)：将Series中的数据类型更改为float类型  
s.replace(1,'one')：用‘one’代替所有等于1的值  
df.rename(columns=lambda x: x + 1)：批量更改列名  
df.set\_index('column\_one')：更改索引列  
数据处理：Filter, Sort, GroupBy

df\[df\[col\] > 0.5\]：选择col列的值大于0.5的行  
df.sort\_values(col1)：按照列col1排序数据，默认升序排列  
df.groupby(col)：返回一个按列col进行分组的Groupby对象  
df.groupby(col1).agg(np.mean)：返回按列col1分组的所有列的均值  
df.pivot\_table(index=col1, values=\[col2,col3\], aggfunc=max)：创建一个按列col1进行分组，并计算col2和col3的最大值的数据透视表  
data.apply(np.mean)：对DataFrame中的每一列应用函数np.mean  
数据合并

df1.append(df2)：将df2中的行添加到df1的尾部  
df.concat(\[df1, df2\],axis=1)：将df2中的列添加到df1的尾部  
df1.join(df2,on=col1,how='inner')：对df1的列和df2的列执行SQL形式的join  
数据统计

df.describe()：查看数据值列的汇总统计  
df.mean()：返回所有列的均值  
df.corr()：返回列与列之间的相关系数  
df.count()：返回每一列中的非空值的个数  
df.max()：返回每一列的最大值  
df.min()：返回每一列的最小值  
df.median()：返回每一列的中位数  
df.std()：返回每一列的标准差  
Pandas支持的数据类型

int 整型  
float 浮点型  
bool 布尔类型  
object 字符串类型  
category 种类  
datetime 时间类型  
补充：

df.astypes: 数据格式转换  
df.value\_counts:相同数值的个数统计  
df.hist(): 画直方图  
df.get\_dummies: one-hot编码，将类型格式的属性转换成矩阵型的属性。比如：三种颜色RGB，红色编码为\[1 0 0\]

参考文章：

Pandas官网

Pandas官方文档

Pandas Cheat Sheet -- Python for Data Science

10 minutes to Pandas

二、房价预测案例

根据给定的训练csv文件，预测给出的测试csv文件中的房价。

import numpy as np  
import pandas as pd  
from Cython.Shadow import inline  
  
import matplotlib.pyplot as plt  
\#matplotlib inline  
  
\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#1 oridinal data\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#  
train\_df = pd.read\_csv('input/train.csv', index\_col=0)\#数据导入  
test\_df = pd.read\_csv('input/test.csv', index\_col=0)  
  
print("type of train\_df:" + str(type(train\_df)))  
\#print(train\_df.columns)  
print("shape of train\_df:" + str(train\_df.shape))  
print("shape of test\_df:" + str(test\_df.shape))  
  
train\_df.head()\#数据查看  
\#print(train\_df.head())  
  
\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#2 smooth label\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#  
prices = pd.DataFrame(\{"price":train\_df\["SalePrice"\], "log(price+1)":np.log1p(train\_df\["SalePrice"\])\})  
print("shape of prices:" + str(prices.shape))\#数据创建  
prices.hist()\#直方图  
\# plt.plot(alphas, test\_scores)  
\# plt.title("Alpha vs CV Error")  
plt.show()  
  
y\_train = np.log1p(train\_df.pop('SalePrice'))  
print("shape of y\_train:" + str(y\_train.shape))  
  
\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#3 take train and test data together\#\#\#\#\#\#\#\#\#\#\#\#\#\#  
all\_df = pd.concat((train\_df, test\_df), axis=0)\#数据合并  
print("shape of all\_df:" + str(all\_df.shape))  
  
\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#4 make category data to string\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#  
print(all\_df\['MSSubClass'\].dtypes)  
all\_df\['MSSubClass'\] = all\_df\['MSSubClass'\].astype(str)\#数据格式转换  
all\_df\['MSSubClass'\].value\_counts()\#相同数值个数统计  
print(all\_df\['MSSubClass'\].value\_counts())  
  
\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#5 fill null\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#  
all\_dummy\_df = pd.get\_dummies(all\_df)\#one-hot编码,颜色RGB，则R编码为\[1 0 0\]  
print(all\_dummy\_df.head())\#下一行进行数据清洗，找到为空的属性，并按照空的数量对属性排序  
print(all\_dummy\_df.isnull().sum().sort\_values(ascending=False).head())  
  
mean\_cols = all\_dummy\_df.mean()\#数据统计，均值  
print(mean\_cols.head(10))  
  
all\_dummy\_df = all\_dummy\_df.fillna(mean\_cols)\#数据清洗，用()中的值代替空值  
print(all\_dummy\_df.isnull().sum().sum())  
  
\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#6 smooth numeric cols\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#  
numeric\_cols = all\_df.columns\[all\_df.dtypes != 'object'\]\#选取属性不是object，即数值型数据  
print(numeric\_cols)  
  
numeric\_col\_means = all\_dummy\_df.loc\[:, numeric\_cols\].mean()\#按照括号的索引选取数据，并求均值  
numeric\_col\_std = all\_dummy\_df.loc\[:, numeric\_cols\].std()  
all\_dummy\_df.loc\[:, numeric\_cols\] = (all\_dummy\_df.loc\[:, numeric\_cols\] - numeric\_col\_means) / numeric\_col\_std  
  
\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#7 train model\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#  
dummy\_train\_df = all\_dummy\_df.loc\[train\_df.index\]  
dummy\_test\_df = all\_dummy\_df.loc\[test\_df.index\]  
print("shape of dummy\_train\_df:" + str(dummy\_train\_df))  
print("shape of dummy\_test\_df:" + str(dummy\_test\_df))  
  
from sklearn.linear\_model import Ridge  
from sklearn.model\_selection import cross\_val\_score  
  
X\_train = dummy\_train\_df.values  
X\_test = dummy\_test\_df.values  
  
alphas = np.logspace(-3, 2, 50)  
test\_scores = \[\]  
for alpha in alphas:  
clf = Ridge(alpha)  
test\_score = np.sqrt(-cross\_val\_score(clf, X\_train, y\_train, cv=10, scoring='neg\_mean\_squared\_error'))  
test\_scores.append(np.mean(test\_score))  
plt.plot(alphas, test\_scores)  
plt.title("Alpha vs CV Error")  
plt.show()  
  
from sklearn.ensemble import RandomForestRegressor  
max\_features = \[.1, .3, .5, .7, .9, .99\]  
test\_scores = \[\]  
for max\_feat in max\_features:  
clf = RandomForestRegressor(n\_estimators=200, max\_features=max\_feat)  
test\_score = np.sqrt(-cross\_val\_score(clf, X\_train, y\_train, cv=5, scoring='neg\_mean\_squared\_error'))  
test\_scores.append(np.mean(test\_score))  
  
plt.plot(max\_features, test\_scores)  
plt.title("Max Features vs CV Error")  
plt.show()  
  
\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#8 stacking\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#  
ridge = Ridge(alpha=15)  
rf = RandomForestRegressor(n\_estimators=200, max\_features=.3)  
ridge.fit(X\_train, y\_train)  
rf.fit(X\_train, y\_train)  
  
y\_ridge = np.expm1(ridge.predict(X\_test))  
y\_rf = np.expm1(rf.predict(X\_test))  
  
y\_final = (y\_ridge + y\_rf)/2  
  
\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#9 submission\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#  
submission\_df = pd.DataFrame(data = \{'Id':test\_df.index, 'SalePrice':y\_final\})  
print(submission\_df.head())

转载于:https://www.cnblogs.com/aibabel/p/11011614.html