前言
pandas 是基于 numpy 构建的含有更高级数据结构和工具的数据分析包类似于 numpy 的核心是 ndarray，pandas 也是围绕着 series 和 dataframe 两个核心数据结构展开的 。series 和 dataframe 分别对应于一维的序列和二维的表结构。pandas 约定俗成的导入方法如下：
from pandas import series,dataframe
import pandas as pd
1.1. pandas分析步骤
1、载入日志数据
2、载入area_ip数据
3、将 real_ip 请求数 进行 count。类似如下sql:
select inet_aton(l.real_ip),
count(*),
a.addr
from log as l
inner join area_ip as a
on a.start_ip_num = inet_aton(l.real_ip)
group by real_ip
order by count(*)
limit 0, 100;
1.2. 代码
cat pd_ng_log_stat.py
#!/usr/bin/env python
#-*- coding: utf-8 -*-
from ng_line_parser import nglineparser
import pandas as pd
import socket
import struct
class pdnglogstat(object):
def __init__(self):
self.ng_line_parser = nglineparser()
def _log_line_iter(self, pathes):
“””解析文件中的每一行并生成一个迭代器”””
for path in pathes:
with open(path, ‘r’) as f:
for index, line in enumerate(f):
self.ng_line_parser.parse(line)
yield self.ng_line_parser.to_dict()
def _ip2num(self, ip):
“””用于ip转化为数字”””
ip_num = -1
try:
# 将ip转化成int/long 数字
ip_num = socket.ntohl(struct.unpack(“i”,socket.inet_aton(str(ip)))[0])
except:
pass
finally:
return ip_num
def _get_addr_by_ip(self, ip):
“””通过给的ip获得地址”””
ip_num = self._ip2num(ip)
try:
addr_df = self.ip_addr_df[(self.ip_addr_df.ip_start_num