Python处理日志文件

Posted on Edited on In

背景

公司视频应用,需要针对用户的操作数据分析,得出应用内所有的视频的权重分析,方便以后更好的进行推荐视频操作.目前已有的log格式主要有:

1
2
3
action             vid           uid
view video      video_001        NaN
play video      video_001       user_00x

用户分为注册用户与游客两种,操作主要有view,like,comment,upload,download几种.为了方便处理,第一阶段只考虑针对游客访问的视频进行处理.这样形成一个只有action与uid的字典形式.处理数据主要使用python的pandas库.

清洗数据

日志初期action包括view,like,comment,upload,uid包括用户id以及代表游客的NaN空值.第一步需要把这些action具体量化,为了简单量化初期:view = 1 , like = 2 , comment = 3 , upload = 3 , download = 3.日志文件以csv格式存储.导入数据后进行处理:

1
2
3
4
5
6
7
8
9
10
11
import pandas as pd
# pandas读取csv
data = pd.read_csv('data.csv')
# action为playvideo的设置为1,即打分为1
data.loc[data['action']=='playvideo','action']=1
# 把NaN(即游客)值重置为0
data=data.fillna(value = '0')
# 过滤出uid值为0的数据
tourist_data = data.loc[data['uid']=='0']
del tourist_data['uid']
# tourist_data为action与uic的矩阵

DataFrame如下

1
2
3
4
5
6
   action                               vid
0       1  2c9f91345c3ed855015c5ee9cb904681
1       1  2c9f91345c3ed855015c52649f962d4f
2      -1  2c9f91345c2007f8015c2deee14c18cb
3       1  2c9f91345c3ed855015c5ee9cb904681
4       3  2c9f91345c2007f8015c2deee14c18cb

针对这个数据,需要进行清洗.找出重复的vid,并相加对应的action.得出的矩阵就是需要的vid的score排列.
首先DataFrame生成vid的list
vid_list=tourist_data['vid'].values.tolist()
问题分解为寻找list中重复的数值,并把数值对应的index记录下来.需要用到两个库enumerate与defaultdict
enumerate可以把list生成带有index的dict
defaultdict可以对list形成的dict进行统计处理

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# 生成带有index的列表
In [189]: [(v,i) for i,v in enumerate(vid_list)]
Out[189]:
[('2c9f91345c3ed855015c5ee9cb904681', 0),
 ('2c9f91345c3ed855015c52649f962d4f', 1),
 ('2c9f91345c2007f8015c2deee14c18cb', 2),
 ('2c9f91345c3ed855015c5ee9cb904681', 3),
 ('2c9f91345c2007f8015c2deee14c18cb', 4),
 ('2c9f91345c3ed855015c52649f962d4f', 5),
 ('2c9f91345c2007f8015c2deee14c18cb', 6),
 ('2c9f91345bf13cac015bfce28ef31002', 7),
 ('2c9f91345c3ed855015c5ee9cb904681', 8),
 ('2c9f91345c3ed855015c52649f962d4f', 9)]

# 利用defaultdict生成对重复vid处理后的dict
In [190]: vid_dict = defaultdict(list)

In [191]: for key, value in [(v, i) for i, v in enumerate(vid_list)]:
     ...:   vid_dict[key].append(value)
     ...:

In [192]: vid_dict
Out[192]:
defaultdict(list,
            {'2c9f91345bf13cac015bfce28ef31002': [7],
             '2c9f91345c2007f8015c2deee14c18cb': [2, 4, 6],
             '2c9f91345c3ed855015c52649f962d4f': [1, 5, 9],
             '2c9f91345c3ed855015c5ee9cb904681': [0, 3, 8]})

vid对应的list即为对应的index位置,利用index位置就可以为score_list进行处理累加

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# 生成score list
score_list = tourist_data['action'].values.tolist()
In [222]: vid_dict = defaultdict(list)

In [223]: for key, value in [(v, i) for i, v in enumerate(vid_list)]:
     ...:     vid_dict[key].append(value)

In [224]: vid_dict
Out[224]:
defaultdict(list,
            {'2c9f91345bf13cac015bfce28ef31002': [7],
             '2c9f91345c2007f8015c2deee14c18cb': [2, 4, 6],
             '2c9f91345c3ed855015c52649f962d4f': [1, 5, 9],
             '2c9f91345c3ed855015c5ee9cb904681': [0, 3, 8]})
In [227]: rank_list=[]
     ...: for i in vid_dict:
     ...:     score = 0
     ...:     for index in vid_dict[i]:
     ...:         score += int(score_list[index])
     ...:     rank_list.append(score)

In [231]: vid_list = []

In [232]: for i in vid_dict:
     ...:     vid_list.append(i)
In [233]: vid_list
Out[233]:
['2c9f91345bf13cac015bfce28ef31002',
 '2c9f91345c2007f8015c2deee14c18cb',
 '2c9f91345c3ed855015c5ee9cb904681',
 '2c9f91345c3ed855015c52649f962d4f']

In [234]: rank_list
Out[234]: [1, 4, 5, 3]
In [237]: vid_score = pd.DataFrame({'score':rank_list,'vid':vid_list})

生成score与vid的矩阵

针对生成的数据进行分析

使用altair对数据进行画图分析

1
2
3
4
5
# 导入csv文件
import pandas as pd
from altair import Chart, load_dataset
%matplotlib inline
vids_score = pd.read_csv('./res/vid_score.csv')
1
2
bins = [0, 10, 20, 30, 40, 50,60,70,80,90, 100,150,200,300,400,500,1000,1500]
scores = pd.cut(vids_score['score'], bins)
1
2
def get_stats(group):
    return {'count': group.count()}
1
2
grouped = vids_score['score'].groupby(scores)
bin_counts = grouped.apply(get_stats).unstack()
1
2
3
4
5
bin_counts
bin_counts.index = ['0~10', '10~20', '20~30', '30~40', '40~50', '50~60', '60~70',
                    '70~80', '80~90', '90~100','100-150','150-200','200-300','300-400','400-500','500-1000','1000-1500']
bin_counts.index.name = 'score'
plt=bin_counts.plot(kind='bar', alpha=0.5, rot=1,width = 0.8,align='center',figsize=(15,4))

生成统计图表png