韩友洪 2010年04月09日 星期五 16:57 | 2490次浏览 | 0条评论
httpsqs
在一个简单的项目中使用了httpsqs
流程非常简单:
1,python脚本定时扫描日志-》排重-》写入httpsqs队列
2,python daemon读日志,取出数据,处理
最初工作状态还不错,正常工作了2个星期
但是清明自己休息了5天,回来公司后发现数据不正确
[root@rrd ~]# curl "http://127.0.0.1:1218/?name=target_adv&opt=status"
HTTP Simple Queue Service v1.2
------------------------------
Queue Name: target_adv
Maximum number of queues: 100000000
Put position of queue (2nd lap): 9814174
Get position of queue (1st lap): 14309449
Number of unread queue: 95504725
[root@rrd ~]#curl "http://127.0.0.1:1218/?name=target_adv&opt=get"
HTTPSQS_GET_END
出现错误:
1,无法读取数据!
2,Number of unread queue: 95504725数值大了一个数量级
3,现在把以前的HTTPSQS停掉,换上新端口,仍然无法读取,但是可以查看!
root@rrd mailapp]# curl "http://127.0.0.1:1300/?name=target_adv&opt=get"
HTTPSQS_GET_END
[root@rrd mailapp]# curl "http://127.0.0.1:1300/?name=target_adv&opt=view&pos=1"
yansss8@sina.com,2010-03-30 11:45:32,中国,北京,北京,1214900334
4, 使用python接口直接调原始文件,正常
from tokyocabinet import btree
db = btree.BTree()
f="/data2/quene2/httpsqs.db"
db.open(f,btree.BDBOWRITER | btree.BDBOCREAT)
cur = db.cursor()
cur.first()
emails = {}
while True:
data = cur.rec()
#key = data[0]
line = data[1].strip()
o = line.split(",")
email = o[0]
ts = o[1]
#self.compareTs(ts,ts)
if emails.has_key(email):
try:
pre = emails[email]
pre_ts = pre.split(",")[1]
#当前时间更加新
if self.compareTs(ts,pre_ts):
emails[email] = line
except:
print "split error"
else:
emails[email] = line
# db.out(key)
# time.sleep(1)
#print line
try:
cur.next()
except:
print "NO record"
break
time.sleep(1)
for line in emails.values():
self.deal(line)
Zeuux © 2024
京ICP备05028076号
暂时没有评论